Cristian Carreño
LinkedIn ES

QA that agents can call

QA Console

Claude writes the Playwright script, the console runs it, and only an assertion that held makes a criterion pass.

Role
Sole author
Period
Oct 2026
Status
In use, v0.11
Code
Private, personal project
commits in 3 days
24
versions, 0.1 to 0.11
18
tests
136
MCP tools
11
  • Python
  • FastAPI
  • Playwright
  • SQLite
  • MCP
  • OAuth 2.1
  • bubblewrap
  • Claude Code

01 Context

Why it exists

Agents say UI work is done, and checking that by hand does not scale. Asking another model whether it looks right is an opinion, not a test. I wanted a verdict that is evidence: a criterion passes only if an assertion about it held in a real browser.

Two more problems shaped it. "It failed" covers two different findings, the test broke or the product broke, and a report has to keep them apart. And running QA requests from other people is dangerous: without fences, a console like this is a remote shell and a proxy into your network with a nice UI.

02 Architecture

How it works

A spec is free-text instructions plus acceptance criteria. The console groups the criteria by persona, has a headless Claude Code session write one Playwright script per group, checks the script, runs it itself in a fresh process, and grades every criterion from what its own harness recorded. No model takes part in grading.

QA Console run lifecycle Callers come in through one door with their own token and limits. The model only writes scripts; the console lints them, runs them in a fresh process, sandboxed for outside requests, and grades from recorded checks. Questions only a person can answer go to the operator's inbox. Callers Operator Console Sandbox, for outside requests Agent sessions request_qa CI, claude.ai MCP, own token Operator inbox in Agents Control Door policy + budget Intake validate, no model Lint falsifiable checks Grader no model, redacted Author claude -p, capped Executor fresh process spec spec + token one per persona script accepted check records ask a person
Callers come in through one door with their own token and limits. The model only writes scripts; the console lints them, runs them in a fresh process, sandboxed for outside requests, and grades from recorded checks. Questions only a person can answer go to the operator's inbox.

  1. Intake, without a model

    The spec is validated. Requests from outside may not define commands or environment variables for personas, and their target has to be on an allowlist. Only one module writes submitted state.

  2. Personas decide the groups

    A persona is either a fresh identity the script signs up, or an entry in the operator's local catalog; only the catalog may read secrets or run a provisioning command. There is one script per persona, and choosing them is deterministic.

  3. The model writes, inside a fence

    A headless claude -p runs with no user settings, no MCP servers, no saved session, a spending cap and a JSON schema for its answer. It writes the script against a small harness and may rehearse it against the live target, but a rehearsal is never the verdict.

  4. Lint before running

    A pass over the script's syntax tree refuses any check without an assertion, a try inside a check, and any reach into the harness's own recording. A refused script is a broken test: its criteria are blocked and it gets a repair attempt.

  5. Run, re-run, grade

    The console runs the script in a fresh process and streams progress live. If anything failed it runs the identical script once more: failing twice is failed, failing once is flaky. Grading is a pure function from recorded checks to passed, failed, flaky, blocked, needs a person, or not tested.

  6. People decide what assertions cannot

    A criterion only a person can judge becomes a question with screenshots in the operator's inbox in Agents Control Center. The answer is stored beside the harness's status, never over it.

03 Decisions

The calls that shaped it

  1. Decision 01

    The model writes; the console executes and grades

    Chose
    A verdict comes only from an assertion that held in the console's own run.
    Over
    A model as judge, parsing verdicts out of prose, or counting the model's rehearsal as the result.
    Because
    A verdict should be evidence, not opinion. No model can write "passed", and nothing parses it out of model text.
    Cost
    Authoring is slow and costs money, and criteria the model cannot assert end up as not tested or waiting for a person.

  2. Decision 02

    Fence the model's code, and never let it repair a failure

    Chose
    Lint, an identical re-run with no model involved, and repair attempts only for broken tests.
    Over
    Trusting the script, or letting the model fix whatever failed.
    Because
    When the product fails, that is the finding. "Fixing" a product failure is exactly how an agent fakes a pass, so the repair prompt forbids changing what any assertion requires.
    Cost
    A failing group runs twice, and the lint proves an assertion exists, not that it checks the right thing.

  3. Decision 03

    Two sandboxes, never the same one

    Chose
    The process that writes code and holds the model's login runs in one jail; the script that touches the target runs in another, with only the persona's values and network access to the target alone.
    Over
    One shared sandbox, or quietly running without one when the sandbox tool is missing; that case fails instead.
    Because
    A hostile spec could otherwise steer a script into reading the model's credentials and sending them to a target the requester controls.
    Cost
    Linux only, an allowlisting proxy to maintain, and outside requests are refused on machines without the sandbox.

  4. Decision 04

    Pre-verified clients over MCP, each with its own limits

    Chose
    Every client has its own hashed token, scopes, allowed targets and personas, daily budget and concurrency, and can be revoked alone. claude.ai connects through OAuth 2.1 where the invitation is the pre-authorisation.
    Over
    A shared secret, where revoking one client revokes everyone, and a full MCP SDK.
    Because
    Callers range from CI jobs to other agents to claude.ai, and each needs a budget and an off switch of its own.
    Cost
    A second listener for remote clients, and an OAuth server and an MCP server to maintain, both small.

  5. Decision 05

    Keep each kind of state with one writer

    Chose
    Submitted, observed, derived and human state in separate tables, each written by one module. A person can mark a criterion as a test problem; that needs a note and is counted apart, without touching the harness's status.
    Over
    Letting a person or a model overwrite a result.
    Because
    A re-run cannot erase what a person wrote, and a model cannot write an observation.
    Cost
    Only the test-problem mark changes the counts; a person's own pass or fail sits beside the harness result instead of replacing it.

04 What is interesting

Worth a closer look

  • Rehearsals are explicitly not evidence

    The model can explore the live UI as much as it likes while writing. None of that feeds a verdict; the verdict comes from re-running the exact script in a new process.

  • A failure taxonomy enforced as architecture

    Failed means the same script failed twice with no model involved. Blocked means the test itself broke. Repairs are allowed only for blocked, so an agent cannot turn a product failure into a pass.

  • Metering sessions before they finish

    The CLI reports cost only when a session ends, so sessions stream and every message is priced as it arrives. Checked against a real session, the estimate was within a fraction of a cent of the reported cost.

  • Built for agent callers

    Any session in Agents Control Center can ask for a run. Its run is sandboxed, paid by the asking session's own account, and a person's note about a broken test travels back to the session that asked.

05 Built with agents

How agents fit in

All 24 commits, written over three days, have a Claude co-author. The first commit already carried the rules in its rules file, each one borrowed from PR Review Console or Agents Control Center, where it had been paid for by an incident. One rule says to state what was measured and never more, and the CHANGELOG follows it with run ids and measured numbers.

The contract with Agents Control Center was agreed between two agent sessions, one working on each app, and each side shipped its half. A supervisor agent watches the console, recovers stuck runs through the CLI only, and turns a failure seen twice into a test and a pull request. It never marks a criterion, never weakens a check and never pushes to the main branch.

It runs on my Claude subscriptions, never an API key. Each run takes the account with the most of its week left and skips any account past 90%.

Contact

Want to go deeper on QA Console?

Reach me on LinkedIn