Self-triaging review queue
PR Review Console
For each pull request waiting on you, it decides what to do about it, then does the safe part.
- Role
- Lead author, with two teammates
- Period
- Aug 2026 – now
- Status
- In daily use
- Code
- Private, built at Enara Health
- of 420 commits are mine
- 330
- test functions
- 2,827
- queue rows decided by rules alone
- 47 of 49
- posted verdicts matched the model in shadow mode
- 8 of 8
- Python
- FastAPI
- htmx
- SQLite
- Claude Code
- GitHub
- Bitbucket
01 Context
Why it exists
A tech lead's review queue can run to seventy-odd open pull requests across two forges, and most of them do not need that person's review. The real cost is deciding, every morning, which code to read. A 30-line migration outranks a 900-line UI refactor. A pull request stacked on an unreviewed parent cannot be reviewed until the parent moves. One you approved last week still shows up as if it needed you.
It replaced a model-driven classifier that produced a snapshot and forgot it. The reviewer needs three things instead: what state each pull request is in, how much attention it deserves, and what changed since they last looked.
02 In use
What it does
The real app, filled with an invented company's pull requests and run with every worker switched off. The rules engine sorted the queue itself.
03 Architecture
How it works
A local FastAPI app collects the queue on a schedule, decides each row with a pure rules engine, and asks a model only about the few rows the rules cannot settle. Reviews run as headless Claude Code sessions and stream into the browser. Only the console itself posts to a forge.
-
Collect in tiers
One loop does all the collecting. Every three minutes it runs one search and enriches only the pull requests whose update time moved; a lighter pass and a full sweep run less often. Observations go into SQLite.
-
Three kinds of state, one writer each
What the forge said, what the person did and what the rules decided live in separate column families. The collector's upsert never names a user column, and a test checks that byte for byte.
-
Rules first
A pure function with no I/O and no clock sorts each row into act, review, delegate, blocked, monitor or draft, and ranks it. Every decision carries a rule id, a templated reason and the rules version. For two judgement calls it reads the actual diff.
-
A model for the ambiguous band
Only rows the rules cannot settle, and whose signal moved, go to the model, in one batched call with a budget and a timeout. The answer is checked against hard overrides, and a reply that breaks one is discarded.
-
Reviews as headless sessions
A review is a headless Claude Code run: an orchestrator on a strong model fans out to specialists picked by what the change touches, such as security, performance or data. Output is saved before it is streamed to the browser, and the verdict comes back in a structured file, not parsed from prose.
-
Posting without a model
A person submits from the dialog, or the console posts unattended if a reviewed per-repository policy allows that verdict and five guards pass: the head has not moved, you have not answered already, no review is waiting on you, a change request is not sent unattended (with two narrow exceptions), and one post goes out per tick.
04 Decisions
The calls that shaped it
-
Decision 01
Deterministic triage, with the model only for an ambiguous band
- Chose
- A rules engine, with the model as an advisor inside a fenced region.
- Over
- The earlier approach of a model classifying the whole queue.
- Because
- It was measured. On a frozen queue of 49 pull requests the model and the rules agreed on 45, and the rules were right on all four disagreements. Rules are also reproducible, cost nothing and give stable reasons; a reworded reason would flood the change feed.
- Cost
- The rules have to track policy by hand: raising the size ceiling changed one pinned row in the golden test. Part of the win was computing facts the old pipeline never had, such as whether you already approved.
-
Decision 02
No model in the posting path; widen autonomy by measurement
- Chose
- A review proposes; a reviewed policy file decides whether that verdict may go out unattended. Unattended posting is off by default and ran in shadow mode first.
- Over
- Letting the review skill post, and posting change requests unattended, which would start a re-review loop the pause switch cannot reach.
- Because
- In shadow mode, of the eight verdicts a person actually posted, the person agreed with the model on all eight and left the body unchanged on all eight.
- Cost
- You lose the last-second look, and it takes several guards to make unattended posting safe.
-
Decision 03
Admission control that counts the whole machine
- Chose
- A review starts only if machine-wide CPU and memory are under their ceilings, counting reviews started by other sessions too.
- Over
- A task queue such as Celery or RQ, or a concurrency limit inside the app.
- Because
- Several sessions each fanned out to about eight sub-agents and the machine ran out of CPU and memory. A per-app limit does not help when none of the load is the app's own.
- Cost
- Foreign reviews are recognised by their command line, which is fragile and pinned by tests.
-
Decision 04
Supervise like OTP, not like cron
- Chose
- An in-process supervisor with permanent, transient and temporary children, exponential backoff, and escalation after five restarts in a minute.
- Over
- Cron jobs or a scheduler library.
- Because
- A monitoring loop once stopped in silence, and a standby instance served a stale queue for 67 hours.
- Cost
- A custom supervisor and a leader election to own.
-
Decision 05
Independent reviewers across Claude accounts
- Chose
- Several accounts with linear, round-robin or load-balanced assignment, and a hard rule: an account that wrote code on a pull request may not review it.
- Over
- Fixed roles per account, which turned a spent review account into a dead end.
- Because
- Reviews keep flowing when one account's plan runs out, and nothing grades its own work.
- Cost
- Sessions stay pinned to the account that created them.
05 What is interesting
Worth a closer look
-
The rules-versus-model result is a test
The golden test runs the engine over the frozen 49-row queue and compares it with what the model said. It names each disagreement, so a change to the rules has to argue with recorded evidence.
-
Honest about where the boundary leaks
The docs list where "the model never posts" is not airtight: an assistant drawer can post when asked, and allowing one CLI command allowed writes. Write confinement ended up as a fail-closed hook, because the documented permission flags did not confine writes.
-
Learned by running the CLI
A budget flag is a circuit breaker, not a cap: it is checked between turns, and a measured run spent about 3.8 times a small limit. The count of permission denials, not the error flag, turned out to be the health signal.
-
Escalate the finding, not the agent
Specialists run on a cheaper model while the orchestrator keeps the strong one. A finding that needs more thought goes to an advisor on the strongest model, at most three times per review.
06 Built with agents
How agents fit in
Most of the code was written with Claude Code: 332 of the 420 commits carry a Claude co-author trailer, 247 of them among my 330. I wrote the engine, the model tier, supervision, admission, the account balancing, unattended review and posting, and the MCP server; a teammate built much of the review dialog and the pages.
There is no rules file in this repository. The working contract is a short guide for pull requests: say what was measured, and never more. Every failure it lists is an agent-shaped overclaim, and the pull requests are reviewed by a Claude agent as well as people. Twice, that agent corrected a claim made in a commit message.
The app is also a heavy user of agents: full reviews with specialist lanes, quick checks, predictions before a review, fixes for review comments on a separate account, and resumable conversations when a person has to decide.