What's VibingAI dev news
MemberAnthropic Partner
$0 to start

Training · learner work

AI drafts the review, a person decides what the student reads

A learner on the AI Systems Architect Accelerator built the application that reviews the Accelerator’s own Stress Tests. It drafts each review against written, versioned rules for that test, then refuses to finish without a named human approver. This record follows what she built, story by story, and what the repository can and cannot prove about it.

verified

All student projects
The queue, one review decided point by point, and what the pilot measured
Industry
Education
Capability
Operational AI drafting a review a person must approve
Status
In progress
Built by
Swati Raman
Published
2026-11-04

The situation

Students on the AI Systems Architect Accelerator submit Stress Tests and ask for a critique. Every request was answered by hand: a reviewer opened each link, checked the work against the rules for that Stress Test, and wrote the feedback themselves.

The heaviest of the six tests was also the one that came up most often, so reviewer time concentrated exactly where it cost most. Requests queued behind whoever happened to be free, and a student who asked for feedback could wait the better part of a working day, sometimes longer, before hearing anything back.

The rules being applied lived in reviewers’ memories and in the feedback they had written before. Two reviewers could read the same submission and reach different conclusions, and neither could point at the rule a particular comment came from.

The question was never whether a model could write feedback. It was whether feedback a model wrote could be trusted to reach a student. The answer this project settled on is that it cannot, and should not: the model drafts, and a person decides every point.

What it had to do

  • Give the reviewer something to react to instead of a blank page.
  • Keep the final say with a person, on every individual point.
  • Make reviews consistent by checking each submission against written, versioned rules rather than recollection.
  • Leave an append-only record of which decisions were the model’s and which were the reviewer’s.

What constrained it

  • The historical review database is read-only. The application reads project history from it and never writes to it.
  • Nothing reaches a student automatically. Every finding is approved, edited or rejected by a reviewer before anything is prepared for sending.
  • Every rule is versioned per Stress Test, so a review can be traced to the rules that were in force when it was written.
  • The repository is private, so this record describes the work without exposing its contents.
The queue, and the four steps a review passes through
The queue, and the four steps a review passes throughThe reviewer’s first screen, taken from the builder’s own recording of the running application. Seven tiles count where every review currently sits: open, pending, drafted by the model, in review, feedback given, answered in Basecamp, and not from the project’s student. Underneath them the four steps are written out in one line that ends by saying nothing reaches the student without a reviewer. Two reviews are waiting, both on Stress Test 0. Every student on this screen is captioned fictional. This is the builder’s demonstration data, not a queue anyone is working, and no real submission or student name appears on it.

Decisions that made the difference

Five choices shaped the system. Each one lives at a specific point in the drawing above.

  1. At Decide

    The model drafts, a person decides

    A model that writes convincing feedback can be convincingly wrong, and the reader is a student being told what to fix.

    No finding reaches a student without a reviewer approving, editing or rejecting it individually. The guardrail was the first thing committed, on day one, rather than added once the drafting worked.

    Evidence Committed 13 September 2026 as the R4 safety guardrail, before any drafting existed.

    152Every rejection is recorded and feeds the next version of that rule, so the reviewer’s disagreement improves the rules instead of being absorbed silently.

  2. At Draft

    The history database is read-only, by construction

    The application needs years of past reviews to keep feedback consistent, and that database is the record of real student work.

    Connect read-only and never write. The review history informs a draft; nothing the application does can alter it.

    Evidence Read-only access committed 25 September 2026 alongside the ST0 to ST5 history extraction.

    0 of 10A whole class of failure is removed rather than guarded against: there is no code path that could corrupt the record, so no amount of retrying can.

  3. At Draft

    Rules are versioned files, one per Stress Test

    The rules lived in reviewers’ heads, so two reviewers could reach different conclusions and neither could point at the rule behind a comment.

    One live file and one draft file per Stress Test, older versions kept. Rules ship one test at a time, and a review can be traced to the rules in force when it was written.

    Evidence Rule files landed 26 October 2026; ST2 alone reached v20 by 3 November, each version written from points reviewers actually made.

    76Changing how a Stress Test is marked is a reviewable file change rather than a conversation, and a past review stays explainable.

  4. At Draft

    Two ways to run the model, one setting

    Paying per token suits steady volume; a subscription suits bursts. Committing to one of them at build time makes the other expensive.

    One setting switches between the API key and the Claude Code CLI on a subscription. Both paths use the same prompts, the same rules, the same daily limit, the same audit trail and the same circuit breaker.

    Evidence Subscription path committed 21 October 2026; the pilot ran on both and switched between them mid-flight.

    2The cost model became a deployment decision rather than an architectural one.

  5. At Queue

    Every outside call can fail, and says so

    Three external dependencies, any of which can be slow, rate-limited or down, with a reviewer waiting.

    A circuit breaker on Basecamp, the review database and Claude, each surfaced on a health view, with timeouts and capped retries. Repeated failures pause further calls rather than hammering an outage.

    Evidence Circuit breaker committed 27 October 2026 and shown on the health view.

    12A failed draft is recorded and retried rather than lost, and a reviewer can see which dependency is the reason the queue is quiet.

The build

  1. Project specification and governance docs
  2. The safety guardrail goes in first
  3. Submissions arrive from Basecamp
  4. Read-only access to the review history
  5. Append-only audit trail
  6. The Review Queue
  7. Reviewers sign in through Basecamp
  8. The first rule module ships
  9. A second way to run the model
  10. Rules become files with versions
  11. Circuit breaker on every outside call
  12. Rules still moving with reviewer feedback
Notes on 12 of the 12 steps
  • Project specification and governance docs. First commit. The spec, the governance docs and the connection to the platform land before any feature.
  • The safety guardrail goes in first. Never finalize a review without genuine human approval, committed on day one rather than retrofitted.
  • Submissions arrive from Basecamp. Retrieves submission data over OAuth 2.0, so a request is a real comment on a real thread.
  • Read-only access to the review history. Connects to the historical review database and extracts the ST0 to ST5 review history. Read-only by design.
  • Append-only audit trail. Every submission action is recorded, which is what later makes it possible to say which decisions were the model’s.
  • The Review Queue. The reviewer surface: queue, review status and history, the rules page, and every page load audited.
  • Reviewers sign in through Basecamp. Sign-in through the posting app’s registration, with the sign-in code redacted from the access log.
  • The first rule module ships. ST5 v2 ships while ST2 stays a draft: rule modules are switched on one Stress Test at a time, not all at once.
  • A second way to run the model. Drafts and Polish can run on a Claude subscription with no API key, selected by one setting.
  • Rules become files with versions. One live and one draft rule file per Stress Test, with older versions kept, so a review can be traced to the rules in force.
  • Circuit breaker on every outside call. Basecamp, the review database and Claude each get a breaker, surfaced on the health view.
  • Rules still moving with reviewer feedback. ST2 v20, ST3 v20 and ST4 v9 live, written from points reviewers actually made.
Approve, save edit, or reject. Every point, one at a time.
Approve, save edit, or reject. Every point, one at a time.The decision surface, which is the part of the application this record is named for. Each drafted finding is its own card with the suggested wording in an editable box and three buttons under it: approve, save edit, reject. There is no bulk action and nothing is pre-selected, so a reviewer who wants to finish cannot do it by agreeing once. Each card also carries the rule it came from and a link asking what that rule means. Below them the model’s problem findings are listed separately under a line saying they are read from the comment headings and are not a verdict. The content shown is demonstration text, prefixed as such in the recording.

What was built

One Python service, no build step on the pages, so the whole thing runs and tests on a single machine. A student posts a Critique comment on their project; that comment is the only thing that starts a review, and its id is the key everything downstream is built on.

Stack

  • Python 3.12
  • Fastapi
  • Pydantic v2
  • Uvicorn
  • Microsoft SQL server
  • Lancedb
More on what was built

The parts, in the order they run. The pickup finds new Critique comments twice a day, or when an admin presses Bring in and creates at most one review per comment, so running it twice costs nothing. The rule loader reads only the module for the identified Stress Test, at the version in force, never generic rules and never another test’s. The history reader pulls that student’s earlier submissions and feedback through ten stored queries, all of them reads. A vector store of local embeddings finds earlier reviews close to this one, so the same issue is not described three different ways across three rounds. Claude returns structured findings against those rules, each with a severity. The reviewer surface puts every finding in front of a person who approves, edits, rejects or adds to it. Only what that person approved is prepared for posting back.

The ordering carries the design. The idempotency key comes before any paid call, so a repeat pickup spends nothing. The rule module is chosen before the draft, so the model is never asked to invent a standard. The human decision comes after everything and before anything leaves, which is the one ordering the project is named for.

Failure is written into the same path rather than bolted beside it. Every outside call has a timeout and a capped retry: Basecamp 15 seconds and three attempts, the history database 30 seconds to connect and 120 per query, the model 60 seconds on the API and 180 through the command-line engine. Five failures inside the window open a breaker, calls stop for sixty seconds, the page says so, and one trial call closes it. A failed draft is written down with an error class and a correlation id and retried. The audit log is append-only, so the record of which points were the model’s and which were the reviewer’s cannot be tidied afterwards.

Capabilities

  • AI drafted review with human approval on every point
  • Versioned rule modules one per stress test
  • Append only audit trail of model and human decisions
  • Read only access to programme history
  • Timeouts capped retries and a circuit breaker on every outside call
  • Idempotent pickup keyed on the originating comment

Integrations

  • Basecamp API for requests and approved feedback
  • Microsoft SQL server read only
  • Claude API for drafting findings
  • Claude code command line engine as the second drafting route
  • Local embedding model for similar past feedback
  • A local assistant server exposing queue ids dates and status only

Data stores

  • Microsoft SQL server read only for programme history
  • Local files for the review queue and an append only audit log
  • Lancedb with local embeddings for similar past feedback
The last step, which is still a person pressing a button
The last step, which is still a person pressing a buttonAfter the reviewer has decided every point, the approved feedback is assembled into the comment that would go back to the student. Two routes are offered on the same screen: post it from here, or copy it and paste it by hand. Posting is switched off until the owner turns it on, so the button exists and the route it names has never been used on a real thread. A dataset advisory above it says in terms that it is advice for the reviewer only, never a failure and nothing to approve. The preparer address is an example one and the feedback body is demonstration content.

The measurement

Sixteen figures, in two kinds. Seven were counted in the repository at the commit this record pins and carry a verified badge: the stories, the commits, the lines, the test functions, the parametrized groups, the seventy-six rule versions and the ten stored queries.

The other nine come from the application’s own audit trail, its usage log and the read-only history database. None of those is reachable from outside the build, so they are labelled self-reported. Reading them that way is the point of the exercise rather than a caveat on it.

Two comparisons are estimates and are carried as estimates: about thirty minutes to review ST2 by hand, and about two minutes for the lighter tests with the application. Only the ST2 figure of about twelve minutes was measured, over four reviews.

The walkthrough at the top of this page closes on two slides that state those comparisons as percentages. They rest on the same reviewer estimates, and they are shown there in the builder’s own words rather than restated here as measured results.

Full notes on all 16 metrics

  • 0 of 10 stored queries write to the history database

    Stored queries that write to the history database

    verified

    Baseline
    The constraint the owner set: the program history database is never written to.
    Sample
    All ten .sql files in the application at commit 40bc448.
    Methodology
    Read-only inspection of the committed files, one at a time, with the write search run across the set rather than per file.

    Limitations

    • Six of the ten begin WITH and four begin SELECT; none of the five writing keywords appears in any of them.
    • The application keeps its own queue and audit log in its own storage, which is why it needs nothing from this database but reads.

  • 40 of 76 rule versions belong to two of the six tests

    Rule versions belonging to the two tests whose rules were rewritten most

    verified

    Baseline
    None: one count.
    Sample
    Every rule module version across the six Stress Tests at commit 40bc448: the files under each test’s versions/ directory PLUS the live.json beside it, seventy-six in all.
    Methodology
    Counting each test’s archived version files AND its live module, with the six counts stated separately so they add to the total. An earlier count of seventy missed the six live modules; corrected by the builder on 4 November 2026.

    Limitations

    • ST2 is also the test the builder records as the most requested, so the churn and the volume point the same way.
    • A version exists because a rule was changed; nothing here says whether the change was an improvement.

  • 939 of 2,577 test functions cover the rules and the drafting

    Test functions covering the rules and the drafting

    verified

    Baseline
    None: one cohort.
    Sample
    Every test function in the repository at commit 40bc448: 2,577.
    Methodology
    Matching definitions named test_ at the start of a line across every Python file, then grouping the files by the application package they sit under.

    Limitations

    • The remaining counts are human review 303, Basecamp 165, review queue 146 and 1,024 across everything else.
    • The builder’s own record states 3,349 tests collected by the runner, which is a larger number than this because parametrized groups expand into several cases each. This figure is the 2,577 definitions, counted here.

  • 32 stories, built over 51 days

    Written stories built and verified

    verified

    Baseline
    The requirements document in the same repository, which the traceability file maps each story back to.
    Sample
    Story files STORY-000 through STORY-031 at commit 40bc448.
    Methodology
    Listing the story files and reading the traceability map that links each to a requirement.

    Limitations

    • A story being present and marked verified is the builder’s own acceptance record, not an external audit.
    • A separate proposed directory holds work not built; it is not counted here.

  • 297 commits

    Commits from the first to the head of the branch

    verified

    Sample
    The full commit log to 40bc448: first commit 13 September 2026, head 3 November 2026.
    Methodology
    Counting the commit log at the head of the default branch.

    Limitations

    • The builder’s record states 292 as of 2 November; five more landed before the commit read here.

  • 21,159 lines of Python in the application package

    Lines of Python in the application package, excluding tests

    verified

    Sample
    Every Python file under the application package at commit 40bc448, test paths excluded.
    Methodology
    Line count over an explicit file list, with the scope stated rather than implied.

    Limitations

    • The test files themselves run to 31,518 lines, more than the application they cover.
    • Scope matters here: counting every Python file in the repository rather than only the application package gives 27,511.

  • 233 parametrized test groups

    Test groups that run the same assertion over many inputs

    verified

    Sample
    Every Python file at commit 40bc448.
    Methodology
    Matching the decorator across every Python file.

    Limitations

    • This is why the collected test total is larger than the count of definitions: each of these expands.

  • 103 of 152 drafted findings kept, 68 percent

    Drafted findings the reviewer kept

    anonymized

    Baseline
    None: one pilot.
    Sample
    The 152 drafted findings decided during the pilot, 25 September to 2 November 2026.
    Methodology
    Counts from the append-only audit trail, grouped by the decision recorded against each finding, as reported by the builder.

    Limitations

    • The reviewer also added 18 findings of their own, which are not drafted findings and are not in this denominator. Decisions recorded in all: 170.
    • A rejection is not waste: rejections are reviewed and written by hand into later rule versions, which is part of why seventy-six rule versions exist.

  • 18 findings added by the reviewer

    Findings the reviewer added that the model had not drafted

    anonymized

    Sample
    The pilot, 25 September to 2 November 2026.
    Methodology
    Counts from the audit trail by recorded author, as reported by the builder.

    Limitations

    • The reviewer surface exists to allow this: a draft the reviewer can only accept or refuse is not a review.

  • 40 repeat requests returned the existing review

    Repeat requests answered with the review already made

    anonymized

    Sample
    The pilot, 25 September to 2 November 2026.
    Methodology
    Counts from the audit trail of refusals by key, as reported by the builder.

    Limitations

    • The builder also records 11 repeat paid model calls avoided, keyed on the comment and the rule version together.
    • This is the idempotency rule working, not a count of mistakes by anyone.

  • 12 of 12 failed drafts recorded, none lost, retried within a cap of two

    Failed drafts recorded and retried within the two-attempt cap

    anonymized

    Baseline
    The behaviour the design refuses: a failed draft that disappears.
    Sample
    Every failed draft during the pilot: 12.
    Methodology
    Counts from the audit trail of recorded failures and their retries, as reported by the builder.

    Limitations

    • None was lost, which is the claim. How long any one took to recover is not recorded here.
    • A failure that repeats five times inside the window opens the breaker instead of retrying again.

  • 0 writes to the history database

    Writes to the program history database during the pilot

    anonymized

    Baseline
    The owner’s constraint: read only, always.
    Sample
    The pilot, 25 September to 2 November 2026.
    Methodology
    Counts from the application’s own logs, as reported by the builder.

    Limitations

    • The committed queries say the same thing structurally, and that half is verified here rather than reported.
    • Zero is the finding, and it is only meaningful because the application keeps its own data elsewhere.

  • about 12 minutes against an estimated 30 by hand

    Reviewer time on the heaviest Stress Test, measured in the application

    anonymized

    Baseline
    About 30 minutes reviewing by hand, which is a reviewer’s estimate and not a measurement.
    Sample
    Four ST2 reviews finished between 19 October and 2 November 2026. The readings were about 2, 10, 13 and 23 minutes.
    Methodology
    Active time between reviewer actions in the audit trail, as reported by the builder, set against a stated estimate.

    Limitations

    • Four reviews is a small sample and the readings run from about 2 to about 23 minutes.
    • The by-hand side is an estimate, so the difference between the two is not itself a measured quantity.
    • The middle value is used rather than the mean, because a sample this small lets one long review move an average.

  • about 41 seconds

    Time for the model to return a draft

    anonymized

    Sample
    57 timed drafts during the pilot, 25 September to 2 November 2026.
    Methodology
    Durations from the usage log, as reported by the builder.

    Limitations

    • The builder records that nine in ten finished within about 1.7 minutes, so the tail is longer than the middle suggests.
    • A typical draft carries 17 rule checks, each with a severity; a draft against a heavier test does more work.

  • 10,520 reviews since 2021

    Stress Test reviews already in the program history

    anonymized

    Sample
    The history database, read by the builder, covering 2021 onward.
    Methodology
    A read-only count against the history database, as reported by the builder.

    Limitations

    • The builder records 843 of these in the twelve months to her reading, which is the cohort her reviewer-time estimates are scaled on.
    • This is the corpus the application searches for a student’s earlier feedback; it is read and never written.

  • about 16 hours, median

    Wait for feedback after a student asked, before the application

    anonymized

    Sample
    534 answered requests between January 2023 and September 2026, read from the history database.
    Methodology
    Elapsed time per answered request from the history database, as reported by the builder.

    Limitations

    • The builder records that the slowest quarter waited over 39 hours and the slowest tenth over 65.
    • This is the state before the application. Nothing here measures the wait after it, which would need a cohort the pilot is too small to give.
What the reviewer sees when Basecamp stops answering
What the reviewer sees when Basecamp stops answeringThe failure path, recorded deliberately rather than left to be imagined. After repeated failures the application stops calling Basecamp for a minute and says so in a banner across the top of the page, written in words rather than an error code: it tried again and stopped, and anything that needs Basecamp will fail until it works again. Sign-in is deliberately not blocked by the same breaker, so a reviewer is never locked out of the application because an integration is down. The page below the banner is labelled in the recording as a demonstration whose health answer is made up.

What happened next

Shipped

  • Human approval before anything is sent
  • Submissions retrieved from Basecamp over OAuth
  • Read-only access to the historical review database
  • Append-only audit trail of every action
  • Review Queue and rules pages for reviewers
  • Reviewer sign-in through Basecamp
  • Versioned rule modules for ST0 to ST5
  • Suggested rewording the reviewer can edit
  • Circuit breaker and health view
  • Run on an API key or a Claude subscription

In progress

  • Deployment to a Colaberry server
Notes on 5 of the 11 items
  • Human approval before anything is sent. Committed first, on day one.
  • Versioned rule modules for ST0 to ST5. All six Stress Tests carry live and draft rule files.
  • Suggested rewording the reviewer can edit. Polish produces editable suggestions; accepting one saves the edited text.
  • Run on an API key or a Claude subscription. One setting; both paths share prompts, rules, limits and audit trail.
  • Deployment to a Colaberry server. Open question the builder raised: deploy next, or run more stress tests against the application first.

Meet the builder

Swati Raman

AI Systems Architect Accelerator learner

Project contribution

Designed and built the application end to end as a learner: the review queue, seventy-six versioned rule modules across six Stress Tests, the reviewer surface where every finding is approved, edited, rejected or added to, the append-only audit trail, and the read-only integration with the programme history.

Career facts as confirmed to Colaberry; project contribution from the repository record.

What this project shows

The lesson in this build is not that a model can write feedback. It is what it takes to let one near a student: rules written down and versioned before the drafting existed, a reviewer who decides every point, an audit trail that cannot be tidied, and a last step still switched off because nobody has decided to switch it on. 32 stories, 297 commits, 2,577 test functions, and the figure that matters most is 49, the findings the reviewer threw out.

Build one of these

Start the program that produced this work.

See the program