What's VibingAI dev news
MemberAnthropic Partner
Start learning free

Training · learner work

The Call Was Already Over. The Review Had to Be Fair.

An AI Systems Architect at Colaberry built the one pipeline in an admissions team's calling platform that reads calls the platform never made: the calls staff placed through the CRM's own dialer. It finds them, gets a transcript, decides which of two rubrics applies, scores the person who made the call, and flags the ones someone should look at. The review reads what 8,820 examined calls show about the decisions built into it.

verified

All student projects
What one job does with the calls the system never made, in 119 seconds. Narrated by a synthetic voice; the figures it states are the verified metrics recorded below.
Industry
Education
Capability
Operational AI scoring calls a person made
Built by
Colaberry team
Published
2026-09-28

The situation

The platform already wrote a record after every call it placed itself. The calls it could not see were the ones staff placed by hand through the CRM's own dialer, and those are the conversations that decide whether someone enrolls. Nothing in the platform knew they had happened.

Reviewing them meant someone listening to recordings one at a time, choosing which to listen to without knowing what was in them. In practice that means the calls that get reviewed are the ones somebody already suspects, which is the opposite of a sample.

So the question was not how to score a call. It was how to find calls nobody logged, decide which of them actually happened, get words out of them at a bearable cost, and judge them against a standard that already existed rather than one invented for the occasion.

A score about a named person is not a dashboard number. What gets stored has to be defensible to the person it describes, which is why the row keeps the signals that decided nothing and the reason when there is no score.

An invented example, for shape only. An admissions caller rings a lead at 4:53 PM. The lead says they are driving. The call lasts 40 seconds and ends. Nothing in the platform recorded it, because the platform did not place it. The next morning the scan finds the call at the vendor, sees a completed status and a duration over the floor, fetches the recording, transcribes it, reads the vendor's tags to decide it was a sales call, scores it against the sales rubric, and writes a row. Nobody had to choose to review it.

What constrained it

  • The calls are in a vendor's system, not this one, and the only way to know about them is to ask for them.
  • Transcribing and scoring a call costs money every time, and the history goes back years.
  • A score about a named person is sensitive, so what is stored has to be defensible.
  • The standard a person is judged against has to be one that already exists, not one invented for the occasion.
The three columns kept to settle a question, and the answer they gave: none of them
The three columns kept to settle a question, and the answer they gave: none of themA read-only query typed into the product's own database explorer: the three near-duplicate vendor fields the classifier consults after tags, and the enrolment date it consults after those, each counted against the whole table. All four are empty on every row. Beneath them, the signal that did decide: the vendor's tags on the great majority, the model on a small remainder. The row keeps these columns on purpose, because the specification says not to consolidate them until data says which one is authoritative. This is that data. The counts on this screen are the local copy's, one tenth the size of the production reading the record's own figures come from. Keyless local copy of the platform at the pinned commit with invented rows, not production data; the people on screen are invented and every number is a placeholder; times are Central.

Decisions that made the difference

Three choices shaped the system. Each one lives at a specific point in the drawing above.

  1. At Ship it off

    Ship it switched off, because running it spends money on every call

    A scan that transcribes and scores every call it finds spends money each time it runs, and the call history it can reach goes back years. Deploying that code and starting that spending are not the same decision, and a deploy that silently starts it gives nobody the chance to look at the output first.

    The setting defaults to off with the cost reason written beside it, and the check sits at the top of the cycle before any vendor call is made, so switching it off stops the spending rather than hiding the result. The specification carries it as an acceptance criterion, not an aside.

    Evidence The setting and its comment, the check at the top of the scan cycle, the unit test asserting the cycle is a no-op when it is off, and the specification's acceptance criterion, all at the pinned commit.

    8,820The default is a property of the code and says nothing about any deployment. This record does not read the flag; it reads the rows, and there are 8,820 of them across 6,672 completed cycles, so the deployment turned it on. A reader who checks the source and concludes the feature is off would be wrong, which is why the record says both halves.

  2. At One standard

    Hold the person to the standard already published for the machine

    Scoring a person against a rubric invented for the scoring is unfair and unarguable: there is nothing to appeal to, and a disagreement about the score has nowhere to go.

    The compliance dimension checks a specific list of disclosure rules taken from the organisation's own voice-agent script rather than written for this feature, and the module says so in its own words. The human and the automated caller are held to one published rule.

    Evidence The five compliance checks in the sales rubric, each mapping to a line in the voice agent's own script, and the module's own statement that these are existing rules rather than invented policy.

    5 checksThe loop then closed the other way: three days later the talk tracks people follow were written from the rubric, with an instruction to keep the two in step. That makes the standard harder to change quietly, because changing it now means changing what people were told to say.

  3. At Model last

    Ask the model last, and keep the evidence for the rule not yet written

    Deciding whether a call was sales or support could be a model call every time. It could also be read from signals the vendor already carries, and nobody knew which of three near-duplicate vendor fields was the authoritative one.

    Try the vendor's tags, then the three picklist fields, then an enrolment date, then this system's own call history, and only then a model. Keep the raw value of all three picklist fields on every row whether or not any of them decided anything, because the specification says not to consolidate them until data shows which is authoritative.

    Evidence The priority chain in the classifier, the three deliberately kept columns on the row, and the specification's instruction not to consolidate them yet.

    0 of 8,820Tags answered 6,109 of 6,257 classifications and the model answered 148, so the ordering paid for itself. The kept columns have now answered their own question: all three are empty on all 8,820 rows, as is the enrolment date, so the three middle links of the chain have never once decided anything. The question was worth asking and the answer is that, for this account, the vendor does not populate those fields at all.

Who built it

  • AI Systems Architect
The operator page: every examined call, scored or not, with the reason when there is none
The operator page: every examined call, scored or not, with the reason when there is noneThe staff call quality list as a person opens it, newest first: the five tiles across the top, and beneath them one row per examined call. All four outcomes are visible together, which is the point of writing a row either way: calls that were scored, a call that did not connect, a call that connected and produced no transcript, and the reason the code itself wrote for it. The lead name and phone columns are the product's own, added in a later commit, which is why every person shown here is invented and every number is a 555 placeholder. Keyless local copy of the platform at the pinned commit with invented rows, not production data; the people on screen are invented and every number is a placeholder; times are Central.

The build

  1. The whole vertical arrives in one commit
  2. The query that finds the calls is corrected
  3. The vendor's message list turns out to be nested twice
  4. The operator gets lead columns, a date filter, a tile and a place to fold a long summary
  5. The rubric is written back into the scripts people follow
  6. The scan was retrying forever under mismatched timeouts
Notes on 6 of the 6 steps
  • The whole vertical arrives in one commit. The scan job, two rubrics, the classifier, the row and its migration, the route, the page and the specification.
  • The query that finds the calls is corrected. Two days in, the discovery pass was finding the wrong set, and pagination was walking the wrong way.
  • The vendor's message list turns out to be nested twice. Twenty-eight minutes after the previous fix, a second shape mismatch with the same endpoint.
  • The operator gets lead columns, a date filter, a tile and a place to fold a long summary. Two commits twelve minutes apart, once there was something to look at.
  • The rubric is written back into the scripts people follow. The only commit of the seven with no co-author trailer.
  • The scan was retrying forever under mismatched timeouts. Sixteen days later: the queue's timeout and the job's lease disagreed, so a job still working was picked up again.
Four branches, and they add up to the whole table
Four branches, and they add up to the whole tableThe same database explorer, counting each examined call into exactly one of four outcomes, with the two transcript sources counted beside them. The branches add to the table's total, which is what makes a missing score explainable rather than merely absent: a call that did not connect, a call with no transcript available and a call that was transcribed and never scored are three different things, and each one leaves a row saying which it was. The fallback transcription accounts for most of the transcripts on this copy, as it does on the production reading. Keyless local copy of the platform at the pinned commit with invented rows, not production data; the people on screen are invented and every number is a placeholder; times are Central.

What was built

The pipeline is one scheduled job that does eight things in order and writes a row whichever of them fails. It searches the vendor for calls this system never placed, skips any it has already examined, decides whether the call connected, gets the words two ways, works out what kind of call it was, scores it against the matching rubric and stores the result.

Stack

  • Python
  • TypeScript
  • PostgreSQL
More on what was built

The parts, in the order they run. The scan cycle finds calls this system never made and schedules its own next run. The dedupe check skips a call already examined, so a cycle reaching back twenty-one months does not pay twice. The connection gate treats a call as having happened only if the vendor's status says completed and it lasted at least twenty seconds. The transcript step takes the vendor's own transcript when there is one and transcribes the recording when there is not. The classifier tries the vendor's tags, three vendor picklist fields, an enrolment date and this system's own call history before it asks a model. Two rubrics score a sales call and a support call on different things, out of 100 either way. The row keeps the score, the rubric, the raw signals and the reason when there is no score. One route, one page and one home tile are what an operator sees.

The ordering is the design. The dedupe check comes before any spending, the connection gate before the transcript, and the deterministic signals before the model, so a call that never happened costs nothing and a model call is spent only when the cheap signals are silent.

Everything the job learns ends up on one row, including what it could not do. A call that did not connect, a call with no transcript and a call the scorer could not score all leave a row behind, which is what makes the totals add up and the gaps countable.

Capabilities

  • Vendor API integration
  • Speech to text
  • Rubric scoring with a model
  • Deterministic classification
  • Operational dashboards

Integrations

  • Crm conversations API
  • Crm recordings API
  • Speech to text model
  • Scoring model

Data stores

  • PostgreSQL
The five dimensions behind a score, which no screen in the product ever shows
The five dimensions behind a score, which no screen in the product ever showsTwelve recently scored sales calls with their five dimension scores, the sum of those five, and the total the model actually returned. On most rows the two agree; on one of these twelve they do not, which is permitted, because the rubric tells the model it may return the sum or its own holistic judgment. This had to be queried rather than screenshotted from a page: the product stores the breakdown and renders none of it, and the only thing a row expands to show is the summary text. Keyless local copy of the platform at the pinned commit with invented rows, not production data; the people on screen are invented and every number is a placeholder; times are Central.

The measurement

Eleven figures, each with the number it is out of. The first is the one worth pausing on: three columns were kept on every row so that a question could be answered later from data rather than from opinion, and after 8,820 calls the answer is that none of them ever carried a value. Six more candidate figures were refused rather than published, because nothing in the tables supports them: how many calls a person actually opened, whether anything followed for the person scored, how many disclosure rules were actually broken, whether scores improved over time, what any of it saved, and how the six people compare.

Full notes on all 11 metrics

  • 0 of 8,820 rows carry a value in any of the three fields

    Rows carrying a value in any of the three vendor fields kept to decide the question

    verified

    Baseline
    None: the count is the finding.
    Sample
    Every row in the staff call quality table on 28 Sep 2026: 8,820.
    Methodology
    Read-only counts, one column at a time, with the totals stated per column rather than combined.

    Limitations

    • The columns were kept deliberately, to answer later from data which of the three is authoritative. They have answered, for this account and this period, and the answer is none of them.
    • This says nothing about whether another account would populate them.

  • 6,109 of 6,257 classifications, 97.6 percent, were decided by tags alone

    Classifications decided without spending a model call

    verified

    Baseline
    None: one cohort.
    Sample
    Every row that was classified at all: 6,257. The remaining 2,563 rows were never classified, because the call did not connect.
    Methodology
    Read-only counts grouped by the column the code writes to record which signal decided.

    Limitations

    • The model decided the other 148, 2.4 percent.
    • The three middle links of the chain decided nothing, because the fields they read are empty on every row.

  • 5,147 of 6,260 transcripts, 82.2 percent, came from the fallback

    Transcripts the builder's fallback produced rather than the vendor

    verified

    Baseline
    None: one cohort.
    Sample
    Every row carrying a transcript: 6,260.
    Methodology
    Read-only counts grouped by the column the code writes to record where the transcript came from.

    Limitations

    • The vendor's own transcription produced the other 1,113.
    • A transcript existing does not mean it is accurate; nothing here checks the words against the audio.

  • 6,246 of 8,820 examined calls produced a score

    Examined calls that produced a score

    verified

    Baseline
    None: one cohort.
    Sample
    Every row: 8,820.
    Methodology
    Read-only counts over the four mutually exclusive branches, which add to the whole table.

    Limitations

    • A row exists for every examined call, including the ones that produced nothing, which is what makes the remainder countable.
    • Of the 14 that were transcribed and not scored, 11 carry no recorded reason.

  • 70,098 of 139,000 points, 50.4 percent, came from compliance

    Share of all rubric points that came from the compliance dimension

    verified

    Baseline
    None: one cohort.
    Sample
    The 5,489 scored sales calls whose five dimension scores add exactly to the total the model returned.
    Methodology
    Read-only sums over exactly the calls where the parts and the total agree, so that the parts of the chart add to its denominator.

    Limitations

    • 219 sales calls are excluded because the model returned a total that is not the sum of its own parts, which the rubric permits.
    • Across all 5,708 the five parts total 146,966 against totals of 148,392, which is why the chart is drawn over the 5,489.

  • 5,489 of 5,708 scored sales calls have parts that add to their total

    Scored sales calls where the model's own parts add to its own total

    verified

    Baseline
    None: one cohort.
    Sample
    Every scored sales call with five numeric dimensions and a numeric total: 5,708.
    Methodology
    Read-only comparison per row of the sum of the parts against the stored total.

    Limitations

    • The rubric explicitly allows the model to return either the sum or a holistic total, so the 219 are permitted rather than wrong.
    • The differences run from 10 below the sum to 26 above it.

  • 442 of 5,708 scored sales calls, 7.7 percent

    Sales calls the model recorded as ending with a confirmed next step

    verified

    Baseline
    None: one cohort.
    Sample
    Every scored sales call: 5,708.
    Methodology
    Read-only counts grouped by the flag the rubric asks the model to return.

    Limitations

    • This is the model's reading of a transcript, not a booking anyone has confirmed in another system.
    • On the support side the model recorded the request as resolved on 184 of 538 calls.

  • 1,834 different wordings across 3,251 flagged rows

    Different wordings among the flags the model wrote

    verified

    Baseline
    None: one cohort.
    Sample
    Every row carrying a flag: 3,251.
    Methodology
    Read-only distinct counts and length statistics on the flag column. No flag text is quoted anywhere on this record.

    Limitations

    • 1,682 of the wordings appear on a single row and never again; they run from 2 to 237 characters.
    • The field is typed as a string with no controlled vocabulary, so this is a description of the design, not a defect in the model.

  • 8,820 calls examined

    Calls examined

    verified

    Sample
    Every row: 8,820, written between 27 Aug 2026 and 27 Sep 2026.
    Methodology
    Read-only count, with the earliest and latest call time and write time read alongside.

    Limitations

    • The calls themselves run from 27 Dec 2024 to 27 Sep 2026, so the scan reached back about twenty-one months.
    • Six staff identifiers and 2,445 contacts appear across the rows. Neither is printed, and no comparison between people is made.

  • 6,672 scan cycles completed, 1 failed

    Scan cycles completed

    verified

    Sample
    Every scan job between 25 Aug 2026 and 28 Sep 2026.
    Methodology
    Read-only counts grouped by status on the scheduled jobs table.

    Limitations

    • Three were pending at the moment of reading.
    • A completed cycle means the job finished, not that it found or scored anything.

  • 6,234 of 6,246 stored rubrics have the ten keys asked for

    Stored rubrics with the shape the prompt asked for

    verified

    Baseline
    None: one cohort.
    Sample
    Every row carrying a stored rubric: 6,246.
    Methodology
    Read-only counts of the number of top-level keys per stored rubric.

    Limitations

    • Every one of the 6,246 does carry the overall score, so no score is lost.
    • The 12 that differ include keys the schema never anticipated, and on one row a key more than 1,900 characters long. None of that text is quoted here.
What the home page says about it: a count of calls, and a share of them flagged
What the home page says about it: a count of calls, and a share of them flaggedThe analytics band on the product's home page, where this pipeline appears as one tile: how many calls were scanned and what share of them carry a flag. The flag is a label the model wrote into a free-text field; nothing in the schema records a person opening a row, so this number counts what the model marked, not what anyone reviewed. Keyless local copy of the platform at the pinned commit with invented rows, not production data; the people on screen are invented and every number is a placeholder; times are Central.

What happened next

Observed

  • The flag field is free text, so flags cannot be grouped
  • Twelve stored rubrics do not have the shape the prompt asked for
  • Eleven connected calls produced a transcript, no score and no reason
  • The five dimensions are stored and never shown
  • The scoring model reaches for round numbers
  • The running code sits on a branch the default branch does not carry
Notes on 6 of the 6 items
  • The flag field is free text, so flags cannot be grouped. 3,251 rows carry a flag and there are 1,834 different wordings among them, 1,682 of which appear on a single row. A dashboard can count that a flag exists, and does. It cannot count flags by kind.
  • Twelve stored rubrics do not have the shape the prompt asked for. The rubric is stored as JSON with no schema check, exactly as the model returned it. 6,234 rows carry the ten keys asked for; 12 carry something else.
  • Eleven connected calls produced a transcript, no score and no reason. The scorer is built never to raise, which is why the rows exist at all. On these eleven nothing recorded why there is no score.
  • The five dimensions are stored and never shown. No screen renders the rubric breakdown. The only row expansion on the page is the summary text.
  • The scoring model reaches for round numbers. 4,998 of 6,246 scores are divisible by five, which is worth knowing before treating a one-point difference as meaningful.
  • The running code sits on a branch the default branch does not carry. Opening the repository at its default branch will not show these files. Every claim on this record is pinned to the commit that runs, and the pin is what the evidence behind each claim is recorded against.

Meet the builder

AI Systems Architect

Project contribution

Built the scan, the two rubrics, the classifier and the row, and decided what the row keeps: the raw signals that decided nothing, the reason when there is no score, and the flag when a person should look.

Career facts as confirmed to Colaberry; project contribution from the repository record.

What this project shows

The pipeline does the thing that is easy to describe and hard to do: it looks at work nobody asked it to look at, against a standard that was already written down, and it records what it could not do as carefully as what it could. The rows that carry no score are the ones that make the rest of the numbers trustworthy.

Build one of these

Start the program that produced this work.

See the program