Training · learner work
The interview was in the inbox, the student still had to be told
Thirty students were job-hunting at once and their mail arrived in one place: job-board digests, application receipts, rejections, and occasionally a real interview request with a date on it. An AI Systems Architect at Colaberry built something that reads that inbox every two hours, files what is routine, and puts the few messages that need a decision in front of a person. No student is told anything until someone approves the words.
verified
All student projects- Industry
- Education
- Capability
- Operational AI triage with a human approval gate
- Status
- Shipped
- Built by
- Kesetebirhan Delele · Colaberry
- Published
- 2026-11-05
The situation
Colaberry places learners into data roles, and during a placement push thirty students are interviewing at once. Their job-search mail arrives centrally so staff can act on it.
The volume is the problem. Across fifty days the builder reports more than twenty thousand messages, and almost all of them are job-board digests, application receipts and rejections. The interview requests that carry a date, a format and a company are a small fraction of that, and they are the ones with a deadline attached.
Reading everything is how nothing gets missed, and reading everything does not scale to thirty simultaneous job searches. A missed interview invitation is not a tidy-inbox problem; it is a student who does not turn up.
But the obvious fix is worse than the problem. A system that reads the mail and messages students directly would be sending job news, in Colaberry’s name, on a model’s judgement.
What it had to do
- Find the few messages that carry a deadline, among many thousands that do not.
- Put every message that reaches a student in front of a person first.
- Make the system’s own uncertainty visible rather than resolving it quietly.
What constrained it
- A student must never be told about an interview by a system acting alone.
- The messages the model is least sure about are the ones a person must see, so low confidence has to route somewhere visible rather than into a folder.
- Running cost had to stay small enough that the economics are not the argument against it.
- Staff are not sitting in the application waiting for work, so it has to run on its own schedule and surface what needs a decision.
Decisions that made the difference
Five choices shaped the system. Each one lives at a specific point in the drawing above.
- At Before a student is told
Draft every notice, and send none of them
The output of this system is a message to a person about their career, sent in Colaberry’s name. A model that is right most of the time is not a sufficient basis for that.
Separate drafting from sending and automate only the first. The application writes the notice; a person approves, edits or rejects it on the Review screen, with the original email beside the draft. A pilot limit caps how many approved notices may begin an outbound sequence at all.
Evidence The Review screen, where each card carries the student, the company, the role, the time and the draft, with Approve, Edit draft and Reject beneath it.
2 OctIt costs a person’s attention on every notice, and the system cannot help outside the hours someone will approve. That was accepted as the price of never surprising a student.
- At When the model is unsure
Route uncertainty towards a person, not away
A classifier that is unsure can either guess or ask. Guessing produces a tidy inbox with unknown errors inside it.
Send low-confidence classifications to a separate review queue instead of filing them under a default category. The queue shows the message, the category chosen and the score that sent it there.
Evidence The Classification review screen, and the review outcomes: staff accepted most of what they opened as classified, moved some to a different category and rejected the rest.
10 OctA second queue that someone has to tend. Staff changed the outcome on about a fifth of the cases they opened, which is the queue earning its keep rather than the classifier failing.
- At When the student is unknown
Hold it rather than guess
Attaching an interview to the wrong student is worse than attaching it to nobody.
When the email does not identify a student, the record says so and waits for a person to connect it or discard it, rather than picking the closest match.
Evidence The unidentified-student state on the Review screen, and the Verify students tab that collects them.
9 OctUnmatched items accumulate until a person resolves them, and nothing chases them automatically.
- At Before anything leaves the system
Two keys on the door, and off by default
The step after approval is an outbound sequence of calls, texts and emails. One flag flipped by accident would reach students.
Live delivery requires the server to be armed AND a separate switch on the Settings page. Either alone leaves the system in shadow, and the default is off. The approval card says plainly that nothing is delivered while it is.
Evidence The commit of 20 October 2026 adding the two-key switch, and the earlier one changing the approval card to stop implying delivery.
20 OctApproval and delivery became different things, which is why this record counts notices drafted rather than students reached for the period it covers.
- At When reporting it
Correct your own cost figure
The system reported its own running cost, and the rate it logged against was the wrong model’s.
Find it, fix the rate used for logging, and report the corrected figure rather than the flattering one.
Evidence The commit of 21 October 2026 changing the logged rate, and the builder’s own report separating what the database measured from what he assumed.
21 OctThe published cost is lower than the system itself had been reporting, and the record says the correction may not reach every earlier row.
Who built it
- Kesetebirhan Delele, AI Systems Architect at Colaberry: took the system over in September 2026 and built the version described here, including the approval gate, the low-confidence review queue and the two-key delivery switch
- Jude Ojobo, Intern at Colaberry, AI Automation Engineer: built the original inbox pipeline, the Gmail label handling, the dashboard and the scheduled run, from March to September 2026, and handed it on in September
The build
- Jude Ojobo starts the original pipeline
- Handed over, and rebuilt
- Student notifications, gated from the start
- A test reaches a real address, and a guard follows
- The approval card stops implying delivery
- Acting today, and looking back, separated
- Live delivery needs two keys, and defaults to off
- The approval link needs a code
- The cost figure was wrong, and was corrected
- A third check on what counts as an interview
- Last commit in this record
Notes on 11 of the 11 steps
- Jude Ojobo starts the original pipeline. Jude Ojobo, then on the Colaberry internship, starts the inbox pipeline this system grew out of: mail import, Gmail label handling, the dashboard and the scheduled run. They commit through to September 2026, 76 commits in all, and hand the work on.
- Handed over, and rebuilt. The repository in its present form begins and Kesetebirhan Delele takes the work on. Every commit from here to the end of October is his, 187 in all.
- Student notifications, gated from the start. Email, SMS and voice channels for student notices arrive already behind a shadow gate, so the path that reaches a student is built switched off.
- A test reaches a real address, and a guard follows. A test sends mail to a real address through a stale code path. The fix is not only the path: a guard is added so a test cannot do it again.
- The approval card stops implying delivery. The card a person approves is changed to say plainly that nothing is delivered while the system is in shadow mode, rather than letting an approver assume it was.
- Acting today, and looking back, separated. The Dashboard becomes the screen for what needs a decision today; Reports becomes the place for pipeline, interviews and system health.
- Live delivery needs two keys, and defaults to off. Delivery to students requires the server to be armed AND a separate switch on the Settings page. Either alone leaves the system in shadow. The default is off.
- The approval link needs a code. Approving from a link now requires a one-time emailed code, and the approval page shows the full original email rather than a summary of it.
- The cost figure was wrong, and was corrected. Logged model spend had been recorded at a more expensive model’s rates, roughly seventeen times too high. The builder found it and fixed the rate used for logging.
- A third check on what counts as an interview. A gate that strips quoted replies and matches an email against earlier interviews, so a reply inside an old thread stops creating a new one. Built shadow-first.
- Last commit in this record. The pinned head of the default branch at the time this record was written.
What was built
A scheduled pass, a classifier that scores its own confidence, an extractor for the messages that turn out to be interviews, and a drafting step that stops short of sending. The approval gate is the point of the design: the application writes the notice, a person releases it.
Stack
- Python
- PostgreSQL
- OpenAI API
- Gmail API
More on what was built
Two-hourly processing run - Pulls new mail and processes it in batches, around the clock rather than in working hours. Each run is recorded with its email count, outcome and duration, so a failed run is visible rather than silent.
Classifier with confidence - Files each message into a category such as Technical Interview, Phone Screen, Offer, Rejection, Job Alert or Spam Review, and attaches a confidence score. The score is what decides whether a human sees it.
Interview detail extraction - For messages that look like real interviews, pulls out the company, position, date, time and format into fields a person can check against the original email, which stays readable beside them.
Student notification drafting - Writes the notice the student would get. It is held, not sent. The Review screen shows the draft, the email it came from, and what the student will receive.
Approval gate and pilot limit - Each notice is approved, edited or rejected by a named role. A pilot limit caps how many approved notices may begin a call, text and email sequence, so the blast radius of a bad batch is bounded by a number someone set.
Classification review queue - Low-confidence classifications go to a separate queue instead of being filed. This is where the application admits uncertainty rather than resolving it quietly.
Verify students, and the unmatched case - An interview email that cannot be tied to a student is held as 'No student identified' rather than guessed at, and waits for a person to connect it or discard it.
Reports - Pipeline funnel, per-student and per-company views, AI spend for the day, average run time, failure rate and the last twenty-five runs.
Capabilities
- Email triage
- Confidence scoring
- Structured extraction
- Human approval gate
- Low confidence review queue
- Scheduled processing
Integrations
- Gmail API for inbox access
- OpenAI API for classification and extraction
- PostgreSQL for mail classifications interviews and run history
Data stores
- PostgreSQL mail classifications interviews run history
The measurement
Every operational figure in this record is the builder’s own, computed from the application’s database for 17 September to 5 November 2026 and reported by email on 4 November. They are published as self-reported, because that database is not reachable from the platform that hosts this record and nobody else has recounted them.
Two figures are not self-reported. The commits in the period, and the share of commits naming an AI coding assistant as co-author, were counted directly from the public repository at a pinned commit, independently of the builder. Any reader can open that repository and recount them, which is what makes them verified rather than reported.
Two things are stated as assumptions rather than as fact. The staff cost avoided is priced at one assumed rate, thirty dollars an hour, which is the average of the three placeholder rates the builder supplied; the platform owner reviewed that figure on 5 November 2026 and accepted the rate, so it is the rate this record publishes rather than one waiting to be swapped. Accepting it settles which rate to use without making the rate a measurement, and the spread shown on the figure is the spread in the estimated hours, not in the rate. And there is no claim that the application is more accurate than a person, because no human error rate for this task was ever measured. The builder states this himself and asks that the comparison not be made.
Reading the repository alongside those figures turned up two things the report did not say. Delivery to students sits behind two independent keys, the server being armed and a separate switch on the Settings page, and it defaults to off; a notice being drafted and approved therefore does not establish that a student received it during the measured period, and this record does not claim one did. The platform owner confirmed on 5 November 2026 that the system is live, deployed and running, which is about the system today rather than about the fifty days these figures cover.
And the logged model cost was recorded at a more expensive model’s rates until late October, so the spend figure here depends on how far back the builder carried the correction.
The figures visible inside the walkthrough video are demonstration data on seeded records, not these production figures, and the two do not match. Every artifact caption says so.
Full notes on all 9 metrics
189 commits between 17 September and 28 October
Commits to the public repository during the measured period
verified
- Baseline
- None.
- Sample
- All commits on the default branch of the public repository.
- Methodology
- The repository is public. Its commit list was read directly at the pinned head commit, independently of the builder.
Limitations
- A count of commits measures activity, not working software.
234 of 267 commits name an AI coding assistant as co-author
Commits naming an AI coding assistant as co-author
verified
- Baseline
- None.
- Unit
- share of commits
- Sample
- All commits on the default branch.
- Methodology
- Read from the public repository directly, independently of the builder.
Limitations
- A trailer records involvement, not the division of work.
- It is stated because the record would otherwise imply the code was written without assistance.
20,773 messages across 30 students
Messages processed for thirty students in fifty days
illustrative
- Baseline
- None. There is no measured record of how this mail was handled before, so this sizes the work rather than comparing it with anything.
- Unit
- emails
- Sample
- All messages processed between 17 September and 5 November 2026 for thirty students.
- Methodology
- A query against the application's live database by its builder, covering the full fifty-day window.
Limitations
- Self-reported. The database is not reachable from this platform and the figure has not been independently recounted.
- The published category breakdown sums to 20,613 of 20,773; 160 messages are unaccounted for in the supplied figures.
- Thirty students over fifty days is one placement cycle, not a general rate.
268 of 20,773 messages needed a human decision
Share of processed mail that was an interview-type message needing a decision
illustrative
- Baseline
- None.
- Unit
- share of processed mail
- Sample
- All 20,773 processed messages in the period.
- Methodology
- A query against the application's live database by its builder.
Limitations
- Self-reported, and not independently recounted.
- A false negative, an interview filed as routine, is invisible to this measurement by construction.
- The builder reports one check over a single week, 2,888 messages, which found no clear miss among 21 messages with 'interview' in the subject. That check cannot see messages with generic subject lines.
167 notices drafted, none released without a person approving
Student notices drafted and held for a person to approve
illustrative
- Baseline
- None.
- Sample
- All student notices drafted in the period.
- Methodology
- A query against the application's live database by its builder.
Limitations
- Self-reported, and not independently recounted.
- 'None sent without approval' describes how the application is built, not a measured audit of every send.
- The gap between 174 extractions and 167 notices is not explained in the figures supplied.
- Approval releases a notice; delivery is a separate thing. During the measured period the outbound path sat behind two independent keys and defaulted to off, so this counts notices drafted rather than students reached. The platform is live and deployed as of 5 November 2026, which is after the period these figures cover.
106 of 127 harder cases were accepted as classified
Outcome when staff reviewed the classifications the model was least sure about
illustrative
- Baseline
- None. No human error rate for this task has been measured, so there is nothing to compare these outcomes against.
- Unit
- share of reviewed cases
- Sample
- The 127 low-confidence cases routed to the Classification review queue and opened by staff.
- Methodology
- A query against the application's live database by its builder.
Limitations
- Self-reported, and not independently recounted.
- A deliberately selected hard subset. Reading this share as a system accuracy rate would be wrong in both directions: it understates performance on ordinary mail and says nothing about unreviewed cases.
- The builder also reports the model's own average confidence across all mail, and it is high. A model's confidence in its answer is not the same thing as the answer being correct, and nothing here measures the difference.
571 of 572 scheduled runs completed
Scheduled processing runs that completed successfully
illustrative
- Baseline
- None.
- Unit
- share of runs
- Sample
- All 572 scheduled runs in the period.
- Methodology
- A query against the application's run history by its builder.
Limitations
- Self-reported, and not independently recounted.
- Measures completion, not correctness.
about $327 of model usage across 572 runs
Model usage cost for the period
illustrative
- Baseline
- None.
- Unit
- USD
- Sample
- All model calls across 572 runs between 17 September and 5 November 2026.
- Methodology
- A query against the application's recorded usage by its builder.
Limitations
- Self-reported, and not independently recounted.
- Model usage only. Hosting is excluded, and staff approval time is excluded.
- The staff-time figure published beside this one is an estimate priced at an assumed rate rather than a measurement, and is labelled that way.
- Logged spend used a more expensive model’s rates until the builder corrected it in late October. Whether the earlier rows were recomputed is not established here.
about $2,600 to $7,500 of staff time, against about $350 to run
Staff cost avoided, at an assumed average rate of thirty dollars an hour
illustrative
- Baseline
- None. No measurement exists of how long this work took when it was done by hand.
- Unit
- USD
- Sample
- The same period as the other figures.
- Methodology
- The builder estimated the hours this handling would take manually and gave a low, middle and high figure. Those hours are priced here at one assumed rate, the average of the three placeholders he supplied. The spread shown is the spread in the hours, not in the rate.
Limitations
- The rate is assumed, not measured. It is the average of the builder’s three placeholder rates, accepted by the platform owner on 5 November 2026 as the rate to publish; accepting it settles which rate to use and does not make it a measurement.
- The hours are an estimate of manual effort, not a measurement of it, which is why a spread is shown rather than one number.
- Staff time spent approving notices and clearing the review queue is not subtracted.
- This is the least solid figure on the record: it is a calculation over two assumptions.
What happened next
Not pursued
- Replace the assumed hourly rate with Colaberry’s measured cost
Unknown
- Measure how often a person gets this wrong
- Confirm how far back the model-cost correction was carried
In progress
- Account for the messages outside the three published categories
- Record what happens to a notice after it is approved
- Have the operational figures recounted by someone other than the builder
- Catch interview emails with generic subject lines
Notes on 7 of the 7 items
- Replace the assumed hourly rate with Colaberry’s measured cost. The platform owner reviewed this figure on 5 November 2026 and accepted the assumed rate of thirty dollars an hour as the rate to publish, so no substitution is pending. The figure stays an estimate, because the hours it prices were never measured.
- Measure how often a person gets this wrong. No human error rate for this task has ever been measured, so the system’s accuracy has nothing honest to be compared against. Until it does, the record does not claim the system is better.
- Account for the messages outside the three published categories. The reported category breakdown does not sum to the reported total; the remainder is not broken out.
- Record what happens to a notice after it is approved. Approval outcomes - approved unchanged, edited, rejected - are not reported, so the share of drafts a person accepted cannot be published.
- Have the operational figures recounted by someone other than the builder. They come from a database only the builder can reach, so they publish as self-reported. An independent recount would make them verified.
- Catch interview emails with generic subject lines. The builder’s single-week spot check looked at messages with the word interview in the subject, and could not see the ones without it.
- Confirm how far back the model-cost correction was carried. Logged spend used a more expensive model’s rates until late October. Whether earlier rows were recomputed is not established.
Meet the builder
Kesetebirhan Delele
AI Systems Architect, Colaberry
Project contribution
Took over an existing inbox pipeline in September 2026 and rebuilt it around a single idea: the system may read, sort and draft, but a person releases anything a student sees. Added the approval gate and the pilot limit, the Classification review queue that routes the least certain messages towards a human rather than away from one, the held state for an interview that cannot be matched to a student, interview detail extraction, and the Reports views for pipeline, cost and run reliability. Built the live-delivery switch that needs two independent keys and defaults to off. Most commits name an AI coding assistant as co-author, which this record states rather than hides.
Approved profileSkills demonstrated
- Deciding where the automation stopsDrafts every student notice and sends none of them: approval is a separate, human step, and a pilot limit caps how many approved notices may start an outbound sequence (architecture, decisions).
- Routing uncertainty towards a personLow-confidence classifications go to a visible review queue instead of a default category, and staff changed the outcome on a fifth of the cases they opened (decisions, measurement).
- Holding rather than guessingAn interview email that cannot be tied to a student is held as unidentified and waits for a person, because attaching it to the wrong student is worse than attaching it to nobody (architecture, artifacts).
- Two keys on the thing that reaches peopleLive delivery needs the server armed and a separate Settings switch, and defaults to off, so nothing reaches a student because one flag was flipped by accident (build timeline).
- Correcting his own numbersFound the logged model cost had been recorded at the wrong model's rates and fixed it, which is why the cost figure here is lower than the one the system reported before (build timeline, measurement).
- Separating what was measured from what was assumedReported which figures come from the database and which rest on his own assumptions, and asked that the system not be claimed as more accurate than a person because no human error rate was measured (measurement).
From the repository record.
What this project shows
The part of this build worth copying is not the classifier. It is the decision about where the automation stops: the system reads everything, files what is routine and writes the notice a student would receive, and then holds it until a person releases it. The messages it is least sure about are pushed towards a human rather than away from one, and an interview it cannot match to a student is held rather than guessed at. The figures are the builder’s own and are labelled as such, and the claim that would have been easiest to make - that this beats a person - is the one he asked not to make, because nobody measured the person.
Repositories
Keep reading
Build one of these