Can Jev tell an ad from everything else?

No baseline model was run on these texts, so nothing here says whether Jev is the right choice for this task — only how it did against these labels.

For a two-option Choice, Jev’s confidence is the gap between the two option probabilities. Every one of the 12 errors came back with a small gap. The cost shows up too: a threshold that catches all of them also sends some correct answers to the fallback.

The thresholds were fixed before the run but scored on these same 200 texts, with no holdout. No fallback model was run, so accuracy on the texts routed away is unmeasured. Zero observed errors is not zero errors: at 0.70 the 95% interval still reaches down to 97.8%.

45.5% of these texts are ads — far more than most real feeds. Precision depends on that share. Holding Jev’s measured recall and false-positive rate fixed, the same classifier on a feed that is 5% ads would be right about roughly 38.1% of the texts it flags, and the false-positive rate’s own uncertainty puts that anywhere from about 25% to 54%.

Each row shows the AI-adjudicated label Jev was scored against and the source corpus’s original human label. Rows marked sides with human are the ones where the disagreement is arguably about the label rather than the model. Texts are shown exactly as published, as inert text — some are spam and contain links; none are clickable.

Across all 200 texts, the AI and human labels disagree on 11. Human labels are mapped spam → AD and ham → NOT_AD, which is a proxy: a subscribe-to-my-channel comment is spam without obviously being an ad.

Texts were sampled from 8 strata chosen before labelling. These are those strata. 70 of 200 texts were later re-filed by the adjudicator; the artifact keeps that grouping too, but it ends up one label per group, which makes it uninformative.

n/a where a slice holds no texts of that class. Groups are small; a single text moves a slice’s accuracy by several points.

Deterministic 200-text subset from UCI SMS Spam Collection and YouTube Spam Collection. Two separate source-blind AI annotators assigned AD/NOT_AD under a written guideline; a third AI adjudicated disagreements. The frozen prompt asks one binary Choice per text through TypeSafe's live API. A 20-text schema smoke run preceded the full 200-text run. No baseline or independent holdout was evaluated.

• The labels were assigned and adjudicated by AI agents, not by human annotators; shared model biases may raise apparent agreement and accuracy.
• On 11 of 200 texts the AI label disagrees with the source corpus's original human spam/ham label, and in 5 of Jev's 12 counted errors Jev agrees with the human label. Accuracy ranges from 93.5% to 96.5% depending on which labels are trusted.
• The sample is curated from two existing spam corpora and has 45.5% AD prevalence, which is not a deployment prevalence estimate.
• The source corpora were published in 2011 (SMS) and 2015 (YouTube); no text reflects current advertising formats.
• The 0.50–0.95 confidence sweep uses the same 200 examples as the reported accuracy and has no independent holdout validation.
• Latency is client-observed round-trip time; the runner region and network floor were not recorded.
• Cost is calculated from recorded input tokens and published list price, not reconciled to a provider invoice.