CASE 0157 / CONF 1.00 / WRONG
“How can I check the exchange rate applied to my transaction?”
- EXPECTED
- card_payment_wrong_exchange_rate
- PREDICTED
- exchange_rate
Experimental record / run 002 / rev A
Can Jev know when it is wrong?
[ VERIFIED ]RUN 001 established that Jev answers 80.3% of Banking77 correctly. That number does not tell you whether a cascade is buildable. This record asks the question that does: when Jev is wrong, does it know? A model that is wrong one time in five and flags those cases is a different engineering proposition from one that is wrong one time in five and certain about it.
- DECISIONS AUDITED
- 3,080
- ERROR-DETECTION AUROC
- 0.853
- COVERAGE @ ≥95% ACC.
- 59.1%
- FALLBACK REQUIRED
- 40.9%
This experiment collected no new evidence
RUN 002 makes no new API calls and no new decisions. It re-reads the 3,080 decisions RUN 001 already published and asks a different question of them. Nothing here widens the evidence base, and the site’s run count does not increase: there is still one set of recorded Jev decisions, analysed twice. “Verified” means every figure on this page recomputes from that file — it does not mean an independent party reproduced it, and an analysis we performed on a run we executed is not independent evidence about our own reporting.
01 / The 95% quality operating point
Pick a confidence threshold, keep everything at or above it, send the rest to a fallback. Coverage is the share you keep; retained accuracy is how often the kept decisions are right. The question a buyer actually has is: to reach a quality bar, how much traffic must I pay a second model to handle?
To hold 95% retained accuracy, the best available threshold is 0.96: it keeps 1,820 of 3,080 decisions = 59.1% coverage at 95.44% accuracy. The other 40.9% must go somewhere else. Any cost model for a Jev cascade on this workload starts from that number, not from the headline price.
| THRESHOLD | COVERAGE | FALLBACK | RETAINED ACC. | WRONG KEPT | ERRORS CAUGHT |
|---|---|---|---|---|---|
| 0.50 | 95.75% | 4.25% | 82.67% | 511 | 15.95% |
| 0.55 | 92.86% | 7.14% | 83.88% | 461 | 24.18% |
| 0.60 | 89.87% | 10.13% | 85.77% | 394 | 35.20% |
| 0.65 | 87.05% | 12.95% | 87.02% | 348 | 42.76% |
| 0.70 | 83.70% | 16.30% | 88.29% | 302 | 50.33% |
| 0.75 | 80.91% | 19.09% | 89.25% | 268 | 55.92% |
| 0.80 | 78.05% | 21.95% | 90.10% | 238 | 60.86% |
| 0.85 | 74.90% | 25.10% | 90.98% | 208 | 65.79% |
| 0.90 | 69.29% | 30.71% | 92.60% | 158 | 74.01% |
| 0.95 | 61.56% | 38.44% | 94.78% | 99 | 83.72% |
| 0.96 | 59.09% | 40.91% | 95.44% | 83 | 86.35% |
HIGHLIGHTED ROW = THE 95% OPERATING POINT. IT IS NOT ON THE 0.05 GRID: A COARSE SWEEP WALKS PAST IT.
02 / Risk–coverage curve
The whole trade-off in one line: how much error you still carry at every level of coverage. Flat is good — it means you can keep traffic without taking on risk. The steep right-hand end is where the confidence score stops separating anything.
03 / Low-confidence error capture
The review-queue question. If a human could only re-check the least confident slice of a day’s traffic, how much of the day’s error would they find?
REVIEW LEAST-CONFIDENT 10%
35.2%
OF ALL ERRORS FOUND
- CUTOFF
- ≤ 0.59
- REVIEWED
- 312 (10.13%)
- ERRORS
- 214 / 608
REVIEW LEAST-CONFIDENT 20%
57.4%
OF ALL ERRORS FOUND
- CUTOFF
- ≤ 0.76
- REVIEWED
- 624 (20.26%)
- ERRORS
- 349 / 608
REVIEW LEAST-CONFIDENT 30%
74.0%
OF ALL ERRORS FOUND
- CUTOFF
- ≤ 0.89
- REVIEWED
- 946 (30.71%)
- ERRORS
- 450 / 608
A tie group is always taken whole, so the reviewed share overshoots the target slightly — splitting rows that share a confidence value would be arbitrary. Reading the top row: reviewing one decision in ten surfaces 35.2% of the errors, which is 3.5× better than reviewing a random tenth.
04 / The errors no threshold can reach
29 decisions came back wrong at confidence exactly 1.00.
This is the single most important number on the page, and it is not visible in any accuracy figure. 1,192 decisions (38.7% of the run) sit at maximum confidence. A threshold cannot separate anything inside a tie group, so these 29 errors — 4.8% of every error in the run — are unreachable by confidence routing at any setting. Raise the threshold to 1.00 and you discard 61.3% of your traffic and still ship all 29.
That is the ceiling on this whole approach. Everything above describes a usable filter; this describes the floor of error it cannot get under.
CASE 0378 / CONF 1.00 / WRONG
“My card was declined today when eating and I need to know what's wrong.”
- EXPECTED
- card_not_working
- PREDICTED
- declined_card_payment
CASE 0538 / CONF 1.00 / WRONG
“I attempted to use my card while I was intoxicated, and I failed to input my PIN, and the machine kept my card. How soon can I have it back?”
- EXPECTED
- pin_blocked
- PREDICTED
- card_swallowed
05 / Calibration diagnostics
ECE / 10 EQUAL-WIDTH BINS
0.0861
BRIER (BINARY CORRECTNESS)
0.1231
| BIN | N | MEAN CONF. | ACCURACY | GAP |
|---|---|---|---|---|
| 0.0–0.1 | 0 | — | — | — |
| 0.1–0.2 | 1 | 0.1500 | 0.00% | 0.1500 |
| 0.2–0.3 | 10 | 0.2560 | 10.00% | 0.1560 |
| 0.3–0.4 | 24 | 0.3613 | 25.00% | 0.1113 |
| 0.4–0.5 | 96 | 0.4575 | 28.13% | 0.1763 |
| 0.5–0.6 | 181 | 0.5467 | 35.36% | 0.1931 |
| 0.6–0.7 | 190 | 0.6466 | 51.58% | 0.1308 |
| 0.7–0.8 | 174 | 0.7466 | 63.22% | 0.1144 |
| 0.8–0.9 | 270 | 0.8550 | 70.37% | 0.1513 |
| 0.9–1.0 ] | 2,134 | 0.9838 | 92.60% | 0.0578 |
What these two numbers are not
TypeSafe documents confidence as a summary of how concentrated the Choice distribution is, not as a probability of being correct. ECE and Brier only mean anything against a probability, so read them as a description of how this score happens to behave against correctness in this run — not as evidence that it failed at a job it was never given. The useful finding is directional and survives that caveat: the gap is positive in every populated bin, so the score runs ahead of the outcome everywhere, and it is worst in the middle of the range where a router would most want to trust it.
06 / Method
- SOURCE
artifacts/banking77/run-jev-1.13.0.jsonl- SHA-256
aae26e0a2c20221e1d56bd0def45dbac1b5b423c2dd31b941761dbbad6f3bd84- BYTES
- 7,221,986
- NEW API CALLS
- 0
- DECISIONS
- 3,080 — 2,472 correct, 608 wrong
- ERROR SCORE
1 − confidence, scored against the recordedcorrectflag- AUROC
- Mann–Whitney U with midranks. Ties are material here: 1,192 rows share the value 1.00, and treating that block as anything but tied would inflate the figure.
- CALIBRATION
- Ten equal-width bins; the top bin includes 1.00 so no row is dropped. ECE is the count-weighted mean absolute gap.
- MISSING VALUES
- Reported as missing. Never imputed.
07 / Limitations
- Every threshold here is selected on the same rows it is measured on. 59.1% coverage at 95% accuracy is the best point available in hindsight on this dataset, not a forecast. On unseen traffic it will be worse. There is no held-out split in this analysis, and a threshold tuned this way is the most over-optimistic figure on the page.
- Confidence is the provider’s own reported value, not the maximum of the published probability vector. The two differ on roughly 44% of rows because the stored probabilities are rounded to two decimals.
- Rounding to two decimals also caps threshold granularity. The 1.00 tie group alone is 38.7% of the run, so fine-grained operating points near the top simply do not exist in this data.
- This is one 77-way banking intent task on one model version. Jev’s confidence behaviour on a different task is a separate empirical question — JevBench’s collected evidence already records datasets where routing on Jev’s confidence found no usable threshold at all.
- A fallback model is assumed to be better on the escalated rows. No fallback was run, so the end-to-end accuracy of an actual cascade is not measured anywhere on this page.
- JevBench executed the run this analyses and performed the analysis. It is first-party throughout and carries no evidence tier.
08 / Reproducibility
Every figure on this page is a deterministic function of the 3,080-row result log published with RUN 001. The site does not read a precomputed number from anywhere: it recomputes the sweep, the curve, the capture table and the calibration bins at build time, and the test suite asserts those values against the published threshold-sweep.csv artifact to twelve decimal places. If the page and the artifact ever disagree, the build fails.
Source run executed 2026-09-20. Every failure is preserved. The evidence is the record, not the headline number.