CASE 1039 / WRONG
“On my last transaction it seem that my top-up was not successful.”
- EXPECTED
- top_up_reverted
- PREDICTED
- top_up_failed
- CONFIDENCE
- 1.00
- LATENCY
- 326 ms
Experimental record / run 001 / rev A
Banking77 × Jev
[ VERIFIED ]Complete official test split, sent as zero-shot 77-way typed Choice decisions to jev-1.13.0. “Verified” means the run completed and its public manifest, protocol and summary agree; it does not mean an independent party reproduced it.
Banking77 benchmark summary
- Model
- Jev 1.13.0
- Dataset
- Banking77 / official test split
- Samples
- 3,080
- Accuracy
- 80.3%
- P50 latency (client-observed)
- 310 ms
- P95 latency (client-observed)
- 896 ms
- Estimated API cost (USD)
- $0.22245
- Conducted by
- JEVBench
- Run date (UTC)
- 2026-09-20
Scope: zero-shot 77-way intent classification. Latency includes network and service overhead; cost is estimated, not invoiced. This run does not establish general model quality or independent reproduction.
01 / Experimental setup
- DATASET
- BANKING77 official test split / CC-BY-4.0
- DATASET COMMIT
57ec275d8078af65b7731c2a98be812d844a6d6b- SAMPLES / LABELS
- 3,080 / 77
- MODEL
- jev-1.13.0
- TASK
- zero-shot 77-way intent classification
- RUN DATE
- 2026-09-20
- PROMPT PROTOCOL
- One Choice question; label descriptions normalized only from dataset label names. No examples, tuning or fallback.
02 / Results
How accurate is Jev 1.13.0 on Banking77?
Jev 1.13.0 achieved 80.3% accuracy on the official Banking77 test split: 2,472 correct predictions out of 3,080 cases under this zero-shot protocol.
2,472
correct
608
wrong
19.7%
error rate
How much did the Banking77 run cost?
The estimated API cost for this run was $0.22245 USD.
Cost is an estimate derived from 5,296,450 recorded input tokens at the $0.042/million list price observed 2026-09-20. Latency is client-observed request time and includes network and service overhead; it is not model compute time.
Follow-up / run 002
Can Jev know when it is wrong?
An accuracy figure does not tell you whether a cascade is buildable. RUN 002 audits these same decisions for whether Jev’s confidence flags its own errors — and finds 29 of them returned at maximum confidence, where no threshold can reach them.
03 / Latency distribution
How fast is Jev 1.13.0?
In this Banking77 run, client-observed P50 latency was 310 ms and P95 was 896 ms, including network and service overhead. These are not model compute times.
<250
63
250–349
2241
350–499
423
500–749
156
750–999
150
≥1000
47
BANDS IN MILLISECONDS / COUNTS AT RIGHT
04 / Confidence vs correctness
0.0–0.1
—
0.1–0.2
0.0%
0.2–0.3
10.0%
0.3–0.4
25.0%
0.4–0.5
28.1%
0.5–0.6
35.4%
0.6–0.7
51.6%
0.7–0.8
63.2%
0.8–0.9
70.4%
0.9–1.0
92.6%
BAR = SAMPLE COUNT / RIGHT = OBSERVED ACCURACY
05 / Failure archive
Where does Jev fail?
Jev 1.13.0 made 608 incorrect predictions in this Banking77 run. The cases below show actual intent-label mismatches; the complete archive preserves every failure.
CASE 2545 / WRONG
“My card is just not working at this time.”
- EXPECTED
- virtual_card_not_working
- PREDICTED
- card_not_working
- CONFIDENCE
- 1.00
- LATENCY
- 290 ms
CASE 1935 / WRONG
“My ATM got stuck and I'm not sure what to do.”
- EXPECTED
- card_swallowed
- PREDICTED
- atm_support
- CONFIDENCE
- 1.00
- LATENCY
- 262 ms
06 / Against the collected evidence
JevBench already held a third-party run on this dataset: Banking77 intent classification: Jev vs nano, frontier and a supervised encoder, a 208-row paired sample scoring Jev at 83.2% (95% CI 77.9–88.0%). This run puts the full split at 80.3% (95% CI 78.8–81.6%, Wilson).
The two agree: our full-split figure falls inside the smaller run’s confidence interval. That is the useful result here — not that Jev scored 80.3%, but that an independent 208-row estimate held up when someone ran all 3,080 rows.
This run has no baseline, and that is its biggest weakness. On the same dataset, the third-party run scored BGE-small-en-v1.5 + logistic regression at 93.3% — well above Jev. A number measured against the dataset’s labels says nothing about whether Jev is the right choice for the task.
07 / Replay
The replay is generated from the published result manifest. It is accelerated for review; it does not replay wall-clock timing.
[ WATCH FULL RUN ]08 / Limitations
- No baseline was run. This measures Jev against the dataset's labels, not against any alternative model, so it says nothing about whether Jev is the right choice for this task.
- Latency is client-observed round-trip time with no network floor measured, so it is not model compute time and cannot be compared against a latency figure from any other run.
- The runner issues requests concurrently and the artifacts do not record the concurrency used for this run, which further rules out comparing these latencies with a serially measured figure.
- Cost is derived from recorded input tokens at a list price, not from an invoice. The provider returned no per-request cost on any of the 3,080 calls.
- The 77 label descriptions are produced mechanically from the dataset's label names. Different criteria, worked examples or task decomposition would likely change the result.
- The provider returned no request identifier on any call, so an individual published row cannot be traced back to a provider-side record.
- Three rows record a prediction that is not the argmax of their own published probability vector; in all three the top two labels are exactly tied at two-decimal rounding (0.35/0.35, 0.24/0.24, 0.50/0.50).
09 / What this run does not prove
What does this benchmark not prove?
This run does not establish general reasoning, coding, generation or production quality. Its completed artifacts do not demonstrate independent reproduction.
- It does not show that Jev is accurate in general. This is one 77-way banking intent task, and JevBench's own collected evidence puts Jev's accuracy anywhere from 62.6% to 98.6% depending on the task and the prompt.
- It does not show that Jev beats any alternative on this task, because no alternative was run beside it. A third-party run on a 208-row sample of this same split scored a supervised encoder baseline well above Jev.
- It does not establish that Jev's confidence scores are safe to route on outside this dataset. The threshold table is fitted on the same rows it is measured on.
- It is not an independent result. JevBench chose the dataset, wrote the prompt and published the write-up, and a run we executed ourselves cannot corroborate our own reporting.
- It does not measure the latency you would see. No network floor was measured and the client's region is not pinned in the artifacts.
Every failure is preserved. The evidence is the record, not the headline number.
READ TECHNICAL NOTE 001 →