Experimental records
Everything JevBench has run or analysed, in order. A run collects decisions from a model. An analysis re-reads decisions already collected and asks something else of them, so it never increases the evidence base. A repeat sends a run’s inputs again to check that its answers hold.
Banking77 × Jev — complete test split
The complete BANKING77 test split sent to Jev 1.13.0 as typed Choice decisions, published with its pinned dataset commit, its protocol, the runner, and every per-row result.
- Decisions
- 3,080
- Accuracy
- 80.3%
- Failures kept
- 608
- Run date
- 2026-09-20
Can Jev know when it is wrong?
An offline audit of those same decisions: does Jev's confidence flag its own errors? It makes no new API calls and collects no new evidence.
- Decisions audited
- 3,080
- Error AUROC
- 0.853
- Coverage @ 95%
- 59.1%
- Wrong at conf 1.00
- 29
Can Jev tell an ad from everything else?
200 short texts from two public UCI spam corpora, labelled AD or NOT_AD by AI annotators, sent to Jev as a binary Choice. Published with every text, both label sets, and the original human spam/ham labels to check them against.
- Texts
- 200
- Accuracy (AI labels)
- 94.0%
- Errors siding with humans
- 5 of 12
- Run date
- 2026-09-23
Does Jev give the same answer twice?
RUN 003's frozen texts and request, sent to Jev twice more under a protocol committed in advance, to test whether a single run's figures hold when the same question is asked again.
- Texts × runs
- 200 × 3
- Labels changed
- 2 / 200
- No error kept at 0.70
- 1 of 3 runs
- Last run
- 2026-09-24
Does Jev understand Chinese?
Amazon's MASSIVE test split, the same virtual-assistant commands in English and in professionally localized Chinese with human labels, sent to Jev under a pre-registered protocol: does it lose accuracy when the user writes in Chinese, and does writing the question in Chinese help?
- English input
- 81.7%
- Chinese input
- 79.3%
- Difference
- -2.4 pp
- Items × arms
- 2,974 × 3
Can Jev pick the next click?
Every step of Mind2Web's 1,341 human-demonstrated web tasks, posed to Jev the way a browser agent uses it: pick the next element to act on from the page's top 50 candidates. Pre-registered, and scored per step and per whole task.
- Steps
- 9,378
- Right element
- 38.4%
- Right step
- 27.3%
- Tasks fully right
- 29 / 1341
Does more page context help Jev?
RUN 006's steps sent again under four pre-registered arms — 50 or 10 candidates, each with or without where it sits on the page — to test a public critique that Jev had been given too little context. The action is now read off the chosen element.
- Without context
- 38.4%
- With context
- 42.3%
- Tasks fully right
- 63 / 1341
- Steps × arms
- 9,378 × 4
Is Jev better than a free model?
The first comparison: the free, open BGE-M3 model on the same MASSIVE commands Jev answered in RUN 005, in English and Chinese — once given only what Jev was given, once trained on MASSIVE's labelled examples. Jev's answers are reused; no new requests.
- Jev, no examples
- 81.7%
- Free model, no examples
- 48.3%
- Free model, trained
- 88.1%
- New Jev requests
- 0
Jev vs OpenDecider, Laya and open-jev
Jev against three open decision models, OpenDecider-nano, Laya and open-jev, on the same MASSIVE commands with the identical question and option descriptions and no examples: accuracy, calibration and CPU speed. Pre-registered; Jev's answers reused from RUN 005.
- Jev
- 81.7%
- OpenDecider-nano
- 54.4%
- open-jev
- 52.9%
- Laya
- 37.3%
Supporting pages: the 608-case failure archive · the RUN 001 replay · how evidence is handled