Benchmark archive
Split by who ran it, then grouped by task. Results from different datasets and protocols are never averaged, ranked against each other, or reduced to a single score.
Banking77 intent classification — complete test split
BANKING77 (complete official test split) · n = 3,080 · 2026-09-20 · JevBench
Ad Detector on SMS and YouTube comments (VE-001)
VE-001 AI-adjudicated subset of two UCI spam corpora · n = 200 · 2026-09-23 · JevBench
Chinese versus English virtual-assistant intent on MASSIVE (VE-002)
MASSIVE 1.1 test split, paired en-US and zh-CN · n = 2,974 · 2026-09-25 · JevBench · pre-registered
Next-step element selection on Mind2Web (VE-003)
Mind2Web test splits (cross-task, cross-website, cross-domain) · n = 9,378 · 2026-09-26 · JevBench · pre-registered
Page context and candidate count for next-step selection on Mind2Web (VE-004)
Mind2Web test splits (cross-task, cross-website, cross-domain) · n = 9,378 · 2026-09-26 · JevBench · pre-registered
Jev versus a free open model on MASSIVE intent (VE-005)
MASSIVE 1.1 test split, paired en-US and zh-CN · n = 2,974 · 2026-09-27 · JevBench · pre-registered
Jev against OpenDecider, Laya and open-jev on MASSIVE intent (VE-006)
MASSIVE 1.1 test split, paired en-US and zh-CN · n = 2,974 · 2026-10-02 · JevBench · pre-registered
An analysis re-reads decisions already recorded above and asks a different question of them. A repeat sends a run’s inputs again to check that its answers hold. Neither is counted as a separate run.
Can Jev know when it is wrong?
Selective prediction on the same 3,080 decisions: error-detection AUROC, the coverage available at a 95% quality bar, and the errors returned at maximum confidence that no threshold can reach.
Does Jev give the same answer twice?
RUN 003’s 200 texts sent twice more: 2 labels changed, and its “no errors kept at 0.70” held in 1 of 3 runs.
A comparison page exists only where a single run measured both models on the same data under the same protocol, and both sides have verified pricing. No head-to-head, no page.
Run by a third party with no vendor involvement. Code and dataset or protocol are public, so the result could be re-run. This does not mean anyone has re-run it — see `independentlyReproduced`, which nothing in the dataset sets yet.
Run by a third-party researcher or outlet, but reproduction material is partial or absent.
Published by the vendor of the model being measured. Recorded for completeness, never treated as independent evidence.
A single hands-on session, blog post or social media report. Directional only.
Not a tier and not a grade. These runs sit outside the A–D scale because that scale measures independence from the party with an interest in the result, and here that party is JevBench. Full artifacts are published so the figures can be recomputed rather than trusted.
Tiers grade provenance, not result strength. A small study can be tier A; its size is recorded in the run’s sample count and limitations, not in its tier. Tier A means the artifacts to re-run a benchmark are public — it does not mean anyone has re-run it, and nothing in this dataset has been independently reproduced.