How evidence is handled.
The document our credibility rests on: sourcing, execution, scoring, latency, cost, failure handling, reproducibility and limitations.
- DOCUMENT
- JB-TN-001
- REVISION
- A
- STATUS
- [ CURRENT ]
Every number needs a source.
A figure without a source URL and a verification date does not go on the site. The build fails on one â this is a test, not an intention.
Benchmarks stay attached to their task.
A metric only exists inside a run, and a run always carries its dataset, protocol and date. There is nowhere in the schema to store a model-level score, so a context-free accuracy claim is not something this site can express.
Whoever ran it stays labeled â including us.
Vendor-published results are recorded as tier C and displayed as Vendor-reported. Runs JevBench executed itself carry no evidence tier at all, because the A-D scale grades independence from the party with an interest in the result and on our own runs that party is us. Neither is ever silently mixed into a table beside independent results.
Missing data remains missing.
No estimates, no house defaults, no plausible-looking fills. Ask the cost model for a cached-input discount on a model with no published cache rate and it returns an error rather than a number.
Losses are evidence too.
We evaluate one vendor's model, which is a structural conflict of interest. The only honest answer is to publish where Jev loses as prominently as where it wins, and to make every figure checkable against its original.
A benchmark run is one dataset, one protocol, one date. Every number lives inside a run and travels with it. Across the evidence collected so far, Jev’s recorded accuracy spans 38.4% to 95.4%. A single headline figure would be wrong for every task it was quoted about.
Where a study covered two datasets we split it into two runs. Averaging across datasets destroys the only context that makes the numbers mean anything.
Most of what is here we did not produce: we collect runs other people published and normalize them. Since 2026-09-20 we also run a small number of benchmarks ourselves, starting with RUN 001 â the complete 3,080-row BANKING77 test split against Jev 1.13.0, published with its pinned dataset commit, its protocol, the runner that produced it and the full per-row result log.
RUN 001 was executed against TypeSafe’s public API on a self-funded key. It cost $0.22 and involved no contact with the vendor: no credits, no review, no advance notice.
Each collected run is graded on how we know it, not on how strong the result is. A small study can be tier A; its size is recorded in the run’s sample count and limitations, not in its tier. The scale applies to evidence we collected â runs we executed ourselves sit outside it entirely, for the reason above.
Run by a third party with no vendor involvement. Code and dataset or protocol are public, so the result could be re-run. This does not mean anyone has re-run it — see `independentlyReproduced`, which nothing in the dataset sets yet.
Run by a third-party researcher or outlet, but reproduction material is partial or absent.
Published by the vendor of the model being measured. Recorded for completeness, never treated as independent evidence.
A single hands-on session, blog post or social media report. Directional only.
Every run carries a non-empty limitations list and a statement of what it does not prove. A run with neither has not been read carefully enough to publish. Two examples of why this matters:
Latency is partly geography
In the phishing benchmark, measured from France, Jev’s p50 was 239 ms against Claude Haiku 4.5’s 687 ms â a 2.9x ratio. The author also recorded the network floor inside each figure: 163 ms and 18 ms. We report the measured figures and the floors, and label anything we subtract as derived. The network-floor-adjusted residual, roughly 76 ms and 669 ms, is not model compute time and we never call it that: it still contains queueing, gateway hops, TLS negotiation, serialisation and provider-side overhead. What it does show is that the ranking is not an artifact of where the client sat, while the 2.9x ratio transfers nowhere.
A pre-registered result that came back ambiguous is a result
The Banking77 and CLINC150 study pre-registered its kill/go criteria and hash-pinned them before running. Its primary outcome, the cascade escalation rate, was +0.265 with a 95% interval of [â0.530, +0.595] â spanning zero. Verdict: AMBIGUOUS. We record that verdict at the top of the run rather than quietly reporting the accuracy ordering underneath as though it were decided.
Three scenarios are priced for a workload: B, one baseline call per request; J, one Jev call per request; and H(f), a cascade of one Jev call plus a fallback call on a fraction f of requests. Solving H(f) = B gives the headline:
Four decisions change that answer, and each is a place a cost comparison usually goes wrong. Output tokens are priced on both sides â Jev bills output at $0.00/MTok and a generative baseline does not, so counting only input tokens understates the baseline by roughly an order of magnitude on a classification workload. Jev’s input tokens are independent of the baseline’s, because a typed decision request and a generative prompt are not the same length. The fallback call is modelled separately, since an escalation may carry a different prompt or go to a different model; when F does equal B the formula reduces to f* = 1 â J/B, which we let fall out of the arithmetic rather than baking in. And cache and batch rates are provider-specific, never a house default.
Prices are recorded per model and per provider, since gateway and first-party pricing differ, each with a source URL, the date a maintainer last confirmed it, an optional validity window and a flag for promotional rates. Prices older than 30 days are flagged stale.
| Model | Input | Output | Cache hit | Batch | Verified |
|---|---|---|---|---|---|
| Jev | $0.042/MTok | $0.00/MTok | not published | not published | OpenRouter, 2026-09-19 |
| Claude Haiku 4.5 | $1.00/MTok | $5.00/MTok | $0.1/MTok | 50% | Anthropic (first-party API), 2026-09-19 |
| GPT-5.4 nano | $0.2/MTok | $1.25/MTok | $0.02/MTok | 50% | OpenAI (first-party API), 2026-10-14 |
| GPT-5.6 Terra | $2.00/MTok | $12.00/MTok | $0.2/MTok | 50% | OpenAI (first-party API), 2026-10-14 |
| DeepSeek v4 Pro | No verified pricing. No price confirmed against DeepSeek's own pricing page. The routing run recorded its own measured spend, which stays on that run rather than being reused as a rate. | ||||
| Jev → DeepSeek v4 Pro cascade | No verified pricing. A cascade's cost depends on its escalation rate and on both models' rates, which is what the cost model on this site computes rather than something a provider publishes. | ||||
| TF-IDF + logistic regression | No verified pricing. Self-hosted classical model: cost is compute and engineering time, not per-token billing. Needs a separate cost model before it can appear in the calculator. | ||||
| Jev + TF-IDF ensemble | No verified pricing. An ensemble's cost is the sum of its parts plus orchestration, which depends on how it is deployed. Not a single rate that can be verified against a provider page. | ||||
| Qwen 3.8 27B | No verified pricing. No price confirmed against Cerebras's own pricing page. The workload-suite run recorded its own measured spend, which stays on that run rather than being reused as a rate. | ||||
| Needle 3 | No verified pricing. Run locally rather than billed per token. Its cost is hardware and electricity, which the source explicitly excluded. | ||||
| BGE-small-en-v1.5 + logistic regression | No verified pricing. Self-hosted: cost is compute and engineering time, not per-token billing. Needs a separate cost model before it can appear in the calculator. | ||||
The cost model’s type signature accepts only verified pricing, so an unverified model cannot reach the arithmetic. Batch mode is hidden entirely until both sides of a comparison publish a discount â discounting one side while the other pays list price is worse than not offering the option.
Gateway listing, not TypeSafe's own page. Gateway pricing can differ from first-party API pricing; recorded as observed at OpenRouter. Needs re-verification against TypeSafe's own pricing page when one is confirmed.
First-party pricing page. Vendor-affiliated for Anthropic's own models, which is the normal and correct source for a list price — the vendorAffiliated flag matters for benchmark claims, not for a vendor quoting its own rate card.
Launch coverage. Relays TypeSafe's own speed and cost claims; used here only for descriptive facts about what Jev is, not for performance figures.
Public code and stated methodology. Repository does not state author affiliation or who funded API costs; recorded as unstated rather than assumed independent-funded.
Pre-registration is hash-pinned in the repository. Author states no TypeSafe affiliation, no vendor involvement, no free credits, and that API costs (~$1) were paid by the author.
Compares two Jev variants against each other. Contains no non-Jev baseline; author states no provider key was available at run time.
Public code and stated protocol. The repository does not state a run date, the author's affiliation, or who funded API costs; it does report measured spend of about $1.35 for the full paired run.
Author states 'This is an independent benchmark, not a TypeSafe implementation or parity claim' and warns the timing figures are 'not a controlled speed ranking'. Affiliation and funding are not disclosed.
Not evidence itself. Used as a discovery feed for independent benchmarks; each item it links is verified separately before anything is recorded.
A routing library rather than a benchmark. Its measured results are the tool's own demo output on two public datasets, so the tool has an interest in routing working — it reported that it does not on one of the two, which cuts against that interest. No run date, calibration metrics or intervals are stated.
Six pre-registered gate conditions with stated thresholds, run against an easy corpus and a deliberately hard probe. Author states API costs were paid by the author and used TypeSafe's published $0.042/MTok rate for cost estimates.
Checked specifically for whether it independently reproduces any run recorded here. It does not. The project states it 'reproduces that interface pattern with open models; it does not reproduce Jev's undisclosed model or training', measures open models (Qwen3-0.6B, MiniCPM5-2B, Qwen3.5-4B) rather than Jev, and re-runs none of our recorded datasets. Its only Jev comparison uses TypeSafe's own published figures on 102 aligned rows, and it notes it could not align the vendor's 711-row aggregate. Recorded as a source so the question 'has anything been reproduced?' has a checkable answer; no run is recorded from it.
First-party pricing page. Confirmed 2026-10-14 (Standard tier, short context): gpt-5.4-nano $0.20/$1.25 and gpt-5.6-terra $2.00/$12.00 per MTok input/output; cached input $0.02/$0.20; batch 50% off.
Self-published. JevBench executed this run against TypeSafe's live API using a self-funded API key: no credits, no review, no vendor contact. The complete 3,080-row result log is published alongside the summary so the accuracy, latency and cost figures can be recomputed from the raw rows.
Frozen dataset, annotation audit, per-row results, protocol, evaluator output, source-label comparison, source citations, and checksums. Source text is from two UCI corpora under CC BY 4.0.
Pre-registered protocol, frozen paired MASSIVE 1.1 en-US/zh-CN test split, per-request results for three arms, evaluator output and checksums. MASSIVE is CC BY 4.0 (Amazon).
Pre-registered protocol, per-step ids, choices and scores for all 9,378 Mind2Web test steps, evaluator output, dataset manifest and checksums. Mind2Web's test data itself is not republished, at its authors' request.
Pre-registered protocol, per-request ids, choices and scores for all 9,378 Mind2Web test steps under four arms, evaluator output, dataset manifest and checksums. Mind2Web's test data itself is not republished, at its authors' request.
Pre-registered protocol, per-item predictions of BAAI/bge-m3 (zero-shot and trained) on MASSIVE's 2,974 parallel test items in English and Chinese, metrics against Jev's RUN 005 rows, pinned model revision and checksums.
Pre-registered protocol with addendum, per-item answers of OpenDecider-nano, Laya, Laya-multilingual and open-jev on MASSIVE's 2,974 test commands, metrics against Jev's RUN 005 rows with and without 41 items found in training text, one-at-a-time CPU timing, pinned model revisions and checksums.
If a figure here is wrong, it is a bug and we want it reported. Every number carries its source so it can be checked against the original. When we correct something we say what changed and when.