JevBench
COST RECORD / ENGINEERING CALCULATION SHEET

What did the run cost?

RUN 001 records usage, then derives an API-cost estimate from the list price observed on the run date. The estimator below models other workloads without treating that estimate as an invoice.

RUN 001 / COST RECORD
REQUESTS
3,080
INPUT TOKENS
5,296,450
OUTPUT TOKENS
2,551,021
EST. API COST
$0.222
COST / REQUEST
$0.000072
PRICE OBSERVED
2026-09-20

Derived from recorded usage at $0.042/million input tokens. Output price was recorded as $0.00/million. Provider billing may differ.

Workload estimator

Your workload

These are placeholders describing a plausible classification workload. Replace them with your own measurements — nothing here is a finding.

Break-even fallback rate
99.8%

The Jev cascade stays cheaper until 99.8% of requests fall back to Claude Haiku 4.5. Above that, running everything on the baseline costs less.

Based on your token usage and the verified provider pricing listed below. f* = (B − J) / F.

Baseline onlyB
$5,050
$0.0051 / request
Jev onlyJ
$12.60
$0.000013 / request
Hybrid at 0%J + f·F
$12.60
$0.000013 / request
Hybrid vs baseline at your stated fallback rate: $5,037/month less (99.8% cheaper)
Assumptions
Requests / month1,000,000
Jev call300 in / 0 out
Baseline call500 in / 910 out
Fallback call500 in / 910 out → Claude Haiku 4.5
Fallback rate0.0%
Batch pricingnot applied

Pricing used

Jev$0.042/MTok in · $0.00/MTok out
OpenRouter · verified 2026-09-19 · OpenRouter
Claude Haiku 4.5$1.00/MTok in · $5.00/MTok out · $0.1/MTok cache hit
Anthropic (first-party API) · verified 2026-09-19 · Anthropic

Observed escalation rates

These are measurements from specific runs, shown for scale. They are not defaults and none of them is “Jev’s typical fallback rate”.

40.9%Jev → any fallbackBanking77, complete test split, n=3,080 · 95% retained accuracy→

Measured on the full test split rather than a sample, which makes it the best-supported rate here — but the threshold was chosen on the same rows it is scored on, so it is an upper bound, not a forecast. No fallback model was actually run, so the end-to-end accuracy of the cascade is unmeasured.

22.0%Jev → GPT-5.6 TerraBanking77, n=208 · 90.5% accuracy

Specific to this dataset, cascade and accuracy target. The run's pre-registered verdict was AMBIGUOUS: the difference against the comparison cascade was +0.265 with a 95% CI of [-0.530, +0.595].

48.5%GPT-5.4 nano → GPT-5.6 TerraBanking77, n=208 · 90.5% accuracy

The comparison cascade from the same run, under the same caveat.

11.6%Jev → DeepSeek v4 ProBanking77, n=500 · a confidence threshold of 0.67

The threshold was fitted on this same data, so the rate is optimistic by an unknown amount, and no single-model baseline was measured. The same pipeline found no workable threshold at all on a second dataset.

Before you trust a fallback rate

Can you route on Jev’s confidence?

Every fallback rate above assumes you can tell in advance which requests Jev will get wrong. The recorded evidence splits cleanly, and it splits by how hard the judgment is — not by anything you can read off a spec sheet.

ECE 0.045holds up

Topic membership on 20 Newsgroups. All six pre-registered gates passed.

ECE 0.242breaks down

Graded product relevance on Amazon ESCI. Same author, same gates, same model version — four of six failed.

0 confident missesholds up

Agent tool-call risk. Every incorrect answer came with hedged confidence; the model never returned 1.000 and was wrong.

0.853 → 0.734breaks down

Error-ranking AUROC from Banking77 to CLINC150. On CLINC150 Jev ranks its own errors worse than the nano-class baseline does.

threshold 0.67holds up

A routing pipeline found a workable threshold on Banking77, escalating 11.6% of rows — though it was fitted on that same data.

do not routebreaks down

The same pipeline on Web of Science: no threshold beat the better single model.

On one workload we measured it
RUN 002 audited all 3,080 decisions of our own Banking77 run for exactly this question. Holding 95% retained accuracy needs 40.9% of traffic to escalate — and 29 of the errors come back at confidence 1.00, where no threshold can reach them at all. That is the shape of the answer you should expect on your own task: a usable filter with a floor under it.
What this means for the number above
On easy, well-separated discriminations Jev’s confidence looks usable as an escalation trigger. On graded or subjective judgments it does not, and on one dataset a routing pipeline could find no threshold worth using at all. So a fallback rate is not a property of Jev you can look up — it is something you measure on your own task, which is why this page takes it as an input and never fills it in for you.

How this is calculated

Three scenarios are priced for the same workload:

  • B — baseline only: one call per request to the model you use today.
  • J — Jev only: one Jev call per request.
  • H(f) — hybrid cascade: one Jev call per request, plus one fallback call on a fraction f of requests. H(f) = J + f·F.

Setting H(f) = B and solving gives the break-even fallback rate f* = (B − J) / F. Below it the cascade is cheaper; above it the baseline is.

Four decisions that change the answer

Output tokens are priced on both sides. Jev bills output at $0.00/MTok; a generative baseline does not, and output typically costs several times input. A calculator that asks only for “average input tokens” understates the baseline — usually by an order of magnitude on this shape of workload — and inflates the apparent saving.

Jev’s input tokens are independent of the baseline’s. A typed decision request and a generative prompt with a system prompt, schema and examples are not the same length.

The fallback call is modelled separately. F is not assumed equal to B — an escalation call may carry Jev’s output, a different prompt, or go to a different model. When F does equal B, the formula reduces to f* = 1 − J/B; we let that fall out of the arithmetic rather than baking it in.

Cache and batch rates are provider-specific. We never apply a house discount. Ask for a cached-input share on a model with no published cache price and the model returns an error instead of a number, because that one assumption moves the result more than anything else on the page — in the direction that flatters Jev.

What the arithmetic tends to show
Against a generative baseline on a classification workload, break-even usually lands above 90%. That is worth stating plainly: the cost question is often already settled before you start, and the binding constraint is accuracy. Which is why the benchmark evidence matters more than this page does.

Full reasoning and the pricing record format are in the methodology.