EVIDENCE STATUS / [ VERIFIED ]

Jev vs GPT-5.4 nano

One pre-registered head-to-head exists: 208 Banking77 utterances, both models, same protocol, 2026-09-18. The run’s own verdict was AMBIGUOUS — the accuracy ordering below is descriptive, not a decided result.

What this run does not show

A supervised encoder (BGE-small embeddings with a logistic regression head, trained on Banking77’s own labels) scored 93.3% in the same run — the highest accuracy and lowest latency of anything measured. It is not a zero-shot system, so it is not compared here; it is the option you already have when the label set is fixed and training data exists. A second Jev-vs-nano head-to-head on CLINC150 is noted below.

Jev leads on latency and cost

Latency p50	440 ms vs 850 ms
Latency p95	710 ms vs 1,520 ms
List input price	$0.042/MTok vs $0.2/MTok
List output price	$0.00/MTok vs $1.25/MTok

Accuracy: too close to call

Accuracy	83.2% [77.9%, 88.0%] vs 79.3% [73.6%, 85.1%]
Run verdict	AMBIGUOUS (pre-registered)
Sample	n=208, cut from 300 by rate limiting

The number that matters for a cascade

To reach 90.5% accuracy by escalating hard cases to GPT-5.6 Terra, Jev needed a 22.0% fallback rate; nano needed 48.5% — more than double, measured on the same 208 utterances. Caveat: the run’s pre-registered comparison of the two cascades was itself ambiguous (95% CI [-0.530, +0.595] spans zero), so treat the gap as observed, not proven.

On the accuracy figures

The 3.9-point gap between Jev and nano sits inside heavily overlapping confidence intervals at n=208. The run’s own “does not prove” list explicitly includes the claim that Jev beats a nano-class model on classification — do not read the ordering as a win.

Both generative baselines were JSON-prompted with outputs parsed by regex rather than schema-constrained decoding, and prompts were not tuned per model. That is a fair protocol for a pre-registered run, but it can disadvantage the generative side relative to a production integration built around structured outputs.

On the latency figures

Calls were issued serially from the same client, so the ordering is comparable: Jev’s p50 of 440 ms against 850 ms for GPT-5.4 nano. The run recorded no network floor or measurement origin, so there is no residual analysis here — treat the roughly 1.9x ratio as ordering evidence from one location, not a portable constant.

Cost: what is verified and what is not

Jev, list price

$0.042/MTok / $0.00/MTok

Per MTok input / output. OpenRouter, verified 2026-09-19.

GPT-5.4 nano, list price

$0.2/MTok / $1.25/MTok

Per MTok input / output. OpenAI, verified 2026-10-14.

Measured, this run

$0.000071 / call

Jev only (measured, 77 labels in the prompt). The run's client did not return LLM token usage, so GPT-5.4 nano has no measured cost.

No monthly cost is modelled here: the run did not record token counts per call, and inventing a workload shape would manufacture the answer. Model your own workload with the verified list prices above.

A second head-to-head

CLINC150 intent routing (n=200, 2026-09-18, pre-registered verdict also AMBIGUOUS): Jev 87.0% vs GPT-5.4 nano 79.5%, latency p50 423 ms vs 924 ms — the same directional pattern, weaker evidence (no confidence intervals reported, latency on a 30-item subset).

The run itself

Banking77 intent classification: Jev vs nano, frontier and a supervised encoder

AIndependent · Reproduction artifacts available

Assign a customer banking utterance to one of 77 intent labels, zero-shot.

Banking77 (test split) · n=208

run 2026-09-18

pre-registered

verdict: AMBIGUOUS

source: ickma2311

Model	Accuracy [95% CI]	AUROC	ECE	Latency p50 / p95	Cost	Error Ranking Auroc
Jev	83.2% [77.9%, 88.0%]	—	—	440 ms / 710 ms	$0.000071 / call measured, 77 labels in the prompt	0.853
GPT-5.4 nano gpt-5.4-nano-2026-03-17	79.3% [73.6%, 85.1%]	—	—	850 ms / 1,520 ms	not recorded	0.795
GPT-5.6 Terra	87.5% [82.7%, 91.8%]	—	—	1,510 ms / 5,570 ms	not recorded	0.808
BGE-small-en-v1.5 + logistic regression supervised, not zero-shot	93.3% [89.9%, 96.6%]	—	—	10 ms / 100 ms	not recorded	—

Protocol and limitations (6)

Deliberately zero-shot for the generative models. JSON-prompted output parsed with a regex rather than schema-constrained decoding. Prompts were not tuned per model. Calls issued serially. The supervised encoder is not zero-shot: it is frozen BAAI/bge-small-en-v1.5 CLS embeddings with a scikit-learn logistic regression head (C=10) fitted on the task's own training data.

The pre-registered primary result was AMBIGUOUS: the cascade escalation-rate difference was +0.265 with a 95% CI of [-0.530, +0.595], which spans zero. The accuracy ordering below should be read as descriptive, not as a decided result.
n=208, reduced from the pre-registered 300 because of rate limiting. Intervals are wide and overlap heavily.
Output was parsed from JSON with a regex rather than schema-constrained decoding, which can disadvantage the generative baselines relative to a production integration.
Prompts were not tuned per model.
The supervised encoder is trained on this dataset and is not directly comparable to the zero-shot systems.
LLM costs were not recorded — the client does not return them — so only Jev has a measured cost here.

Cascade result, which is the pre-registered primary outcome: to reach a 90.5% accuracy target, escalating from Jev to Terra required falling back on 0.220 of requests, versus 0.485 escalating from nano to Terra. This is a measured fallback rate on a real task and is the best available empirical anchor for the fallback-rate input in the cost model — with the caveat that the difference was not statistically significant.

A comparison page exists only where a single run measured both models on the same data under the same protocol and both sides have verified pricing. We do not generate a page per model name.