EVIDENCE STATUS / [ VERIFIED ]

Jev vs GPT-5.6 Terra

One pre-registered head-to-head exists: 208 Banking77 utterances, both models, same protocol, 2026-09-18. The run’s own verdict was AMBIGUOUS — the accuracy ordering below is descriptive, not a decided result.

What this run does not show

A supervised encoder (BGE-small embeddings with a logistic regression head, trained on Banking77’s own labels) scored 93.3% in the same run — the highest accuracy of anything measured. It is not a zero-shot system, so it is not compared here. A second Jev-vs-Terra head-to-head on CLINC150 is noted below.

GPT-5.6 Terra leads on accuracy — undecided

Accuracy	83.2% [77.9%, 88.0%] vs 87.5% [82.7%, 91.8%]
Run verdict	AMBIGUOUS (pre-registered)
Not proven	That Jev loses to a frontier model here

Jev leads on latency and cost

Latency p50	440 ms vs 1,510 ms
Latency p95	710 ms vs 5,570 ms
List input price	$0.042/MTok vs $2.00/MTok
List output price	$0.00/MTok vs $12.00/MTok

48x the input price for an undecided lead

GPT-5.6 Terra’s input list price is 48x Jev’s ($2.00/MTok vs $0.042/MTok), and 12x on output. The accuracy gap it buys here — 4.3 points — sits inside overlapping intervals, and the run’s own “does not prove” list explicitly includes a frontier win over Jev. On this evidence, the premium is priced as if the quality gap were settled. It is not.

On the accuracy figures

n=208, reduced from the pre-registered 300 by rate limiting; intervals are wide and overlap in the [82.7%, 88.0%] band. GPT-5.6 Terra’s 87.5% against Jev’s 83.2% is the descriptive ordering of one run, not a finding. Both generative baselines were JSON-prompted with regex-parsed outputs and untuned prompts, which can disadvantage them relative to a production integration.

On the latency figures

Calls were issued serially from the same client: Jev’s p50 of 440 ms against 1,510 ms, roughly a 3.4x ratio. The run recorded no network floor, so no residual analysis — treat it as ordering evidence, not a portable constant.

Note the p95: 5,570 ms. Verbose JSON outputs cost a frontier model twice — once in latency, once on the output-token meter at $12.00/MTok.

Cost: what is verified and what is not

Jev, list price

$0.042/MTok / $0.00/MTok

Per MTok input / output. OpenRouter, verified 2026-09-19.

GPT-5.6 Terra, list price

$2.00/MTok / $12.00/MTok

Per MTok input / output, short context. OpenAI, verified 2026-10-14.

Measured, this run

$0.000071 / call

Jev only (measured, 77 labels in the prompt). The run's client did not return LLM token usage, so GPT-5.6 Terra has no measured cost.

In the same run, GPT-5.6 Terra served as the escalation target: reaching 90.5% accuracy took a 22.0% fallback rate from Jev (the nano-to-Terra cascade needed 48.5% — see Jev vs GPT-5.4 nano). No monthly cost is modelled here: without recorded token counts that would require inventing a workload shape. Model your own workload with the verified list prices above.

A second head-to-head

CLINC150 intent routing (n=200, 2026-09-18, pre-registered verdict also AMBIGUOUS): Jev 87.0% vs GPT-5.6 Terra 91.5%, no confidence intervals reported. Same directional pattern, weaker evidence.

The run itself

Banking77 intent classification: Jev vs nano, frontier and a supervised encoder

AIndependent · Reproduction artifacts available

Assign a customer banking utterance to one of 77 intent labels, zero-shot.

Banking77 (test split) · n=208

run 2026-09-18

pre-registered

verdict: AMBIGUOUS

source: ickma2311

Model	Accuracy [95% CI]	AUROC	ECE	Latency p50 / p95	Cost	Error Ranking Auroc
Jev	83.2% [77.9%, 88.0%]	—	—	440 ms / 710 ms	$0.000071 / call measured, 77 labels in the prompt	0.853
GPT-5.4 nano gpt-5.4-nano-2026-03-17	79.3% [73.6%, 85.1%]	—	—	850 ms / 1,520 ms	not recorded	0.795
GPT-5.6 Terra	87.5% [82.7%, 91.8%]	—	—	1,510 ms / 5,570 ms	not recorded	0.808
BGE-small-en-v1.5 + logistic regression supervised, not zero-shot	93.3% [89.9%, 96.6%]	—	—	10 ms / 100 ms	not recorded	—

Protocol and limitations (6)

Deliberately zero-shot for the generative models. JSON-prompted output parsed with a regex rather than schema-constrained decoding. Prompts were not tuned per model. Calls issued serially. The supervised encoder is not zero-shot: it is frozen BAAI/bge-small-en-v1.5 CLS embeddings with a scikit-learn logistic regression head (C=10) fitted on the task's own training data.

The pre-registered primary result was AMBIGUOUS: the cascade escalation-rate difference was +0.265 with a 95% CI of [-0.530, +0.595], which spans zero. The accuracy ordering below should be read as descriptive, not as a decided result.
n=208, reduced from the pre-registered 300 because of rate limiting. Intervals are wide and overlap heavily.
Output was parsed from JSON with a regex rather than schema-constrained decoding, which can disadvantage the generative baselines relative to a production integration.
Prompts were not tuned per model.
The supervised encoder is trained on this dataset and is not directly comparable to the zero-shot systems.
LLM costs were not recorded — the client does not return them — so only Jev has a measured cost here.

Cascade result, which is the pre-registered primary outcome: to reach a 90.5% accuracy target, escalating from Jev to Terra required falling back on 0.220 of requests, versus 0.485 escalating from nano to Terra. This is a measured fallback rate on a real task and is the best available empirical anchor for the fallback-rate input in the cost model — with the caveat that the difference was not statistically significant.

A comparison page exists only where a single run measured both models on the same data under the same protocol and both sides have verified pricing. We do not generate a page per model name.