Banking77 intent classification: Jev vs nano, frontier and a supervised encoder

AIndependent · Reproduction artifacts availableClassificationBanking77 (test split) · n = 2082026-09-18pre-registeredverdict: AMBIGUOUS

Assign a customer banking utterance to one of 77 intent labels, zero-shot.

Accuracy

83.2%

Jev

95% CI [77.9%, 88.0%]

79.3%

GPT-5.4 nano gpt-5.4-nano-2026-03-17

95% CI [73.6%, 85.1%]

87.5%

GPT-5.6 Terra

95% CI [82.7%, 91.8%]

93.3%

BGE-small-en-v1.5 + logistic regression

95% CI [89.9%, 96.6%]

supervised — trained on this dataset, not zero-shot

Every recorded metric

Model	Accuracy	Recall	FPR	AUROC	ECE	Latency	Cost
Jev via Vercel AI Gateway (typesafe-ai/jev)	83.2% [77.9%, 88.0%]	—	—	—	—	440 ms / 710 ms	$0.000071 / call measured, 77 labels in the prompt
GPT-5.4 nano via Lightning AI	79.3% [73.6%, 85.1%]	—	—	—	—	850 ms / 1,520 ms	not recorded
GPT-5.6 Terra via Lightning AI	87.5% [82.7%, 91.8%]	—	—	—	—	1,510 ms / 5,570 ms	not recorded
BGE-small-en-v1.5 + logistic regression	93.3% [89.9%, 96.6%]	—	—	—	—	10 ms / 100 ms	not recorded

Protocol

Deliberately zero-shot for the generative models. JSON-prompted output parsed with a regex rather than schema-constrained decoding. Prompts were not tuned per model. Calls issued serially. The supervised encoder is not zero-shot: it is frozen BAAI/bge-small-en-v1.5 CLS embeddings with a scikit-learn logistic regression head (C=10) fitted on the task's own training data.

Dataset

Banking77 (test split)

n = 208

77 labels, paired n=208

CC-BY 4.0

Limitations (6)

The pre-registered primary result was AMBIGUOUS: the cascade escalation-rate difference was +0.265 with a 95% CI of [-0.530, +0.595], which spans zero. The accuracy ordering below should be read as descriptive, not as a decided result.
n=208, reduced from the pre-registered 300 because of rate limiting. Intervals are wide and overlap heavily.
Output was parsed from JSON with a regex rather than schema-constrained decoding, which can disadvantage the generative baselines relative to a production integration.
Prompts were not tuned per model.
The supervised encoder is trained on this dataset and is not directly comparable to the zero-shot systems.
LLM costs were not recorded — the client does not return them — so only Jev has a measured cost here.

Note on this run

Cascade result, which is the pre-registered primary outcome: to reach a 90.5% accuracy target, escalating from Jev to Terra required falling back on 0.220 of requests, versus 0.485 escalating from nano to Terra. This is a measured fallback rate on a real task and is the best available empirical anchor for the fallback-rate input in the cost model — with the caveat that the difference was not statistically significant.

What this benchmark does not prove

01That Jev beats a nano-class model or loses to a frontier model on classification. The pre-registered outcome was AMBIGUOUS and the intervals overlap heavily at n=208.
02That a supervised encoder is 'better than' Jev. It was trained on this dataset's own labels and is not solving the zero-shot problem — it is a different option, not a better score.
03That a Jev cascade needs a 22% fallback rate. That figure holds for this dataset, this cascade and a 90.5% accuracy target only.
04That the generative baselines were run at their best. Output was regex-parsed rather than schema-constrained and prompts were not tuned per model.

Source and artifacts

jev-baselines-eval — Pre-registered independent eval of Jev against nano-class, frontier, and supervised-encoder baselines — ickma2311, verified 2026-09-19

Reproduction artifacts: github.com/ickma2311/jev-baselines-eval

Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet

Pre-registration is hash-pinned in the repository. Author states no TypeSafe affiliation, no vendor involvement, no free credits, and that API costs (~$1) were paid by the author.