Confidence routing on Banking77: Jev answers, a larger model catches the rest

AIndependent · Reproduction artifacts availableIntent & routingrouting evaluationBanking77 (routing sample) · n = 500date not stated by sourceverdict: ROUTE — threshold 0.67

Answer with Jev when its confidence clears a threshold and escalate to a larger model otherwise, then measure the cascade end to end.

Accuracy

80.2%

Jev → DeepSeek v4 Pro cascade

no interval reported

Every recorded metric

Model	Accuracy	Recall	FPR	AUROC	ECE	Latency	Cost
Jev → DeepSeek v4 Pro cascade	80.2%	—	—	—	—	302 ms location not stated	$0.2066 / 1k derived from the run's stated total of $0.1033 across 500 examples

Protocol

A confidence-routing pipeline sending each decision to Jev or to DeepSeek v4 Pro according to Jev's confidence. The threshold was fitted on this dataset and the author states no routing parameter carried over to the second dataset. Jev was accessed as jev-latest and jev-1.13.0. Results are the routing library's own demo output rather than a separately designed evaluation.

Dataset

Banking77 (routing sample)

n = 500

500 labelled rows

Limitations (8)

The source does not state a run date.
The routing threshold of 0.67 was fitted on this same dataset, so the accuracy is not a held-out estimate.
No single-model baselines are reported for this dataset, so there is no way to tell from this run alone how much the routing bought over either model on its own.
The numbers are a routing library's demo output. The tool exists to do confidence routing and therefore has an interest in routing working — though it reported the opposite result on its second dataset.
No calibration metrics, despite calibration being what the threshold depends on.
No confidence intervals at n=500.
The latency figure does not state where it was measured from.
The author's affiliation and who paid API costs are not stated.

Note on this run

A second measured escalation rate for the cost model to sit beside the Banking77 cascade already recorded: 11.6% here at a fitted threshold, against 22.0% in the pre-registered study at a 90.5% accuracy target. Different targets and different fallback models — which is exactly why neither is a default.

What this benchmark does not prove

01That a Jev cascade reaches 80.2% accuracy on Banking77. The threshold was fitted on this data, so that figure is optimistic by an unknown amount.
02That routing beats running either model alone. No single-model baseline was measured here.
03That 11.6% is a typical escalation rate. It follows from one fitted threshold on one dataset with one fallback model.
04That confidence routing works. The same pipeline on the author's second dataset returned DO NOT ROUTE.

Source and artifacts

Janus — confidence routing between Jev and a larger fallback model — FirasSX914, verified 2026-09-19

Reproduction artifacts: github.com/FirasSX914/Janus

Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet

A routing library rather than a benchmark. Its measured results are the tool's own demo output on two public datasets, so the tool has an interest in routing working — it reported that it does not on one of the two, which cuts against that interest. No run date, calibration metrics or intervals are stated.