CLINC150 intent routing with out-of-scope detection

AIndependent · Reproduction artifacts availableIntent & routingCLINC150 (test + OOS test splits) · n = 2002026-09-18pre-registeredverdict: AMBIGUOUS

Assign an utterance to one of 150 intents or to an out-of-scope bucket, zero-shot.

Accuracy

87.0%

Jev

no interval reported

79.5%

GPT-5.4 nano gpt-5.4-nano-2026-03-17

no interval reported

91.5%

GPT-5.6 Terra

no interval reported

Every recorded metric

Model	Accuracy	Recall	FPR	AUROC	ECE	Latency	Cost
Jev via Vercel AI Gateway (typesafe-ai/jev)	87.0%	—	—	—	—	423 ms	$0.00011 / call measured, 151 labels in the prompt
GPT-5.4 nano via Lightning AI	79.5%	—	—	—	—	924 ms	not recorded
GPT-5.6 Terra via Lightning AI	91.5%	—	—	—	—	—	not recorded

Protocol

Deliberately zero-shot. JSON-prompted output parsed with a regex rather than schema-constrained decoding. Prompts were not tuned per model. Latency compared on a 30-item serial subset so that the timing comparison is like-for-like.

Dataset

CLINC150 (test + OOS test splits)

n = 200

150 intents plus an out-of-scope label

CC-BY-SA 3.0

Limitations (6)

Pre-registered kill/go criteria returned AMBIGUOUS.
n=200, with no confidence intervals reported for the accuracy figures.
Out-of-scope recall is labelled exploratory by the author, not a pre-registered outcome.
Latency comparison rests on a 30-item subset.
Regex-parsed JSON rather than schema-constrained decoding for the generative baselines.
Prompts were not tuned per model.

Note on this run

Jev's error-ranking AUROC of 0.734 here is below nano's 0.816, the reverse of the Banking77 ordering. Calibration quality does not appear to be stable across tasks, which matters directly for anyone planning to route on Jev's confidence score.

What this benchmark does not prove

01That Jev's confidence score is safe to route on. Its error-ranking AUROC here (0.734) is below the nano baseline's (0.816) and the reverse of the Banking77 ordering.
02That Jev handles out-of-scope input well. The out-of-scope figures are labelled exploratory by the author, not a pre-registered outcome.
03That the accuracy ordering is decided. No confidence intervals were reported at n=200 and the run's verdict was AMBIGUOUS.
04That the latency comparison generalises. It rests on a 30-item serial subset.

Source and artifacts

jev-baselines-eval — Pre-registered independent eval of Jev against nano-class, frontier, and supervised-encoder baselines — ickma2311, verified 2026-09-19

Reproduction artifacts: github.com/ickma2311/jev-baselines-eval

Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet

Pre-registration is hash-pinned in the repository. Author states no TypeSafe affiliation, no vendor involvement, no free credits, and that API costs (~$1) were paid by the author.