Jev vs Claude Haiku 4.5

One reproducible head-to-head exists: 2,000 emails, both models, same protocol, 2026-09-17. Everything below comes from that run and from verified list pricing for both sides.

Jev wins on cost and latency

Cost per 1,000 emails	$0.038 vs $0.462
Latency p50	239 ms vs 687 ms
List input price	$0.042/MTok vs $1.00/MTok
List output price	$0.00/MTok vs $5.00/MTok

Claude Haiku 4.5 wins on every quality metric

Accuracy	62.6% [60.5%, 64.7%] vs 81.3% [79.5%, 82.9%]
Recall on phishing	43.2% vs 76.4%
False positive rate	18.0% vs 13.8%
AUROC	0.689 vs 0.837
Calibration error (ECE)	0.154 vs 0.097

The number that decides a security workload

Jev’s recall on phishing was 43.2%. On this set it missed more than half the phishing emails. Whatever the cost advantage, a detector that misses the majority of what it is looking for is not cheaper at the job — it is doing a different, smaller job. On a workload where a miss is expensive and a false positive is merely annoying, recall is the constraint and Claude Haiku 4.5 clears it by a wide margin.

On the latency figures

Both models were measured from France. The raw p50s are 239 ms and 687 ms, a 2.9x ratio — but Jev’s figure sits on a 163 ms network floor and Claude Haiku 4.5’s on 18 ms.

Subtracting the floors leaves residuals of 76 ms derived and 669 ms derived. A residual is not model compute time: it still contains queueing, gateway hops, TLS negotiation, serialisation and provider-side overhead. What it shows is that the ordering is not an artifact of where the client sat. The 2.9x ratio in the raw numbers is specific to this measurement location and transfers nowhere.

Cost, modelled from verified pricing

Jev, per month

$12.60

1M requests, 300 input tokens, output free

Claude Haiku 4.5, per month

$5,050

500 input + 910 output tokens per request

Break-even fallback rate

99.8%

Where a Jev cascade stops being cheaper

On this shape of workload the cost question is essentially settled before it is asked: the cascade stays cheaper even if almost every request escalates. Which means cost is not the decision — accuracy is. Most of the baseline’s bill is output tokens, which is why a comparison that counts only input tokens gets this badly wrong. Model your own workload.

The run itself

Phishing email detection: Jev vs Claude Haiku 4.5

AIndependent · Reproduction artifacts available

Label an email as phishing or legitimate.

PhishNChips v5.2 · n=2,000

run 2026-09-17

source: anisselbd

Model	Accuracy [95% CI]	AUROC	ECE	Latency p50 / p95	Cost
Jev jev-1.13.0	62.6% [60.5%, 64.7%]	0.689	0.154	239 ms incl. 163 ms network floor residual 76 ms derived	$0.038 / 1k list price at the time of the run
Claude Haiku 4.5	81.3% [79.5%, 82.9%]	0.837	0.097	687 ms incl. 18 ms network floor residual 669 ms derived	$0.462 / 1k list price at the time of the run

Protocol and limitations (4)

Binary decision per email. Jev accessed as jev-1.13.0; Claude Haiku 4.5 accessed via the Anthropic Messages API. Wilson intervals for proportions, bootstrap intervals for AUROC, ECE, Brier and F1; McNemar's test for the accuracy comparison. ECE computed with 10 bins.

Latency is confounded by geography. Measured from France, Jev's 239 ms p50 sits on a 163 ms network floor while Haiku's 687 ms sits on an 18 ms floor. Subtracting the floors leaves residuals of roughly 76 ms and 669 ms, but a residual is not model compute time: it still contains queueing, gateway hops, TLS, serialisation and provider-side overhead. The ranking survives either way; the 2.9x ratio in the raw numbers is specific to this measurement location and neither figure transfers elsewhere.
Single dataset and single task. Nothing here generalises to classification or routing workloads.
The repository does not state the author's affiliation or who funded the API costs.
Only one baseline model was tested.

The clearest published example of the trade-off: on this task Jev was roughly 12x cheaper per 1,000 emails and lower-latency, and materially less accurate.

This is the only model Jev has a published head-to-head against with verified pricing on both sides, so it is the only comparison page that exists. We do not generate a page per model name.