Phishing email detection: Jev vs Claude Haiku 4.5

AIndependent · Reproduction artifacts availableSecurity detectionPhishNChips v5.2 · n = 2,0002026-09-17

Label an email as phishing or legitimate.

Accuracy

62.6%

Jev jev-1.13.0

95% CI [60.5%, 64.7%]

81.3%

Claude Haiku 4.5

95% CI [79.5%, 82.9%]

Cost × accuracy

Hover or tap a point for its full record.

Every recorded metric

Model	Accuracy	Recall	FPR	AUROC	ECE	Latency	Cost
Jev	62.6% [60.5%, 64.7%]	43.2%	18.0%	0.689	0.154	239 ms incl. 163 ms floor residual 76 ms derived from France	$0.038 / 1k list price at the time of the run
Claude Haiku 4.5 via Anthropic Messages API	81.3% [79.5%, 82.9%]	76.4%	13.8%	0.837	0.097	687 ms incl. 18 ms floor residual 669 ms derived from France	$0.462 / 1k list price at the time of the run

Protocol

Binary decision per email. Jev accessed as jev-1.13.0; Claude Haiku 4.5 accessed via the Anthropic Messages API. Wilson intervals for proportions, bootstrap intervals for AUROC, ECE, Brier and F1; McNemar's test for the accuracy comparison. ECE computed with 10 bins.

Dataset

PhishNChips v5.2

n = 2,000

1,000 phishing / 1,000 legitimate

Limitations (4)

Latency is confounded by geography. Measured from France, Jev's 239 ms p50 sits on a 163 ms network floor while Haiku's 687 ms sits on an 18 ms floor. Subtracting the floors leaves residuals of roughly 76 ms and 669 ms, but a residual is not model compute time: it still contains queueing, gateway hops, TLS, serialisation and provider-side overhead. The ranking survives either way; the 2.9x ratio in the raw numbers is specific to this measurement location and neither figure transfers elsewhere.
Single dataset and single task. Nothing here generalises to classification or routing workloads.
The repository does not state the author's affiliation or who funded the API costs.
Only one baseline model was tested.

Note on this run

The clearest published example of the trade-off: on this task Jev was roughly 12x cheaper per 1,000 emails and lower-latency, and materially less accurate.

What this benchmark does not prove

01That Jev has an accuracy of 62.6%. That is its accuracy on this dataset, under this protocol, on this date — nothing more.
02That Jev is 2.9x faster than Claude Haiku 4.5. The raw latency ratio is specific to a client in France and the very different network floors underneath each figure.
03That Jev costs 12x less for your workload. The cost figures follow from this run's token usage; a different prompt shape moves them.
04That either model is unsuitable for phishing detection in general. One dataset, one prompt, one day.

Source and artifacts

jev-phishing-bench — Jev vs Claude Haiku 4.5 on 2,000 phishing emails — anisselbd, verified 2026-09-19

Reproduction artifacts: github.com/anisselbd/jev-phishing-bench

Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet

Public code and stated methodology. Repository does not state author affiliation or who funded API costs; recorded as unstated rather than assumed independent-funded.