Seven synthetic agent workloads: Jev vs Qwen 3.8 27B vs Needle 3
Drive seven simulated application workloads — tickets, routing, driving, guardrails, approvals, scoring and home commands — validating every output before applying a simulated action.
| Model | Accuracy | Recall | FPR | AUROC | ECE | Latency | Cost |
|---|---|---|---|---|---|---|---|
Jev via TypeSafe native API | — | — | — | — | — | 176 ms / 336 ms from browser client | $0.000025 / call derived from the run's stated total of $0.011919 across 480 dispatched requests; the source reports totals, not per-call figures |
Qwen 3.8 27B via Cerebras, schema-constrained output | — | — | — | — | — | 215 ms / 452 ms from browser client | $0.00065 / call derived from the run's stated total of $0.310581 across 476 dispatched requests |
Needle 3 via local runtime on Apple M4 Pro | — | — | — | — | — | 225 ms / 724 ms from local Apple M4 Pro | $0 / call run locally; the source explicitly excludes hardware and electricity |
Qwen 3.8 27B served by Cerebras with schema-constrained structured output; Jev through its native API with batched Choice/Noul questions; Needle 3 as a local tool-calling runtime accessed through the Hugging Face CLI. Every path validates its output before applying a simulated action. Fixture agreement is exact match against locally expected outputs. Qwen and Jev were measured from a browser client on 2026-09-17; Needle was measured separately on an Apple M4 Pro on 2026-09-18.
- The author states plainly that this is 'not a controlled speed ranking': Qwen and Jev were measured from a browser client while Needle ran locally on different hardware, on a different day. The latency figures are not comparable across those two groups.
- No confidence intervals. The author notes that 'repeated synthetic cases are not an independent population sample'.
- The workloads are synthetic and authored by the benchmarker, so label quality cannot be independently checked and nothing here is drawn from a public dataset.
- Fixture agreement is exact match against locally expected outputs. The author states it is 'not evidence of calibration or general security guarantees'.
- There is no single accuracy for this run and we do not compute one: the five scored workloads are different tasks and averaging them would invent a number nobody measured.
- Cost and latency are aggregate across the whole suite and cannot be attributed to any one workload.
- Per-call costs are derived by JevBench from the run's stated totals and dispatch counts, not reported per call.
- The repository does not disclose the author's affiliation or who funded API costs.
What this benchmark does not prove
- 01That Jev is more accurate than Qwen 3.8 27B, or less. It wins two of the five scored workloads, ties two and loses one badly; there is no aggregate and we do not compute one.
- 02That Jev is faster than Qwen. The 176 ms against 215 ms gap is inside what a browser-client measurement can shift, and the author explicitly disclaims a speed ranking.
- 03That Needle 3 is a weak model. It ran on different hardware through a different harness and failed 117 dispatches; this measures the combination, not the model.
- 04That the 26x cost gap transfers to your workload. It follows from this suite's token shapes, and Jev spent 8x more output tokens than Qwen here — which costs nothing only while Jev's output stays free.
typesafe-ai-benchmark — Jev vs Qwen 3.8 27B vs Needle 3 across seven synthetic agent workloads — iammrduncan / Hackers in the Loop, verified 2026-09-19
Reproduction artifacts: github.com/iammrduncan/typesafe-ai-benchmark
Independent: yes · Vendor-affiliated: no · Independently reproduced: not yet
Author states 'This is an independent benchmark, not a TypeSafe implementation or parity claim' and warns the timing figures are 'not a controlled speed ranking'. Affiliation and funding are not disclosed.