Can Jev pick the next click?
The most-shared use of Jev is as a browser agent’s decision step: look at the page, pick what to click. Demos show it succeeding fast. This record asks it every step of Mind2Web’s 1,341 test tasks — people booking, searching and shopping on real websites — and counts the steps and the whole tasks it gets right. The method and how it would be read were committed first.
- RIGHT ELEMENT
- 38.4%
- RIGHT ELEMENT + ACTION
- 27.3%
- TASKS FULLY RIGHT
- 29 / 1,341
- P50 PER STEP
- 151 ms
| SPLIT | TASKS | STEPS | RIGHT ELEMENT | 95% (TASK BOOTSTRAP) | WHEN LISTED | RIGHT ACTION | RIGHT STEP | TASKS FULLY RIGHT |
|---|---|---|---|---|---|---|---|---|
| All splits | 1,341 | 9,378 | 38.4% | 37.2%–39.6% | 44.4% | 66.2% | 27.3% | 29 (2.2%) |
| Cross-task (seen websites) | 252 | 2,094 | 38.3% | 35.4%–41.1% | 43.0% | 66.7% | 26.7% | 2 (0.8%) |
| Cross-website (unseen websites) | 177 | 1,373 | 39.5% | 36.7%–42.4% | 46.3% | 64.8% | 26.0% | 3 (1.7%) |
| Cross-domain (unseen domains) | 912 | 5,911 | 38.1% | 36.8%–39.5% | 44.5% | 66.4% | 27.8% | 24 (2.6%) |
“When listed” counts only the 8,096 steps where the right element is among the 50 candidates. Accuracy barely moves across the three splits, including websites and whole domains that Mind2Web’s training split does not contain.
27 of the 29 completed tasks had 1–3 steps. No task with 6 or more steps was completed (0 of 718). Step accuracy itself also falls as tasks get longer, so the drop is steeper than one error rate compounding.
| STEPS IN TASK | TASKS | FULLY RIGHT | OBSERVED | EXPECTED IF STEPS WERE INDEPENDENT | STEP ACCURACY |
|---|---|---|---|---|---|
| 1–3 | 260 | 27 | 10.4% | 3.50% | 43.1% |
| 4–5 | 363 | 2 | 0.6% | 0.37% | 33.0% |
| 6–7 | 258 | 0 | 0.0% | 0.03% | 26.5% |
| 8–10 | 231 | 0 | 0.0% | 0.00% | 26.2% |
| 11–15 | 162 | 0 | 0.0% | 0.00% | 23.1% |
| 16+ | 67 | 0 | 0.0% | 0.00% | 21.2% |
“Expected” applies the overall step accuracy (27.3%) to every step of each task. Overall, 0.8% of tasks would be expected to finish that way; 2.2% did, because errors cluster in some tasks rather than falling at random.
| THRESHOLD | STEPS KEPT BY JEV | SHARE | STEP ACCURACY ON KEPT | TASKS WHOSE EVERY ERROR IS BELOW IT |
|---|---|---|---|---|
| 0.50 | 4,247 | 45.3% | 41.4% | 288 (21.5%) |
| 0.70 | 1,988 | 21.2% | 51.4% | 718 (53.5%) |
| 0.90 | 459 | 4.9% | 64.5% | 1,192 (88.9%) |
Element confidence separates right from wrong choices with an AUROC of 0.732 — useful, but weaker than on the classification records. The last column is an upper bound: tasks a fallback that is always right would rescue if it took every step below the threshold. At the thresholds where that number is large, Jev keeps only a small share of steps, so the fallback would be doing most of the work. No fallback model was run.
| DEMONSTRATED ACTION | STEPS | RIGHT ELEMENT | RIGHT ACTION TYPE |
|---|---|---|---|
| CLICK | 7,896 | 34.6% | 65.1% |
| TYPE | 1,179 | 61.8% | 83.3% |
| SELECT | 303 | 45.9% | 27.4% |
- The action-type question was asked alongside the element question, so Jev chose it without knowing which element it had picked. That is how the protocol fixed it; an agent that infers the action from the element would likely score higher on this part. Element accuracy is unaffected.
- Jev answered “none of the listed elements” on 325 of the 1,282 steps where that was true, and on 881 steps where the right element was listed.
Pre-registered before any request. Each step is one request with two Choice questions: which of the authors' top-50 ranked candidate elements (listed in page order, plus none_of_the_above) to act on next, and whether the action is CLICK, TYPE or SELECT. State is the task and the last five demonstrated actions. Elements are described by tag, visible text and salient attributes. Typed values are not evaluated. Step-level intervals resample whole tasks. 9,378 requests, $0.66 at list price, all resolved to jev-1.13.0.
- Each step starts from the demonstrated history on a recorded page. A live agent that makes a mistake sees different pages afterwards; this record cannot say how Jev recovers from its own errors.
- The operation question is asked alongside, not after, the element question, so Jev chooses the action type without knowing which element it picked. An agent that infers the action from the element would likely do better on that part; element accuracy is unaffected.
- Candidates come from the Mind2Web authors' ranker, which lists the right element on 86.3% of steps; the rest are counted wrong, as in Mind2Web's own scoring.
- Elements are described in JevBench's own compact format (tag, first 12 words of visible text, salient attributes), without the surrounding page HTML or a screenshot. Browser agents that give Jev more of the page may do better.
- Typed text and selected option values are not evaluated, because Jev does not generate text.
- Latency is client-observed round-trip time from a cloud runner whose region was not recorded.
- Cost is derived from recorded input tokens at list price, not reconciled to an invoice.
- It does not measure any particular agent product, including Browser Use's Jev integration, whose prompts, page representation and recovery logic differ.
- It does not show how Jev compares with any other model; no other model was run under this setup.
- It does not show what better element descriptions, a better candidate ranker or a second model for low-confidence steps would achieve.
- It is not an independent result: JevBench chose the setup, wrote the protocol, ran Jev and wrote the analysis.
Mind2Web: Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H. and Su, Y. (2023), Mind2Web: Towards a Generalist Agent for the Web, NeurIPS 2023. Dataset licensed CC BY 4.0. The paper reports other models under a different setup (page HTML, several rounds of five-way choices, typed values scored); those figures are not comparable with these and are not repeated here.
Why the test data is not here. Mind2Web’s authors ask that the unzipped test split not be redistributed, so JevBench publishes only step ids, Jev’s choices, confidences and scores. The frozen inputs can be rebuilt byte for byte from the official downloads with the published script; their hashes are in the manifest.