How do you benchmark a System One model? What JevBench scores, and what calibration says it cannot

You benchmark a System One model on four things: whether its decisions beat chance, whether its probabilities match how often it is right, how fast it answers, and what 1,000 decisions cost. JevBench, posted on 22 September 2026, scores all four. Its v1.4.1 leaderboard, scored on 23 September, puts Jev first at 63.3 out of 100.

What no public benchmark can tell you is whether those probabilities hold on your own data. That is the argument of Jev Can't Be Calibrated, published by Alex Molas a day later. The two pieces answer different questions. Use the benchmark to shortlist a model, and your own labelled cases to decide where its thresholds go.

If System One models are new to you, the explainer covers what they are. In short: a small, fast model that takes a state and a typed question and returns a typed answer with a probability, next to the LLM that does the open-ended work.

What does JevBench measure?

Four axes, each from 0 to 100, combined into one score. The leaderboard and its MIT-licensed harness define them:

• Intelligence: accuracy corrected for chance, so guessing scores zero. 534 public decisions make up 80 percent of it, and 308 sealed ones, added in v1.4 with chance at 29.3 percent, make up the other 20.

• Calibration: expected calibration error, which asks whether answers given 80 percent confidence are right about 80 percent of the time.

• Speed: median latency on a log scale.

• Cost: US dollars per 1,000 decisions, also on a log scale. Jev comes to $0.040 at about 950 input tokens per decision.

The overall score is a harmonic mean: 4 / (1/I + 1/C + 1/S + 1/K). A harmonic mean punishes the weakest axis, so a fast, cheap model that guesses cannot buy its way up. An Intelligence, Speed or Cost score below 50 is also multiplied by (axis/50)². A model whose public accuracy beats its sealed accuracy by more than 25 points loses Intelligence as well.

Every row also discloses how its result was obtained: whether its authors tuned against the public items, what an overlap scan found and whether an operator's endpoint saw sealed text. The results come as JSON with a SHA-256 checksum. The page also discloses that its authors run jev-router.com, which offers self-hosted open decision models and is not a ranked entrant.

Why did Jev score 74.4 last week and 63.3 now?

Because the benchmark changed, not the model. Under v1.3, the Show HN version, Jev 1.13.0 scored 74.4, SemIf 73.1 and djev 73.0. I can only find those figures in search-engine copies of the page. The launch post on X had Jev at 75.3 and SemIf at 74.6, which look like an earlier release. On 23 September, v1.4.0 added the sealed set and switched to the harmonic mean. v1.4.1 followed the same evening with six more systems and unchanged scoring.

The top nine of the v1.4.1 board:

Numbers marked ~ are the benchmark's own estimates for hosting an open model at a large provider's rates. Laya, the most-starred open alternative, ranks 36th at 30.3. It is also the cheapest system in the top 40, at about $0.003 per 1,000 decisions on a CPU, roughly half the price of the next one.

Is a 1.3-point gap between Jev and SemIf meaningful?

No. Under v1.3, SemIf trailed Jev by 1.3 points. Under v1.4, with the same models and a new formula and item set, it trails by 15.6 and sits ninth. A ranking that moves that much when the method changes is telling you about the method.

The sealed set explains most of it. Jev answered 86.6 percent of the public decisions correctly and 36.7 percent of the sealed ones. Nearly every system on the board shows a gap of about 50 points, which the gap penalty turns into roughly a quarter off its Intelligence score. Some of that is because the sealed items are harder: chance on them is 29.3 percent. Some of it is saturation. The public items have been online since launch, and several entrants disclose developing against them. The changelog says as much: "Fresh sealed decisions keep the benchmark moving as public items saturate."

So read the top ten as a band, not a podium. Differences of a few points between neighbours sit inside what one formula change can undo. Gaps of 30 points or more, like Jev's 33 over Laya, survive it.

What does "Jev can't be calibrated" mean in practice?

It means a probability from Jev is a score, not a promise. Molas's argument is about the math, not about Jev in particular. Calibration is a property of a model and a data distribution together. TypeSafe calibrated Jev on its training distribution. Your support tickets, fraud flags or pull requests are a different distribution, so its 0.8 does not have to mean 80 percent for you. His advice is to "treat its outputs as scores rather than probabilities".

The examples in the discussion go both ways. Molas reports Jev saying a fair coin lands heads with probability 0.92. A follow-up post measured 83 percent average confidence and 19 percent accuracy on a hidden fair die roll. In the other direction, a LessWrong study used Jev to flag backdoored code and found it "quite shy". It caught 90 percent of backdoors at a 2 percent false-positive rate, with an AUROC of 0.976, while its raw probabilities sat too low. The author's verdict: "a great ranking and a poor probability".

Jev was over-confident on the coin and the die, and under-confident on backdoored code, where it still ranked cases well. That is what the calibration argument predicts. JevBench's Calibration axis measures calibration on JevBench's questions, which is useful for comparing models and says nothing certain about yours. TypeSafe's own documentation is careful here too: its rates "describe groups of predictions, not a guarantee about any single answer."

Should you threshold on the model's probability or on your own data?

On your own data. The standard recipe:

1. Label a few hundred real cases from the traffic the model will see. Molas says "a few hundred labeled examples from your actual data can be enough".

2. Hold some of them out. Run the model on all of them and keep the probabilities.

3. Fit a calibration map on the rest. Platt scaling is Molas's suggestion. Isotonic regression, from the same scikit-learn page, is my addition for when you have more labels.

4. Pick the threshold on the held-out cases for the error rate you can accept, not for a round number like 0.9.

5. Refit when your traffic changes. A new product line or a new language is a new distribution.

Test option order while you are at it. regolo.ai found that reversing the order of options raised Jev's probabilities from a range of 0.84-0.89 to 0.93-0.96. Shuffle the options on your held-out set and see whether your threshold still holds.

Which latency numbers can you compare?

Only ones measured the same way, and JevBench measures three ways at once. API systems, Jev included, are called one request at a time from a server in Germany, over the internet. Self-hosted endpoints get a flat "×2 + 0.15 s" added, which the README calls an assumption, not a measurement. Local CPU and GPU runs are timed directly.

That is a reasonable choice for one table. It also means Jev's 0.65 seconds on the board is not the 70 to 500 ms on TypeSafe's launch page. Both numbers are honest, and they time different trips. Workload matters as much as setup. One independent benchmark timed Jev at about 120 ms for one question and 306 ms for seven questions about the same state. The alternatives piece lists how every project's headline latency was measured, and no two were measured the same way.

What should you measure yourself before production?

Five things, all on your own traffic:

• Accuracy on your labelled cases, per question type, not one blended number.

• Calibration after fitting, on the held-out cases. Check it separately for any language or segment you care about.

• Latency end to end, with your state size and question count, from where your service runs.

• Cost per 1,000 of your decisions, since token counts vary with state size.

• Stability: the same input twice, and the same options in a different order.

This fits the Stackness move Instrument LLM and agent applications with tracing and automated evals. A decision model is one more component whose output you log and score. The slot it fills is the move Use a fast small decision model instead of an LLM call for structured filtering and scoring.

Which of these models do people list in their stacks?

Almost none yet. As of 24 September 2026, Jev has one listing on Stackness, the founder's. JevBench, SemIf, djev, Laya, Kev, Open-Jev and jev_local have none. JevK5, Hopper, Winnow-12B, reflex 4B, Jev-Omni and metask-jev-4b joined the catalog today. Jev launched on 15 September, nine days ago. Until people list these models, a benchmark score is the only public signal, and this one is run by a team that sells self-hosted open decision models. When profiles start listing them, their tool pages will show who keeps which one next to what.

Key numbers

• 63.3 is Jev 1.13.0's JevBench v1.4.1 score, first of 77 ranked systems, scored 23 September 2026 (JevBench).

• 74.4, 73.1 and 73.0 were Jev, SemIf and djev under v1.3, the version posted on 22 September.

• 534 public and 308 sealed decisions make up v1.4. The sealed set is 20 percent of Intelligence, with chance at 29.3 percent.

• 86.6 percent against 36.7 percent is Jev's accuracy on public against sealed decisions.

• $0.040 per 1,000 decisions and 0.65 s median latency is Jev on the board.

• 0.976 AUROC with probabilities that ran too low is Jev as a code monitor in the LessWrong study.

• 1 Stackness profile lists Jev as of 24 September 2026 (see data sources).

Quick answers

How do you benchmark a System One model? Score accuracy above chance, calibration, latency and cost per 1,000 decisions, on questions the model has not been tuned on. JevBench does this publicly. Repeat the accuracy and calibration checks on a few hundred of your own labelled cases.

What does JevBench measure? Intelligence, Calibration, Speed and Cost, each 0 to 100, combined with a harmonic mean that punishes the weakest axis. Version 1.4 added 308 sealed decisions to limit tuning against public items.

Which System One model scores highest on JevBench? Jev 1.13.0 at 63.3 in v1.4.1, then JevK5 at 62.0 and Hopper at 59.4. The top ten sit within about 16 points.

Are Jev's probabilities calibrated? On TypeSafe's data, probably. On yours, not necessarily. Calibration depends on the data distribution, so treat the output as a ranking score and fit your own calibration map.

How many labelled examples do you need to calibrate a decision model? A few hundred from your real traffic can be enough for Platt scaling. Keep some aside to pick the threshold.

Can you compare latency across JevBench rows? Only roughly. API calls are timed over the internet from Germany, self-hosted endpoints get an assumed overhead added, and local runs are timed directly.