Distillation
Jev
What if a model answered with a decision and a probability instead of a paragraph?
A decision, not a paragraph.
Most models in a RAG pipeline are asked to produce prose: a retriever returns a list, a generator writes an answer. Between them sits a much narrower question, does this passage actually answer the question?, and almost nobody asks a model directly.
Jev does. TypeSafe’s Jev 1.13 is a small “System One” model: given a question and one candidate passage, it returns a typed decision, not a paragraph and not a similarity score, but a verdict with a probability attached. A free open-weights lookalike, OpenJev, is built for the same job, which makes the two comparable.
The difference between a score and a decision matters. A cross-encoder A model that reads two texts together, query and document, and outputs a single relevance score. Slower, because nothing can be precomputed. score orders candidates well, but “0.81” means little on its own: not comparable across queries, not comparable across models. A calibrated probability can be compared to a threshold. That turns ranking into a policy you can write down.
Nothing moves when the threshold is high, because the model only touches candidates it is sure about. Lower it and the model takes over more of the list. Lower it further and it starts deciding passages it already rated as noise. A decision layer is a dial, not a switch.
Calibration is a promise about being right.
A probability is useful only if it means something. Saying “p = 0.8” is a promise: among all decisions rated 0.8, roughly eight in ten should be correct. A model that keeps that promise is calibrated; one that does not is just emitting a ranking with extra decimals.
Calibration is measured against outcomes, not vibes. In the benchmark’s free-clone analysis, candidate probabilities on XQuAD scored ECE 0.0245 (2.5%) and Brier 0.0086, and at p ≥ 0.5, precision was 94.9% against a base rate of about 5%. That is what makes a threshold meaningful: a number you can lean on.
Math
Two calibrated operations replace “rerank everything”:
decide(d) = 1[p(d) ≥ t]order = sort({d : p(d) ≥ t}, by p desc) ++ sort({d : p(d) < t}, original order)
Generation is gated the same way. For N queries, calls scale with the probability that the best candidate clears the threshold:
calls = N · P(max p ≥ t)
Expected calibration error bins predictions and compares each bin’s confidence with its observed accuracy:
ECE = Σ_b (n_b / n) · |accuracy(b) − confidence(b)|
In the benchmark, the same model’s ECE rose from 0.0245 on XQuAD to 0.0906 on SciFact: the promise is domain-dependent. One threshold does not transfer between corpora.
Deeper
The threshold itself is not a property of the model; it is a policy. It depends on the base rate of the corpus: in a collection where 5% of passages are relevant, p ≥ 0.5 behaves very differently than in one where 40% are. The right threshold is chosen, documented and re-validated per domain, and the only way to know is to measure.
This is also why gating pays: it is not a smarter ranking, it is refusing to spend a generation call on a weak candidate. In the benchmark, answerability gating removed 31% of generation calls while raising answer F1 from 31.9% to 33.0%.
Where the decision layer sits.
One question, and a corpus that cannot be scanned exhaustively. Everything starts with cheap recall.
The benchmark’s question is not “is Jev good?” but does the model actually improve the system? Four branches, identical frozen candidates, 1,490 queries, nDCG@10:
| Branch | Method | XQuAD-EN | SciFact |
|---|---|---|---|
| A | no reranker, hybrid order (baseline) | 98.11 | 71.67 |
| J | OpenJev 0.1, free, open weights | 98.03 | 63.92 |
| N | NVIDIA cross-encoder | 99.37 | 78.70 |
| T | TypeSafe Jev 1.13 | 98.93 | 79.29 |
- Jev helps, measurably. +0.82 nDCG points over baseline on XQuAD (95% CI +0.35 to +1.31) and +7.62 on SciFact (CI +4.88 to +10.38).
- The free clone is not a substitute. OpenJev was flat on XQuAD and harmed ranking on SciFact by 7.75 points. Same approach, different model.
- Against a trained cross-encoder, a tie on SciFact. +0.59 points with a CI crossing zero, while NVIDIA led XQuAD by 0.45 points. A decision interface is not an automatic win.
- Calibration is the sleeper feature. Confidence-gated reranking added +0.66 points over baseline on XQuAD, and answerability gating cut generation calls by 31% while improving answer F1.
- Every number cost $0. Free tiers only, so anyone can re-run it.
Read the full case study or open the live leaderboard on Hugging Face.
The honest part is the benchmark.
- The free clone is a trap. It looks like the same idea and produces confident probabilities, yet it does not improve retrieval. Testing the brand instead of the model would have shipped a regression.
- Latency arrives with the decision. In the published run, batch decisions took about 4 seconds at the median per query, an artifact of free-tier pacing; a single request answered in 0.6s. For interactive search, a 4-second reranker is not shippable. Publish latency with its conditions or it is misinformation.
- The threshold is a policy, not a truth.
t = 0.5is a default. It depends on the base rate of your corpus, and copying it into a new domain is a guess until you measure. - Hosted models retire. Three NVIDIA models were retired mid-project and had to be replaced. Pin versions, record every resolved model, or your benchmark stops being reproducible.
- Ceilings are findings too. On XQuAD the baseline was already at 98.1%, and a benchmark where nothing can be won can only measure noise. A decision layer shines where recall is good but ordering is bad, not everywhere.
- The evaluation tax. Frozen candidate pools, paired bootstrap confidence intervals, negative results kept, exploratory analyses labelled. A benchmark nobody can re-run is a blog post.
The real product of a benchmark is not “the model is good”. It is a map: where the decision layer helps, where it does not, and what it must never be used for.
If you remember one thing
Jev is a System One model that returns typed decisions with calibrated probabilities, turning “which passages should reach the model?” into a policy you can measure.