Jev RAG Benchmark

The question

Jev 1.13 returns typed decisions with calibrated probabilities instead of text — a natural fit for one of RAG's quietest jobs: deciding which retrieved passages deserve to reach the model, and when it is safe to skip a generation call entirely. The benchmark puts Jev, a trained NVIDIA cross-encoder and the unranked hybrid order on identical frozen candidates, then asks a single question: does the model actually improve the system?

The constraints

• 01

  English data only: XQuAD-EN and SciFact — no translation confound.

• 02

  $0 budget. Free tiers or the benchmark does not get re-run.

• 03

  Framework-neutral. The numbers must describe models, not wrappers.

• 04

  Negative results stay in. Exploratory analyses say so.

The decisions

1. D1

   A free stack, or a more reproducible one?

   Context

   A benchmark nobody can afford to re-run is a blog post. Reranking evaluations burn tokens on every method you compare.

   Options

   Paid APIs · local models on a laptop · free tiers of hosted models

   Chosen

   Free tiers only: Vercel AI Gateway for Jev 1.13, NVIDIA NIM for embeddings and the cross-encoder, Codiv for generation.

   Why

   Every published number can be reproduced by anyone with an account and no budget approval. The whole benchmark — 1,490 reranked queries across four methods — cost $0.

   The cost

   Model churn. Three NVIDIA models were retired mid-project and had to be replaced; each swap is recorded in the run manifest so results stay attributable.

2. D2

   A RAG framework, or a harness of my own?

   Context

   Reranking comparisons are easy to run in LlamaIndex or LangChain — and easy to contaminate with their prompt wrappers, parsers and batching.

   Options

   Adopt a framework · build a thin harness · fork an existing benchmark

   Chosen

   A framework-neutral harness with three runtime dependencies.

   Why

   A benchmark should measure models, not wrappers. Fewer layers also means clean latency numbers — and latency is a headline claim for System One models.

   The cost

   The glue has to be written and tested: BM25, reciprocal rank fusion, metrics, calibration, export. That work is the repo.

3. D3

   How to keep the comparison honest

   Context

   If retrieval re-runs between methods, a lucky or unlucky draw gets published as a model difference.

   Options

   Live retrieval per method · frozen candidate pools · frozen pools plus frozen contexts

   Chosen

   Freeze the top-20 candidates once, compare every reranker on that pool, then replay generation over one frozen top-5.

   Why

   Retrieval, reranking and generation effects are separated by construction. Paired bootstrap confidence intervals compare methods on identical queries.

   The cost

   The stages cannot adapt to each other. A real system could rerank the pool differently per query; the benchmark deliberately cannot.

4. D4

   Publish the runs where the hypothesis loses

   Context

   Jev helps where the first stage leaves headroom. On XQuAD the baseline is already at 98.1% nDCG — and a trained cross-encoder still leads by 0.45 points.

   Options

   Report only the wins · report everything · report everything and label what came later

   Chosen

   Report everything, with a preregistered protocol and an explicit changelog that marks later analyses as exploratory.

   Why

   A benchmark's credibility comes from the runs where its hypothesis loses. 'No measurable difference' is a finding, not a failure.

   The cost

   The headline is less shiny, and later additions had to carry an honest 'exploratory' label instead of being framed as confirmatory.

5. D5

   Always-on reranking, or confidence-gated decisions?

   Context

   Ranking every candidate by independent probabilities is noisy. The model's own calibration offered a different interface.

   Options

   Always-on reranking · all-or-nothing switching · confidence-partitioned decisions

   Chosen

   Keep candidates with p ≥ 0.5 in the model's order, fall back to the hybrid order for the rest; skip generation entirely when the top probability is low.

   Why

   The decision layer beat the unranked baseline on both datasets (+0.89 pts XQuAD, +4.14 pts SciFact), and answerability gating removed 9.8% of generation calls while raising answer F1.

   The cost

   A threshold must be chosen, documented and re-validated per domain. Gating is a policy, not a property of the model.

The result

nDCG@10 on identical frozen top-20 candidates. A = no reranker (hybrid order), T = TypeSafe Jev 1.13 batch noul, N = NVIDIA cross-encoder.

• Jev improves retrieval where there is headroom. +0.82 nDCG points over the unranked baseline on XQuAD (95% CI +0.35 to +1.31) and +7.62 on SciFact (CI +4.88 to +10.38).
• Against a trained cross-encoder: a tie on SciFact. +0.59 points, CI −1.25 to +2.42 — no measurable difference — while NVIDIA leads XQuAD by 0.45 points. Jev is not a universal ranker.
• Calibration is the sleeper feature. ECE 1.33% on XQuAD (6.25% on SciFact); top-1 confidence 0.96 when correct versus 0.74 when wrong. At p ≥ 0.7, precision is 94.9% against a 5% base rate — a 19x lift that a ranking metric alone would never show.
• Confidence-gated decisions beat the baseline everywhere. Partitioned reranking added +0.89 points on XQuAD (CI +0.45 to +1.36) and +4.14 on SciFact (CI +2.13 to +6.29); answerability gating removed 9.8% of generation calls while raising answer F1 from 30.6% to 31.3%.
• Every number above cost $0. Vercel AI Gateway free tier for Jev, NVIDIA NIM trial for embeddings and the cross-encoder, Codiv for generation. At list price the Jev calls would have been $0.33.

Bonus: tool-calling governance

Decision models can also govern tool calls: which tool to use, whether a call may run without review, whether arguments are valid, and whether observed text tries to manipulate the assistant. Same deterministic cases for all three: Jev through the Vercel free tier, Laya (open weights) through its official Hugging Face demo Space, and a free generative LLM asked for the same decisions as JSON.

• Laya's 200-option result is a refusal, not a wrong answer. Its English checkpoint rejects the question outright: options do not fit in head_max_len=192 tokens. Degradation starts earlier — 40% at 50 options.
• A free LLM matches Jev on accuracy here. Selection 82.5% vs 85%, approval 80% vs 70%, validation and injection tied at 100%. Jev's edge is operational: typed answers with no parsing, probabilities as a first-class API, many questions in one parallel request, input-only pricing.
• Fairness notes. Cases are synthetic and template-generated (one seed); Laya was served as its English base checkpoint; the LLM received the same state and questions with a JSON-only instruction; latency includes each hosted route (Laya's own reported model time is ~121 ms).

What failed

• Expecting gains everywhere. On XQuAD the baseline was already at 98.1% nDCG; there was almost nothing left to win, and always-on reranking added noise instead of value.
• Expecting to beat a trained cross-encoder. NVIDIA still leads XQuAD by 0.45 points; the honest summary is "useful decision layer", not "best ranker".
• Ignoring provider lifecycle. Three NVIDIA models were retired mid-run; the pipeline now pins versions and records every resolved model.
• Expecting free-tier throughput. The gateway answered bursts with 429 and Retry-After: 60; the run settled at roughly one request per second and checkpointed every 100 rows.

What I learned

• A benchmark's most valuable output is often the constraint map: where the model helps, where it does not, and what it must never be used for.
• For decision models, calibration is the product. Probabilities are what let a pipeline save money — ranking is just one consumer of them.
• Report latency with its conditions. Jev's 4-second median in the run is an artifact of free-tier pacing; a single request answered in 0.6 seconds. Publishing one number without the other would be misinformation.
• "No measurable difference" is a real finding. A trustworthy benchmark needs a home for it.