World Knowledge: TriviaQA, NQ & SQuAD

Separate the question from the evidence contract

A benchmark name alone does not define the experiment. Record what the system receives, what it may retrieve, which answer forms count, and when abstention is correct.

Freeze the run manifest

• Data identity: source, immutable revision, split, question IDs, evidence documents, exclusions, and deduplication result.
• System contract: model and index revisions, prompt, retrieval depth, reranker, decoding settings, and tool access.
• Answer contract: accepted aliases, normalization, citation requirement, abstention token, and maximum answer length.
• Report contract: exact match, token F1, retrieval recall, no-answer metrics, slices, uncertainty, cost, and latency.

Build a portfolio that matches the product claim

A support assistant, an internal-document reader, and a closed-book trivia feature need different evidence. Use the lab to choose a product claim, sample budget, and evidence mode. The benchmark allocation, critical-trait coverage, blind spots, and release status update together.

Use each benchmark for the decision it represents

TriviaQA tests questions against independently gathered evidence

The original TriviaQA release contains more than 650,000 question-answer-evidence triples built from about 95,000 question-answer pairs. Questions came from trivia enthusiasts, while web and Wikipedia evidence was gathered independently. That separation creates lexical and syntactic gaps between a question and its evidence.

• Score against all documented answer aliases rather than one literal string.
• Keep Wikipedia, web, verified, and unfiltered variants distinct.
• For retrieval experiments, report whether an accepted answer appears in the retrieved set before scoring the reader.
• Treat evidence that merely contains an answer string as distant supervision, not proof that every document supports the answer correctly.

Natural Questions tests real search behavior

Natural Questions uses anonymized, aggregated queries issued to Google Search. Annotators receive a Wikipedia page from the search results and may mark a long answer, a short answer, yes or no, or no supported answer.

• Preserve the difference between long-answer selection and short-answer extraction.
• Keep null answers in the denominator; filtering them creates an easier task.
• Slice by answer type, query form, document length, and evidence position.
• Do not call a reader-only experiment end-to-end search unless retrieval is actually part of the evaluated system.

SQuAD 2.0 tests extraction and abstention inside one passage

SQuAD 2.0 combines the answerable SQuAD 1.1 questions with more than 50,000 adversarially written unanswerable questions. The unanswerable questions resemble answerable ones, so a reader must compare its best span score with a no-answer score.

• Tune the no-answer threshold on a development split, then freeze it before test evaluation.
• Report answerable and unanswerable subsets beside the combined score.
• Retain exact match and token F1 because partial span overlap and exact extraction answer different questions.
• Do not use SQuAD alone to claim current knowledge, retrieval quality, or open-domain coverage.

Read the metrics as a failure map

Keep the arithmetic explicit

For answerable questions in a simplified retrieval pipeline:

supported correct rate = retrieval recall x reader correctness given retrieved evidence

If retrieval recall is 80% and reader correctness is 90%, at most 72% of answerable items are supported correct under those assumptions. Improving only the reader cannot recover the 20% of questions whose evidence never arrived.

Diagnose the stage that loses the answer

The next lab partitions one evaluation set into supported correct answers, retrieval misses, reader errors, correct abstentions, and unsupported answers. Change the answerable share and each stage quality. Every question remains in exactly one outcome bucket, so the largest failure source identifies the next experiment.

Implement scoring and attribution as auditable boundaries

The first example normalizes only documented surface differences, scores against all aliases, and treats an empty prediction as abstention only for an unanswerable item.

The second example assigns every evaluated question to one pipeline outcome. Production records should also retain the query, retrieved document IDs, corpus revision, prompt, raw answer, scorer revision, and latency.

Test the evaluator itself

• Add fixtures for aliases, punctuation, articles, empty answers, Unicode, duplicate tokens, and malformed records.
• Fail closed when an answerable item has no accepted aliases or when required stage evidence is missing.
• Keep scorer changes versioned and rerun old predictions before comparing reports.
• Manually review a bounded sample of exact matches, partial matches, abstentions, and retrieval misses.

Operate the benchmark as a measurement system

1. 1

   Scope

   Freeze product-shaped slices

   Sample stable facts, time-sensitive facts, ambiguous queries, answerable questions, and unsupported questions in proportions that reflect the declared claim.

2. 2

   Measure

   Run one traceable pipeline

   Persist retrieval and reader inputs, outputs, versions, timings, and policy decisions for every item without silently retrying failures.

3. 3

   Diagnose

   Attribute before tuning

   Separate corpus gaps, retrieval misses, reader errors, alias failures, unsupported answers, and justified abstentions before changing the system.

4. 4

   Release

   Gate with multiple signals

   Require critical-slice floors, retrieval coverage, answer quality, no-answer behavior, cost, latency, and contamination checks to pass together.

Minimum production report

1. Overall exact match and token F1 with the denominator and uncertainty interval.
2. Retrieval recall at each declared k, plus oracle-reader performance when gold evidence is available.
3. Answerable and unanswerable performance, including unsupported-answer and over-refusal rates.
4. Results by benchmark, source, question type, temporal sensitivity, answer length, and critical product slice.
5. Corpus, index, model, prompt, and scorer revisions with tokens, latency, and cost.

Know what these benchmarks cannot prove

• Public benchmark items may overlap with training, prompt libraries, retrieval corpora, or repeated development runs.
• Wikipedia-centered evidence does not represent every language, culture, domain, or current event.
• Exact match and F1 do not verify every claim in a long generated answer or the quality of a citation.
• A fresh corpus does not guarantee fresh answers if retrieval misses the relevant document or generation ignores it.
• A high no-answer score can hide blanket refusal unless answer coverage and usefulness are reported.
• Static benchmark success does not replace private product cases, red-team questions, monitoring, or incident review.

For a release claim, combine public benchmarks with protected product-shaped holdouts, time-stamped freshness cases, citation verification, and operational monitoring.

Ground the protocol in the original releases

• The TriviaQA paper defines the question-answer-evidence construction and the original dataset variants.
• The Natural Questions publication documents real search queries, long and short annotations, null cases, and public split sizes.
• The SQuAD 2.0 paper explains the adversarial unanswerable-question design and the need to abstain.

Use the official release's schema, split, scorer, and license as the baseline. Add product diagnostics as separately named evidence rather than silently redefining the public score.