Knowledge & Comprehension Testing
Design valid knowledge and comprehension evaluations with claim taxonomies, coverage blueprints, slice-level evidence, uncertainty, and release gates.
What is knowledge and comprehension evaluation?
Knowledge evaluation is a measurement process for testing whether an AI system can retrieve, explain, and apply information for a defined domain, source set, time boundary, and intended use. It can test a model alone, a retrieval-augmented system, or a tool-using product, but the protocol must say which one is being measured.
This matters because one high accuracy number can hide the difference between memorizing familiar questions, explaining a concept, applying it to a new case, and using current evidence correctly. The core invariant is that a knowledge score supports only the claim represented by its items, protocol, scoring rules, and decision-relevant slices.
A benchmark does not prove human-like understanding. It provides bounded behavioral evidence under stated conditions.
Turn a broad knowledge claim into defensible evidence
Every stage narrows what the final score may honestly claim.
Domain, time, use
Define the claim
Name the users, topics, authoritative sources, knowledge cutoff, languages, tools, and consequences represented by the evaluation.
Coverage contract
Build the blueprint
Allocate protected items across domains, cognitive operations, difficulty, formats, and critical slices before seeing model results.
Comparable run
Freeze the protocol
Pin prompts, retrieval access, tools, sampling, answer format, graders, versions, and failure handling.
Bounded decision
Qualify the result
Report slice evidence, uncertainty, contamination checks, reviewer agreement, and the use the evidence does or does not support.
Design the evidence blueprint before writing questions
An evaluation blueprint is the planned distribution of items across the knowledge dimensions needed for a claim. It prevents the easiest item type from becoming the benchmark simply because it is cheap to author and score.
Define at least these dimensions:
- Domain: the subject area, product surface, policy family, or document collection.
- Time boundary: the date or version through which an answer should be correct.
- Operation: recall, explanation, application, comparison, or evidence use.
- Access mode: closed book, supplied context, retrieval, or approved tools.
- Population and language: the users, regions, languages, and terminology represented.
- Consequence: which errors are merely inconvenient and which require stronger evidence or human review.
Critical cells need their own minimum item counts. A large general-knowledge sample cannot compensate for a thin security, exception-handling, or current-policy slice.
Build a knowledge evidence blueprint
Loading the evaluation blueprint...
Loading the blueprint...
Use cognitive levels as an item-writing lens
Bloom-style levels can help authors vary what an item asks the system to do. They should not be treated as proof that a model has a human cognitive process or developmental stage.
Coverage
Remember and understand
Ask for a fact, definition, summary, classification, or causal explanation. These items establish breadth and basic coherence, but they do not establish transfer.
Transfer
Apply and analyze
Give a new constrained case. Require the system to choose a rule, calculate a result, compare mechanisms, diagnose a failure, or separate relevant from irrelevant evidence.
Judgment
Evaluate and create
Ask for a decision or design under an explicit rubric. Score factual support, constraint satisfaction, and justification separately so fluent prose cannot hide a wrong result.
Pair levels within the same domain. If recall is strong but application is weak, the system has not demonstrated the transfer implied by an expert-assistant claim.
Match the item form to the knowledge evidence
Different answer formats reveal different failure modes. Use the least subjective form that still represents the intended task.
Fast breadth
Selected response
Multiple-choice or classification items are easy to score and useful for broad coverage. Rotate option positions, inspect distractors, and avoid treating guessing-adjusted accuracy as application evidence.
Direct retrieval
Short constructed answer
Use normalized exact match, aliases, numeric tolerances, or structured fields. Preserve the raw answer and parse status so formatting failures stay visible.
Transfer
Explanation or scenario
Use an observable rubric with required claims, prohibited errors, and supporting evidence. Blind reviewers to system identity and measure agreement on an overlapping sample.
Current evidence
Grounded answer
Supply or retrieve versioned sources. Score answer correctness, citation entailment, source authority, and date compliance as separate fields.
Closed-book and open-book results answer different questions. Closed book probes what the model can produce from its parameters and prompt; open book evaluates the whole system's ability to find and use supplied evidence.
Score slices before computing the headline
A slice is a named subset whose performance matters independently, such as a domain, language, operation, difficulty level, source version, or error consequence. Store one result row per attempted item, including failures and abstentions.
Exact
Deterministic checks
Use answer keys, executable checks, schemas, units, or citation verification when correctness is mechanically decidable.
Calibrate
Judgment checks
Validate rubrics and automated graders against blinded human review; report disagreement by slice.
Bound
Sampling uncertainty
Report item counts and intervals for important slices instead of ranking close scores by point estimate alone.
Retain
Failure records
Keep refusals, parse errors, missing citations, tool failures, latency, and cost in the attempted-item denominator.
The interval in the example covers finite binary-item sampling under its assumptions. It does not cover ambiguous keys, prompt variation, model sampling, rubric disagreement, domain mismatch, or contamination; those require separate evidence.
Gate the intended use, not the leaderboard position
An evidence gate maps predeclared measurements to a bounded decision. Thresholds should become stricter as the consequence of a wrong answer rises.
A useful gate can require:
- a minimum aggregate score for broad coverage;
- a minimum item count and uncertainty-aware lower bound for every critical slice;
- limited sensitivity to reasonable prompt variants;
- acceptable human or expert agreement for rubric-scored items;
- item provenance and near-duplicate checks;
- no unresolved parser, tool, or data-access failures that change the interpretation.
Missing validity evidence is not a pass. Mark it as unknown and investigate before the claim is expanded.
Decide what the evidence supports
Loading evaluation evidence...
Loading the evidence model...
Prevent familiar questions from masquerading as knowledge
Benchmark contamination occurs when evaluation items, solutions, or close variants influence training, tuning, prompt selection, or grader development. It can make familiarity look like general capability.
- Keep final items protected and versioned; develop prompts and parsers on separate data.
- Record item origin, creation date, licensing, transformations, and exposure history.
- Search for exact and near-duplicate overlap against available training, tuning, retrieval, and benchmark corpora.
- Use fresh or rotating item families when model training data cannot be audited.
- Include controlled variants that preserve the concept while changing surface form.
- Treat unexpectedly large gains as an audit trigger, not automatic proof of improvement.
Contamination checks reduce one threat to validity; they do not prove an item is representative, correctly keyed, or resistant to grader gaming.
Operate the suite as a versioned measurement system
1 Instrument
Author and review
Write from the blueprint, verify source authority and dates, review ambiguity and cultural assumptions, and protect the final split.
2 Scorer
Calibrate scoring
Build deterministic fixtures or a rubric set, measure reviewer agreement, adjudicate disputes, and pin parser and grader versions.
3 Protocol
Run and preserve
Execute the frozen model, prompt, retrieval, tools, and limits; retain raw outputs, traces where appropriate, errors, costs, and exact configuration.
4 Evidence
Analyze and decide
Review aggregate and slice results, uncertainty, prompt sensitivity, contamination, failures, and claim limits before applying the release gate.
Re-run golden item, parser, and scorer fixtures whenever the model, prompt, retrieval index, tool permissions, rubric, or evaluation code changes. A benchmark is a maintained measurement product, not a static folder of questions.
Read the primary guidance and benchmark designs
- NIST AI 800-2, Practices for Automated Benchmark Evaluations of Language Models is a January 2026 initial public draft covering evaluation objectives, validity, contamination, protocols, uncertainty, reproducibility, and qualified claims.
- NIST AI RMF 1.0 frames valid and reliable measurement as part of test, evaluation, verification, and validation.
- HELM uses explicit scenario and metric taxonomies, standardized protocols, raw predictions, and transparent coverage gaps.
- Measuring Massive Multitask Language Understanding introduced MMLU across 57 academic and professional subjects and showed why per-subject results matter alongside an average.
- MMLU-Pro increased question difficulty and answer choices, removed noisy items, and measured prompt stability to improve discrimination among stronger models.
- LiveBench illustrates a frequently refreshed, broad benchmark design intended to reduce test-set contamination.