Composite Benchmarks: MMLU, MMMU & AGIEval

Turn questions into a reproducible result

An MMLU run is simple enough to audit, but small protocol changes can move the result. Treat the prompt and scorer as versioned evaluation code rather than incidental strings.

1. 1

   Manifest

   Freeze the protocol

   Record dataset revision, split, subject list, prompt template, zero-shot or few-shot setup, answer ordering, model version, and inference settings.

2. 2

   Inference

   Generate one choice

   Ask each test question without exposing its answer. Use deterministic decoding when possible and preserve the raw output for extraction audits.

3. 3

   Evidence

   Score every item

   Extract exactly one valid option, compare it with the reference, and retain subject, category, correctness, and invalid-response counts.

4. 4

   Decision

   Report the distribution

   Publish the declared aggregate beside per-subject scores, denominators, confidence intervals, prompt variants, and contamination findings.

Protocol choices that belong in the result

• Prompt regime: zero-shot, few-shot, and chain-of-thought prompts are different evaluations. Demonstrations must come from an allowed development split.
• Answer extraction: a strict extractor should distinguish B from prose that merely mentions option B. Invalid outputs need their own count.
• Answer ordering: option positions can create bias. Freeze ordering or run a declared permutation study rather than changing it between candidates.
• Aggregation: item-weighted, subject-macro, category-macro, and product-weighted scores answer different questions. Name the formula and publish its weights.
• Uncertainty: retain the numerator and denominator for every reported slice. A point estimate without n hides how much evidence supports it.

See how an overall score can hide a domain failure

The lab uses synthetic category scores for two model candidates. Broad Generalist is steady across domains. Headline Optimizer wins high-volume categories but fails a professional-domain floor. Change the category weights and the minimum acceptable professional score.

The purpose is not to recreate an official MMLU leaderboard. It is to make the aggregation contract visible: a weighted average can improve even while the domain that matters most to a product becomes unsafe or unusable.

Choose the summary for the decision

There is no neutral average. Each aggregation rule assigns influence to different evidence.

For model selection, report at least four views:

1. The standard benchmark score under a frozen, published aggregation.
2. Per-subject and per-category accuracy with correct counts and denominators.
3. Product-relevant slices with predeclared weights and hard minimums.
4. The difference between candidates with paired or bootstrap uncertainty where the evaluation design supports it.

A model can be the benchmark winner and still be ineligible for a product. The aggregate ranks candidates under one formula; the hard floor protects a requirement that must not be traded away.

Know what the benchmark does and does not prove

Multiple-choice accuracy can come from knowledge, elimination, memorized wording, answer-position cues, or genuine reasoning. MMLU alone cannot identify which mechanism produced a correct choice. Use targeted counterfactuals, generated variants, rationale checks, and product tasks when the mechanism matters.

Use MMLU, MMMU, and AGIEval as different lenses

These benchmarks are often grouped because each spans many tasks. They do not establish the same claim.

MMLU is broad and text-only

• Covers 57 subjects with four-option questions drawn from academic and professional material.
• Supports clean accuracy calculations and subject-level comparisons.
• Does not require visual interpretation or open-ended response quality.

MMMU adds multimodal evidence

• Uses questions where images, diagrams, charts, or other visual material are necessary.
• Tests whether visual perception and language reasoning work together across disciplines.
• Adds failure sources such as image preprocessing, resolution, layout understanding, and visual grounding.

AGIEval uses exam-shaped tasks

• Collects questions from standardized and professional examinations across languages and task formats.
• Tests performance on formal human assessment conventions.
• Can reward familiarity with exam structure and still miss interactive product behavior.

Separate weak evidence from a real slice failure

An aggregate can look strong for four different reasons that do not justify release: a critical slice has too few items, the holdout overlaps with training or prompt development, results swing across valid prompt variants, or the release floor was chosen after seeing the score.

The next lab keeps the observed overall and worst-slice accuracies visible while you change:

• independent sample size, which changes random uncertainty;
• estimated contamination, which weakens the generalization claim;
• prompt-run variance, which penalizes an unstable protocol;
• the predeclared slice floor, which changes the release contract.

More samples can narrow an interval. They cannot remove contamination, stabilize a flawed prompt, or make a below-floor slice acceptable.

Implement a denominator-aware evaluation boundary

The example below scores item records, preserves per-subject numerators and denominators, computes Wilson intervals, applies declared category weights, and returns pass, hold, or block for a critical slice. It uses only the Python standard library so the reporting logic is easy to audit.

Keep the runner and the policy separate:

• The runner owns prompts, model calls, raw outputs, extraction, and immutable item records.
• The report owns grouping, aggregation, uncertainty, contamination notes, and protocol sensitivity.
• The release policy owns weights, hard floors, minimum denominators, allowed contamination, rollout ceiling, and approver.
• A changed dataset, prompt, extractor, model, or policy creates a new versioned result rather than silently replacing the old one.

Operate MMLU as one signal in an evaluation system

Rerun and investigate when

• the model, system prompt, decoding settings, answer ordering, extractor, or few-shot examples change;
• dataset overlap, leaked answer keys, suspiciously exact rationales, or abrupt subject jumps appear;
• a product-critical subject regresses even when the overall score rises;
• valid prompt variants disagree beyond the declared stability threshold;
• user traffic, supported languages, tools, retrieval, or safety policy changes the deployed task.

Preserve an auditable release record

Store the manifest, item hashes, raw outputs, parser version, aggregate formula, weights, slice counts, confidence method, contamination audit, prompt sensitivity, failed and waived gates, approver, rollout ceiling, monitoring plan, and rollback trigger.

Passing MMLU should earn another evaluation step, not a certificate. A defensible release still needs product-shaped tasks, open-ended quality and factuality checks, safety and fairness gates, latency and cost measurements, and a bounded canary with monitoring.