Composite Benchmarks

Distinguish a composite score from a dashboard

Both views can use the same underlying results. They answer different questions.

Use a scalar when the combination rule is part of the decision contract. Use a dashboard when dimensions are not meaningfully substitutable. A safety violation rate should usually be a hard gate, not a number that high accuracy can cancel.

Start with the capability claim and decision

A benchmark cannot be “holistic” in the abstract. Write the supported claim before choosing tasks: for example, “compare two text models for multilingual support triage under this prompt and tool policy.” Also write the claims the benchmark does not support.

1. 1

   Claim

   Bound the decision

   Name the model behavior, users, languages, modality, environment, and release action the evidence should support.

2. 2

   Coverage

   Map required capabilities

   Translate the claim into independently useful capabilities, critical subgroups, failure modes, and operational constraints.

3. 3

   Tasks

   Choose evidence

   Select tasks with known provenance, scoring validity, enough cases, and limited redundancy. Record known gaps.

4. 4

   Policy

   Freeze interpretation

   Predeclare normalization, weights, hard floors, uncertainty, missing values, exclusions, and the release rule.

A defensible scope statement includes

• the behavior being measured and the user or workload population it represents;
• the dataset revision, prompt, tools, decoding, scorer, and model version;
• the decision the result can influence, such as experiment selection or a bounded canary;
• explicit exclusions, such as unsupported languages, open-ended factuality, or real production latency;
• hard requirements that cannot be traded against unrelated strengths.

Select tasks for coverage, not task count

More tasks do not guarantee broader evidence. Begin with a coverage map, then ask what independent signal each proposed task adds.

For every component, record

• Capability: the behavior it is intended to measure.
• Population and slice: users, domains, languages, difficulty bands, and failure modes represented.
• Protocol: prompt, adaptation method, scorer, sample unit, and allowed tools.
• Provenance: source, collection period, license, known exposure, and version.
• Uncertainty: item count, sampling unit, variance, and whether candidate comparisons are paired.
• Relationship to other tasks: shared examples, authors, templates, scorers, or latent skills that can create correlation.

Coverage is a matrix, not a count. Ten English multiple-choice tests can still provide no evidence about multilingual conversation, tool use, image understanding, or production recovery.

Normalize direction and scale before weighting

Raw component values can use incompatible units and favorable directions: accuracy rises with quality, while latency and harmful-output rate improve when they fall. Define a baseline and reference for each component before combining them.

One transparent teaching transform is:

• higher is better: (observed - baseline) / (reference - baseline);
• lower is better: (baseline - observed) / (baseline - reference);
• clamp or otherwise handle out-of-range values according to a predeclared policy.

The baseline might be random performance, an existing production model, or a minimum useful level. The reference might be a validated target or meaningful upper anchor. Neither should be selected after seeing which candidate benefits.

The next lab uses synthetic values with mixed units. Change only the policy and watch the ranking move.

Match the aggregation rule to the claim

There is no universally correct mean. Each rule answers a different question.

Keep several views when the decision needs them:

1. A canonical benchmark aggregate for stable historical comparison.
2. A product-weighted view tied to observed or planned workload.
3. Hard floors for non-exchangeable safety, legal, quality, or subgroup requirements.
4. The complete component profile with denominators and uncertainty.

If two candidates trade wins across components, a Pareto view may be more honest than forcing one total order.

Make baselines, weights, and missing values policy

Weights are not discovered by the benchmark score. They come from a use case, risk model, stakeholder contract, or declared equal-treatment rule.

Weighting checks

• Normalize weights so their effective shares are visible.
• Explain whether weights represent item frequency, equal capabilities, product traffic, risk, or another policy.
• Stress-test plausible alternatives and report rank reversals.
• Do not estimate weights on the same candidates being ranked.
• Keep hard requirements outside the compensatory average.

Missing-value policies

• Block: require every component before computing a decision score. This is safest for must-pass evidence.
• Score at baseline: apply a documented penalty. This avoids rewarding non-participation but can be harsh when data is missing for unrelated reasons.
• Renormalize available weights: compare on shared evidence only. This changes the meaning of the score and must show a coverage ratio.
• Report no rank: preserve the partial profile until candidates have comparable evidence.

Never treat missing as zero silently, and never let an easier subset appear superior to a complete evaluation.

Quantify uncertainty and ranking sensitivity

A decimal-heavy aggregate is not precise merely because it has many components. Tasks may share items, templates, annotators, prompts, or scorers, so their errors are correlated.

Report at least

• component numerators, denominators, point estimates, and intervals;
• paired candidate differences when both candidates answer the same cases;
• a composite interval produced by resampling the full scoring pipeline at the independent sampling unit;
• rank frequency or rank intervals across resamples when rank is a decision input;
• sensitivity to predeclared weights, baselines, references, exclusions, and prompt variants.

Do not combine component confidence intervals as if components were independent unless that assumption is justified. A bootstrap should resample clusters together when examples share a source, document, user, or template.

Treat correlation as repeated influence

Two tasks can have different names and still measure nearly the same thing. Correlation can arise from shared capabilities, data sources, question templates, judges, or training exposure.

Audit redundancy in three passes

1. Design audit: group tasks by intended capability, provenance, format, scorer, and sampling unit before running models.
2. Empirical audit: inspect candidate-level and item-level associations with enough models and uncertainty to avoid overreading small samples.
3. Decision audit: remove or down-weight one component at a time and check whether the ranking or release decision changes.

High correlation does not prove two tasks are interchangeable, and low correlation does not prove both are valid. Use correlation as one diagnostic beside content review and failure analysis.

Build an evidence portfolio for the release claim

The next lab separates selected task count from independent coverage. Add or remove synthetic evaluations, switch the candidate and release decision, and decide whether a contamination-flagged task can contribute evidence.

The release gate uses conservative capability slices and hard floors. Its headline average remains visible so you can see why “best aggregate” and “eligible candidate” are different statements.

Preserve subgroup and failure slices

An aggregate can rise while the product becomes worse for a critical population or failure mode. Keep slices that connect directly to the claim.

Useful slices include

• supported language, region, accessibility need, or demographic group;
• task subtype, difficulty band, input length, modality, and tool path;
• safe refusal, over-refusal, factual error, hallucinated citation, and malformed output;
• fresh versus potentially exposed data;
• normal traffic, degraded dependency, timeout, retry, and recovery behavior;
• latency and cost percentiles rather than only their averages.

Predeclare critical slices and minimum evidence. If a slice is too small, return inconclusive instead of silently merging it into the aggregate. If a hard slice fails, report a failure even when the weighted mean leads.

Learn from MMLU, BIG-bench, and HELM without copying a headline

These projects illustrate different ways to organize broad evaluation. Their historical counts describe published versions, not permanent definitions.

MMLU: many subjects under one test format

The original MMLU paper introduced 57 academic and professional tasks and reported multitask accuracy. It is useful for broad text multiple-choice comparison under a frozen protocol. Its subject-level results matter because the paper itself found lopsided performance and near-random behavior on some socially important subjects.

BIG-bench: a large collaborative task collection

The BIG-bench repository describes more than 200 tasks and a 24-task BIG-bench Lite subset. Each task declares a preferred score, which makes metric selection part of the task contract. The BIG-bench paper analyzes aggregate and per-task behavior; retain the latter when a task drives the claim. The repository also places canary strings in task files to help detect web-scraped leakage.

HELM: broad coverage with multiple metrics

The original HELM paper evaluated seven metrics across 16 core scenarios when possible and explicitly documented missing or underrepresented areas. Its purpose was to expose trade-offs rather than let accuracy erase robustness, fairness, bias, toxicity, calibration, or efficiency.

The current HELM site organizes distinct capability, safety, modality, and domain leaderboards. HELM Lite used mean win rate to compare scenarios with different units, while warning that the value depends on the comparison set and that the scenarios do not cover every capability. That is a concrete reminder that aggregation semantics and cohort dependence belong in the report.

Version the full evaluation contract

Create a new result version when any of these change:

• task data, split, sample, reference answers, or contamination exclusions;
• prompt, few-shot examples, tool access, inference settings, or model endpoint;
• parser, judge, rubric, normalization anchors, weights, or aggregation;
• capability claim, supported population, release floor, or comparison cohort.

Do not splice scores from incompatible versions into one leaderboard without an explicit bridge study.

Implement a small, auditable scoring boundary

The example below normalizes higher- and lower-is-better components against declared anchors, supports three aggregation rules, makes missing-value behavior explicit, and returns the normalized component profile with the final score.

Keep the surrounding system separate:

• The runner owns model calls and immutable item records.
• Component scorers own task-specific correctness and diagnostics.
• The composite layer owns only declared conversions and aggregation.
• The release policy owns hard gates, evidence minimums, rollout scope, and approvals.
• The report owns provenance, uncertainty, sensitivity, and retained component results.

Use the composite as evidence, not a certificate

A release-ready review should answer:

• What exact claim does the benchmark support?
• Which required capabilities and populations remain uncovered?
• Which tasks repeat the same signal or share contamination risk?
• Does the ranking survive reasonable normalization, weight, and cohort changes?
• Which component or subgroup failures are hidden by the aggregate?
• Are score differences larger than sampling and protocol uncertainty?
• Do all non-compensatory floors pass on independent evidence?
• Can the team reproduce the result and roll back the release?

Passing a composite benchmark can qualify a candidate for the next bounded step. It does not replace product evaluation, safety testing, canary monitoring, incident response, or human judgment about consequences the metric does not encode.