Mathematical Reasoning: MATH & GSM8K

Build evidence from tasks to a claim

A benchmark result is the end of a measurement pipeline. Each stage can change the score, so each stage belongs in the run manifest.

1. 1

   Dataset

   Freeze the task manifest

   Record the dataset revision, split, task IDs, subject and difficulty metadata, exclusions, and contamination policy.

2. 2

   Inference

   Generate under one contract

   Pin the prompt template, model revision, decoding settings, candidate count, token budget, tool access, and retry policy.

3. 3

   Scoring

   Extract and verify

   Parse the declared final-answer field, normalize only allowed forms, compare under explicit mathematical assumptions, and retain parse failures.

4. 4

   Claim

   Aggregate with slices

   Report denominators, accuracy, uncertainty, subject and difficulty slices, cost, exclusions, and verifier failures together.

Minimum run manifest

• Task identity: dataset source, immutable revision, split, task IDs, deduplication result, and any excluded examples.
• Generation contract: exact prompt, demonstrations, model build, temperature, top-p, seed policy, candidate count, and maximum output tokens.
• Answer contract: required final-answer delimiter, parser version, allowed notation, units policy, variable assumptions, and timeout behavior.
• Reporting contract: micro or macro aggregation, slice definitions, confidence interval method, cost, latency, and failure taxonomy.

Use MATH and GSM8K for different questions

Read a MATH item as a structured task

Consider an angle-bisector problem where AB = 12, AC = 8, and BC = 15. If the bisector from A meets BC at D, the theorem gives BD / DC = 12 / 8 = 3 / 2. Since BD + DC = 15, the two parts are 9 and 6, so BD = 9.

The final answer tests correctness under the answer contract. The worked solution can support a separate diagnostic rubric, but it is not the only valid derivation.

Read a GSM8K item as a translation task

If 16 eggs are produced, 3 are eaten, 4 are used for baking, and each remaining egg sells for $2, the executable relation is (16 - 3 - 4) x 2 = 18. A useful error taxonomy separates:

• Interpretation error: the response adds consumed eggs instead of subtracting them.
• Planning error: the response multiplies before calculating the remaining inventory.
• Execution error: the selected operations are right, but an intermediate arithmetic step is wrong.
• Answer-contract error: the mathematics is right, but extraction or formatting fails.

Separate answer correctness from trace diagnosis

Final-answer scoring and trace review answer different questions. A symbolic checker can recognize equivalent expressions, but broad simplification can also introduce false positives when domains or variable assumptions are missing. A correct final answer can coexist with a broken visible derivation, while a valid derivation can still end with a transcription mistake.

Use the lab to change the response case and judging policy. Watch the answer verdict, trace verdict, evidence claim, and failure classification change independently.

Define a fail-closed scoring contract

Extract before comparing

Require a machine-readable final field such as FINAL: ... or the dataset's documented delimiter. If extraction fails, record parse_failure; do not search the whole explanation for a convenient number. The response may contain examples, intermediate values, or a correction after an earlier answer.

Normalize only declared syntax

• Remove presentation-only wrappers such as a documented \boxed{...} marker.
• Normalize commas, surrounding whitespace, and an explicitly allowed currency or unit notation.
• Parse integers, decimals, and fractions into an exact numeric representation where possible.
• Use a symbolic parser only with a restricted grammar, explicit variables and assumptions, time limits, and retained parse errors.
• Keep incomplete sets, intervals, units, and multiple roots semantically distinct from a single scalar.

Score the trace separately

If the product needs an explanation, define a human or programmatic rubric for stated assumptions, valid transformations, arithmetic, completeness, and the final link from derivation to answer. Call this a visible trace diagnostic, not a measurement of hidden internal reasoning.

Design the benchmark portfolio around the claim

MATH and GSM8K sample different task distributions. A tutoring assistant, a competition solver, and a general-purpose assistant therefore need different mixtures and verification depth. The next lab turns those choices into a visible coverage and release decision.

Change the product claim, task allocation, sample size, contamination estimate, and verification policy. The task counts, uncertainty estimate, evidence quality, blind spots, and release status update together.

Report a result without hiding its limits

Publish the score as a vector

1. Overall final-answer accuracy with the exact denominator and an uncertainty interval.
2. MATH accuracy by subject and difficulty, including empty or excluded slices.
3. GSM8K accuracy plus interpretation, planning, execution, and answer-contract failure rates.
4. Parse failures, symbolic-checker timeouts, invalid outputs, refusals, and dropped tasks.
5. Prompt tokens, generated tokens, latency, candidate count, verifier cost, and tool usage.

Add robustness checks outside the original headline

• Reword problems without changing their mathematical meaning.
• Change names, units, values, or surface order while preserving the solution structure.
• Add distractors and boundary cases that reveal shortcut behavior.
• Rotate independently authored holdouts and search for train-test or prompt-library overlap.
• Compare answer-only performance with product-specific visible-trace or tool-use requirements.

Implement the evaluator as an auditable boundary

The example below keeps extraction, parsing, equivalence, and aggregation separate. It uses exact rational arithmetic for supported numeric answers, returns explicit failure classes, and never treats a visible solution trace as proof of hidden reasoning.

Operational checks

• Version the parser and run it against known-equivalent, known-inequivalent, malformed, unit-bearing, set-valued, and adversarial fixtures.
• Retain raw output, extracted final field, normalized value, parser status, task metadata, and scorer version for every item.
• Alert on score changes together with parse-failure rate, slice regressions, output length, verifier timeout rate, contamination findings, and cost.
• Manually audit a bounded sample of passes and failures after any prompt, parser, model, or dataset change.
• Keep benchmark test items and answer keys out of prompt development, fine-tuning, retrieval indexes, and interactive debugging logs.

Ground the claims in the original releases

• The MATH paper introduces 12,500 competition mathematics problems with worked solutions.
• The GSM8K paper introduces the grade-school math word-problem benchmark and studies verifier-based selection.
• Use immutable dataset revisions and preserve each release's license, schema, split, and scoring protocol in the run manifest.

Public benchmarks are useful regression instruments. Release confidence comes from combining them with independent, product-shaped tasks and a verifier whose behavior is itself tested.