Common Sense Reasoning: PIQA, HellaSwag & More

What are common-sense reasoning benchmarks?

Common-sense reasoning benchmarks are controlled question sets that probe whether a model can choose a plausible conclusion using everyday physical, social, causal, temporal, or relational knowledge. They usually present a short context and a small set of answer choices, which makes scoring repeatable but narrows what the result can prove.

They matter because fluent text can still violate basic expectations: a tool may not fit the task, an event continuation may be impossible, or a pronoun may resolve to the wrong person. Invariant: a benchmark score supports only the capability, item distribution, answer format, and evaluation protocol that produced it. It does not establish general common sense or product readiness by itself.

Separate the capability from the multiple-choice format

Each benchmark isolates a different slice of common sense. Use the task shape to choose evidence; do not blend the names into one interchangeable score.

PIQA

Physical affordances

Choose which of two methods can physically achieve an everyday goal. The random-guess baseline is 50%.

HellaSwag

Likely continuations

Choose a plausible continuation from four options, including machine-generated adversarial distractors. Chance accuracy is 25%.

WinoGrande

Referent resolution

Resolve a blank to one of two entities when syntax is insufficient and context changes the answer. Chance accuracy is 50%.

SocialIQA

Social inference

Infer a person's likely intent, reaction, need, or emotion from a short social situation. Three choices imply a 33.3% chance baseline.

OpenBookQA

Science plus inference

Combine one of 1,326 core science facts with additional common knowledge to answer a four-choice question.

CommonsenseQA

Concept relations

Answer five-choice questions built from ConceptNet relations and human-authored contexts. Chance accuracy is 20%.

Read every score with four labels

Dataset and split: name the exact version, split, exclusions, and item IDs.
Prompt and decoding contract: preserve answer labels, few-shot examples, system instructions, and extraction rules.
Baseline: report chance accuracy and a named previous model under the same protocol.
Scope: state the capability slice the benchmark probes and the product behavior it does not cover.

Build a portfolio around the product risk

A benchmark portfolio should spend its evaluation budget on the failures that matter for the intended product. Choose a product, sample budget, and prompt contract below. The allocation, chance-correct count, critical-slice coverage, runtime, and decision update together.

Portfolio allocation lab

Spend evaluation effort where failure matters

Choose a product and protocol, then resize the sample. The benchmark mix, chance baseline, coverage, runtime, and evidence status move together.

Loading benchmark portfolio...

Interpret the allocation before running a model

Critical coverage protects the benchmark most closely tied to the product's highest-cost common-sense failure.
Covered product weight counts only benchmark slices with enough sampled items to support useful diagnosis.
Expected chance-correct items depends on the number of answer options; raw accuracy should never be compared without that baseline.
Protocol status becomes diagnostic when a product prompt replaces a frozen comparison prompt. That run may be relevant, but it is no longer directly comparable to a public result.

The item counts are an evaluation-plan model, not permission to sample a hidden test set repeatedly. Draw a versioned, stratified subset once, keep it independent from prompt development, and report the sampling rule.

Understand how the datasets were constructed

Dataset construction determines which shortcuts a model may exploit and which conclusions remain defensible.

1
Claim
Define a narrow capability
PIQA targets physical interactions; SocialIQA targets social implications. A narrow capability makes failures interpretable.
2
Items
Author contexts and choices
Humans or generators create a context, one intended answer, and plausible distractors under a documented procedure.
3
Quality
Filter artifacts and ambiguity
Adversarial filtering, bias reduction, and independent review remove easy lexical cues and disputed gold labels.
4
Evidence
Freeze an independent split
Versioned prompts, protected items, and reproducible scoring preserve a comparison that tuning has not consumed.

Construction choices worth knowing

PIQA contains 16,113 training and 1,838 validation examples in the commonly distributed configuration. Each item pairs a goal with two proposed solutions.
HellaSwag contains roughly 70,000 examples. Its adversarial filtering procedure selects generated distractors that were difficult for contemporary discriminators while remaining easy for people.
WinoGrande scales the Winograd-style task to about 44,000 problems and uses AfLite to reduce embedding-detectable associations. Different published training-size variants must not be mixed silently.
SocialIQA contains about 38,000 three-choice questions about motivations, reactions, and social states. Its authors designed distractor collection to reduce simple stylistic cues.
OpenBookQA contains 5,957 four-choice science questions and a small open book of 1,326 facts, but many answers require additional unstated common knowledge.
CommonsenseQA contains 12,247 five-choice questions constructed from ConceptNet relations and human-authored contexts.

The original benchmark papers report much larger human-model gaps than current systems may show. Treat those numbers as historical construction evidence, not current leaderboards.

Run one frozen scoring contract

Multiple-choice scoring is simple only after the protocol is explicit. The evaluator should reject malformed outputs, preserve case IDs, aggregate by benchmark and product slice, and retain enough evidence to reproduce every result.

Versioned multiple-choice evaluation harness

Store with every prediction

Dataset, configuration, split, item ID, license-compatible content hash, and contamination-screen result.
Model and adapter identity, prompt template, demonstrations, decoding settings, and answer-extraction version.
Raw response, extracted label, gold label, correctness, latency, token use, and failure reason.
Capability slice, product slice, language or culture slice, and reviewer notes for ambiguous cases.
Aggregate definition, confidence interval, critical-slice floor, comparison baseline, and release decision.

Use macro averages when each capability should influence the report equally. Use product weights only when they were declared before seeing the candidate result. Always show the per-benchmark scores beside either aggregate.

Challenge shortcuts before trusting the score

A model can improve by learning answer length, lexical overlap, entity frequency, or familiar wording instead of the intended relation. A counterfactual keeps the underlying answer logic fixed while removing or reversing one incidental cue.

Counterfactual shortcut audit

Change the cue without changing the answer logic

Choose a suspected shortcut, perturb its strength, and set the review policy. The paired score, flip rate, robustness gap, and release gate reveal whether the original result survives.

Loading counterfactual audit...

Diagnose the gap, not just the lower number

Pair the items. Compare each original with a meaning-preserving perturbation so the difference has an interpretable cause.
Preserve the answer contract. Relabel choices after shuffling and verify the counterfactual with an independent human or deterministic rule where possible.
Measure paired stability. Report original accuracy, counterfactual accuracy, and the percentage of pairs that flip from correct to incorrect.
Set the gate first. Declare an acceptable robustness gap and critical-slice floor before evaluating the release candidate.
Repair the evidence. Remove ambiguous items, rotate protected challenge sets, and add product-shaped cases rather than tuning repeatedly against the same public examples.

Paired counterfactual stability audit

Turn benchmark failures into product evidence

A wrong multiple-choice answer becomes useful only after it changes the evaluation or system design.

Frozen protocol

Benchmark run

Record raw outputs and per-slice results.

Cause, not label

Failure review

Separate missing knowledge, invalid inference, ambiguity, extraction error, and shortcut behavior.

Representative case

Product challenge

Recreate the failure with the product prompt, tools, language, and user consequences.

Bounded decision

Release gate

Require critical slices, robustness checks, and a monitored canary before widening exposure.

Operate the suite at three cadences

Per change: run a small protected regression sample with frozen extraction and critical product cases.
Before release: run the full versioned portfolio, contamination checks, counterfactual pairs, uncertainty estimates, and independent review of disputed items.
After launch: sample real failures, measure them by product slice, and promote validated cases into a future protected set without leaking that set into tuning.

A strong public score earns a more specific experiment. It does not erase missing cultures, languages, modalities, open-ended behavior, tool use, or real-world consequences.

Use the original papers as protocol documentation

Read the construction and evaluation sections before reproducing a score. Dataset wrappers and leaderboards can simplify or silently change prompts, splits, and answer extraction.

No quiz questions available

Could not load questions file