Code Generation: HumanEval & MBPP

Turn a prompt into executable evidence

The test suite, not surface similarity to a reference solution, decides whether a candidate passes. A defensible run records enough detail to reproduce both the number and every exclusion.

1. 1

   Contract

   Freeze the manifest

   Version the dataset or product task, prompt, entry point, language image, dependencies, hidden-test policy, decoding settings, and timeout before sampling.

2. 2

   Generate

   Sample candidates

   Keep every completion, seed, token count, model revision, and candidate index. Do not silently resample failures or select a best-looking answer.

3. 3

   Sandbox

   Contain execution

   Parse and execute each candidate in a disposable isolated environment with no ambient credentials, bounded CPU, memory, output, files, processes, and network.

4. 4

   Report

   Score and retain evidence

   Run public and protected tests, retain outcomes and failure classes, calculate the declared pass@k estimator, then report slices, uncertainty, cost, and exclusions.

What belongs in the run manifest

• Task contract: benchmark revision, task IDs, prompt template, function signature, allowed imports, and public versus protected test identities.
• Generation contract: model build, temperature, top-p, maximum tokens, seed policy, candidates per task, and any repair loop.
• Execution contract: immutable runtime image digest, package allowlist, CPU and memory limits, wall-clock timeout, filesystem policy, and network policy.
• Scoring contract: whether every test must pass, the pass@k values, aggregation weights, dropped-task policy, and interval method.

Design the pass@k claim before sampling

For a task with n generated candidates, of which c pass all tests, pass@k estimates the chance that a uniformly chosen set of k candidates contains at least one passing solution. The common unbiased estimator is 1 - C(n - c, k) / C(n, k). It is only meaningful when the candidates, task set, and verifier match the declared protocol.

The lab uses synthetic task slices. Change the observed samples, task mix, verifier strength, timeout, and aggregation rule. It shows why a larger k can improve opportunity while also increasing execution cost, and why visible examples alone cannot support a claim about unseen behavior.

Read pass@k without hiding the denominator

What the metric answers

• pass@1: How often the first sampled completion passes the full declared suite. This is the closest view of an assistant that offers one suggestion.
• pass@k: How often at least one of k samples passes. It measures an assisted search budget, not the quality of the first answer.
• Per-slice pass@k: Which task families benefit from sampling and which remain weak. A macro average prevents a large, easy slice from silently dominating.

What to publish beside the score

1. Candidate count per task, number of tasks, exact k, model and decoding configuration.
2. Benchmark revision, public and protected test policy, timeout, environment digest, and package policy.
3. Aggregate formula plus HumanEval, MBPP, and product-shaped slice scores with denominators and uncertainty.
4. Median and tail execution time, timeout rate, syntax and runtime failures, token and evaluator cost, and any omitted tasks.

Use HumanEval and MBPP as narrow lenses

Limitations that change the claim

• Small and public collections: model pretraining, tuning data, prompt libraries, and benchmark discussions can overlap with tasks or near-duplicates. Measure and disclose contamination; do not call an exposed task an independent holdout.
• Incomplete behavioral coverage: a passing test suite proves only the properties it asserts. Add metamorphic, boundary, adversarial, performance, and integration tests for the product behavior at stake.
• Single-language, function-level shape: Python snippets omit multi-file edits, build systems, configuration, APIs, database migrations, code review, and collaboration constraints.
• Sampling sensitivity: temperature, prompt scaffolding, repair loops, and candidate selection can move pass@k. Predeclare these choices and apply them evenly.

Gate execution, security, and evidence separately

A correct result from an unsafe evaluator is not a valid release result. This gate models a candidate runner: choose the access boundaries and evidence conditions, then inspect the execution path, risk, evidence quality, and release consequence.

The secure default is deliberately restrictive: a pinned local runtime, read-only task files, disposable writable scratch space, no network, no host mounts, no ambient dependencies, protected tests, stable reruns, and a clean holdout. Relaxing one boundary may be necessary for a product scenario, but it must be explicit and independently reviewed.

Implement a minimal evaluator boundary

The example orchestrates a runner rather than pretending that Python resource limits create a complete sandbox. In production, the runner needs a hardened isolation technology appropriate to the platform, an image allowlist, an egress policy, a process supervisor, and an audited artifact store.

Operational checks for every run

• Pin the evaluator image by digest and verify the runner has no production credentials or mounted source repository.
• Record task, candidate, image, policy, test-suite version, start and end time, exit reason, resource use, and redacted failure class in an append-only result record.
• Alert on changes in pass@1, timeout rate, protected-test gap, sandbox-policy violations, contamination findings, and evaluator flakiness.
• Keep a baseline model and a bounded canary path. Offline pass@k earns controlled exposure, not an unrestricted deployment.

Choose evidence that matches the product

For a code-completion feature, measure pass@1, latency, acceptance, and protected-test behavior on representative local edits. For an agent that edits repositories, add multi-file tasks, build and test outcomes, diff size, rollback behavior, dependency changes, secret scanning, and reviewer burden. For security-sensitive automation, add adversarial prompts, permission-boundary tests, and human approval gates.

The practical trade-off is simple: public benchmarks are cheap, stable regression signals; product-shaped, independently held-out tasks are slower and more expensive, but they support the release claim users actually rely on. Use both, and do not let a single leaderboard number decide deployment.