MapReduce

Follow one record through the execution graph

1. 1

   Plan

   Split immutable input

   Create logical input splits, assign record readers, and prefer workers near the HDFS blocks without coupling correctness to locality.

2. 2

   Transform

   Map records independently

   Parse each input record and emit zero or more typed intermediate key-value pairs without relying on mapper execution order.

3. 3

   Compress

   Combine locally when safe

   Apply associative and commutative partial aggregation to reduce mapper spill and network volume; never require a combiner for correctness.

4. 4

   Exchange

   Partition, shuffle, and sort

   Choose a reducer for every key, transfer partitions, merge spills, sort keys, and group values so equal keys meet.

5. 5

   Aggregate

   Reduce each key group

   Apply reducer logic to a key and its values, then write task-attempt output into a temporary commit location.

6. 6

   Publish

   Commit complete output

   Atomically promote only successful task or job output so retries and speculative attempts do not expose duplicates.

Size map waves, shuffle bytes, and reducer partitions

Input bytes alone do not determine duration. Split size controls task count, cluster slots control waves, mapper selectivity controls shuffle, and reducer count controls partition size and output files.

Read the envelope by phase

• Map: input splits divided by concurrent slots determine waves; startup and tail variation make tiny tasks expensive.
• Spill and combine: sort buffers, spill count, compression, and mathematically safe local aggregation determine disk and network work.
• Shuffle: intermediate bytes cross machines and racks while reducers fetch, merge, and sort mapper partitions.
• Reduce: partition bytes, key cardinality, skew, aggregation complexity, and output commit determine the tail.
• Recovery: failed attempts, speculative duplicates, node loss, queueing, and output cleanup consume reserve beyond the healthy path.

Use combiners only when partial aggregation is valid

A combiner is an optional local optimization. The framework may run it zero, one, or several times, so reducer correctness cannot depend on its execution.

Test the mapper, combiner, and reducer with different partition boundaries, value orders, duplicate attempts, and empty groups.

Treat shuffle as a distributed storage and network protocol

Mapper output is partitioned, buffered, spilled to local disk, optionally combined and compressed, then fetched and merged by reducers. Every reducer may contact every mapper.

Reduce exchange cost deliberately

• Filter and project early so the mapper emits only fields needed downstream.
• Use a safe combiner and intermediate compression after measuring CPU versus network cost.
• Size sort buffers to reduce spill count without causing garbage collection or memory contention.
• Prefer map-side joins only when the smaller relation can be distributed and versioned safely.
• Avoid chaining wide shuffles when a different engine or data layout better matches the computation.

Make retries and speculative execution side-effect safe

The runtime may execute a task more than once because of failure, timeout, preemption, or speculation. User code and output protocols must tolerate duplicate attempts.

• Keep mapper and reducer functions deterministic for the same versioned input whenever possible.
• Write only to attempt-scoped temporary paths, then let the output committer publish one winner atomically.
• Do not call external payment, email, or mutation APIs directly from retryable tasks without idempotency and reconciliation.
• Quarantine poison records explicitly instead of retrying the same deterministic failure until the job exhausts its budget.
• Enable speculative execution only when duplicate resource use is acceptable and task duration is comparable.

Choose data layout before tuning tasks

Input design

• Compact many small files into splittable containers or columnar files suited to the downstream engine.
• Partition directories by stable pruning dimensions without creating tiny partitions.
• Use splittable compression when many mappers must scan one large file.
• Preserve schemas, event-time meaning, and input snapshot identity for reproducible jobs.

Output design

• Choose reducer count with both partition work and downstream file size in mind.
• Use map-only jobs when no global key grouping is required.
• Write versioned output to a new location, validate it, then publish through an atomic pointer or rename contract.
• Retain job, code, configuration, input, schema, and output lineage for replay and incident analysis.

Know when MapReduce is the wrong engine

Operate from counters and tail behavior

Monitor every job

• Input records/bytes, split count, mapper and reducer waves, locality, queue time, and slot utilization
• Map output bytes/records, combiner input/output, spill count/bytes, merge passes, and intermediate compression
• Shuffle bytes, failed fetches, cross-rack traffic, reducer partition distribution, and straggler ratio
• Task failures by cause, attempts, speculative duplicates, killed tasks, poison records, and commit failures
• Output records/bytes/files, schema checks, data-quality assertions, and downstream publication state

Prepare recovery

• Re-run from immutable input with versioned code and configuration.
• Preserve failed-task diagnostics and bounded samples of malformed records.
• Test node loss, slow disks, unavailable racks, fetch retries, controller restart, and output-commit recovery.
• Clean abandoned temporary output and old shuffle data without deleting evidence needed for diagnosis.