Data Pipeline Design

Choose a processing mode from the product promise

Batch processes a bounded historical set on a schedule. Streaming evaluates events continuously. Hybrid keeps a durable batch truth while a streaming path maintains recent state. Start with the required freshness, not a preferred tool.

• Choose batch when hours of freshness are acceptable, large scans dominate, and simple reproducible reruns matter most.
• Choose streaming when an event must change a decision within seconds or minutes, such as fraud controls or session recommendations.
• Choose hybrid when serving needs recent events but training, reporting, or correction needs a complete recomputable history.

Ingest events without losing their meaning

An event usually has at least two useful clocks:

• Event time is when the business action happened. Use it for feature windows, labels, and point-in-time joins.
• Ingestion time is when the platform received it. Use it to observe source delay, transport outages, and replay progress.

Keep the raw event immutable and attach a source ID, schema version, event ID, entity key, and permission or retention context. An event can arrive late, out of order, or more than once. The pipeline should decide how to handle those cases explicitly instead of letting processor defaults decide.

Watermarks and late data

A watermark is a practical statement such as "we expect events through 10:00, allowing 15 minutes of lateness." A streaming job can close an event-time window after the watermark, then route very late records to correction or replay. The choice has a trade-off:

• A longer allowance improves correctness for delayed sources but postpones complete aggregates.
• A shorter allowance improves freshness but requires more corrections and makes downstream values temporarily incomplete.
• Never use ingestion time as a shortcut for event time in historical training joins; it can leak future information or omit valid late events.

Make data contracts executable

A data contract is an agreement between a producer and every downstream consumer. It includes more than column names: it states types, optionality, semantic meaning, ownership, delivery expectations, and compatibility rules.

Validate at each boundary

• Schema: required fields, types, allowed enumerations, and additive versus breaking changes.
• Quality: null rate, numeric range, duplicate rate, referential coverage, and business invariants.
• Freshness: maximum source delay, partition completeness, stream lag, and online feature age.
• Distribution: shifts in important feature values and slices, interpreted alongside product context.

Use three outcomes: accept valid records, quarantine repairable records with reason codes, and stop publication for changes that could silently corrupt features. A contract failure should expose an owner and a path to recovery, not only a red dashboard.

Generate features once, then test offline and online parity

A feature is a measurable input to a model, such as a customer's purchases in the prior 30 days. The feature definition must state the entity key, event-time window, source set, defaults, version, and whether it is safe at prediction time.

Avoid two independently maintained implementations of the same feature. Where one runtime cannot be shared, share a declarative definition and enforce replay-based parity tests.

Orchestrate runs and design backfills before an incident

Orchestration schedules dependencies, parameters, retries, and publication gates. A reliable workflow is idempotent: rerunning the same input interval produces the same output version rather than duplicate data.

1. 1

   Scope

   Plan the interval

   Pin input partitions, event-time bounds, schema and feature versions, and the target consumer before starting work.

2. 2

   Build

   Write a candidate

   Compute into an isolated version or staging location. Do not overwrite a serving view while validation is incomplete.

3. 3

   Gate

   Validate and compare

   Run contracts, row counts, distribution checks, and offline/online parity checks against the previous known-good version.

4. 4

   Release

   Publish atomically

   Promote the compatible feature or dataset version, retain lineage, and record exactly which consumers moved.

A backfill recomputes a bounded event-time range after a correction. Prefer the smallest range that repairs the error, but use a full replay when a feature definition or source history changed globally. Backfill both training snapshots and online materializations when the defect could create training-serving skew.

Operate the pipeline as a product

Observability should answer whether data is usable, not only whether a job exited successfully. Attach a run ID, source and feature versions, event-time interval, input counts, output counts, quarantine counts, and publication state to every materialization.

• Watch freshness and lag: source delay, watermark age, queue depth, and online feature age against an SLO.
• Watch correctness: contract failures, duplicate rate, late-event rate, join coverage, parity mismatches, and sampled feature values.
• Watch impact: affected models, feature consumers, prediction slices, training snapshots, and fallback use.
• Recover deliberately: pause publication, preserve raw inputs and checkpoints, scope the bad interval, rebuild a candidate, validate, then release in a reversible step.

Cost controls that preserve correctness

• Partition raw and derived data by event date or a similarly selective key; avoid scanning the whole history for routine work.
• Use incremental state where safe, but retain enough raw history to replay after a bug.
• Set retention and compaction policies per data class; cheap retained lineage is often less expensive than an unreproducible incident.
• Size stream workers from event rate, transform cost, state size, and peak replay traffic, not average throughput alone.

Make trade-offs explicit

• Freshness versus completeness: lower watermark delay improves responsiveness but accepts more corrections for late records.
• One path versus two: a hybrid design can meet distinct latency targets, but every extra implementation increases parity and operating cost.
• Strict gating versus availability: blocking a critical feature protects correctness; serving a documented fallback can protect users when a noncritical source fails.
• Incremental processing versus replayability: incrementality reduces routine cost, while immutable raw data and versioned transforms make corrections defensible.

The best pipeline is the simplest one that demonstrably satisfies data correctness, latency, and recovery promises for the ML decision it supports.