Apache Flink
Master Apache Flink: stateful stream processing, event time semantics, exactly-once processing, and production deployment.
What is Apache Flink?
Apache Flink is a distributed engine for computing continuously over bounded and unbounded data. A Flink job reads records from sources, transforms them through a parallel operator graph, keeps managed state where a result depends on history, and writes results to sinks.
Flink matters when a result must remain useful while events arrive late, machines fail, state grows, or traffic changes. Common examples include fraud rules, live product metrics, device monitoring, enrichment, and continuously updated materialized views.
Core invariant
A correct streaming result needs more than fast record handling. The design must make time, state ownership, recovery position, and external side effects explicit. Flink can coordinate these boundaries, but it cannot infer the business meaning of a timestamp or make an arbitrary external call exactly once.
Event
Carry business time
Event time records when the business event happened. Processing time records when an operator happened to handle it. Choose deliberately because failures and backlog alter processing time.
State
Partition by key
keyBy routes the same key to one logical state partition. Flink can redistribute those partitions when operator parallelism changes.
Recovery
Snapshot consistently
Checkpoints bind managed state to source positions. A failed job restores one completed snapshot and reprocesses input after that point.
Effect
Close the sink boundary
A supported transactional sink or an application-level idempotency key prevents replay from becoming a second business effect.
This lesson builds on Streaming Systems, especially durable logs, partitioning, delivery semantics, and backpressure.
Make event-time progress visible with watermarks
Event time is a timestamp carried by a record that describes when the event occurred. A watermark is Flink's progress signal: watermark t means the stream does not expect more records at or before t, although a late record can still violate that expectation.
An event-time window uses the watermark, not the wall clock, to decide when to emit. Watermark strategy and allowed lateness solve different problems:
When it happened
Event timestamp
Extracted from each record or source metadata
Progress estimate
Watermark
Advances when the source believes older event time is complete
State retention
Allowed lateness
Keeps a fired window available for accepted late updates
Explicit overflow
Side output
Preserves too-late records for review or compensation
Which records reach the window result?
Loading the event-time scenarios.
Loading event-time model
Preparing the arrival-order trace.
The lab is a whole-second teaching model. Before each arrival it computes a watermark from the largest event timestamp already observed minus the configured disorder bound. It then asks whether the record's window is still waiting, retained for late updates, or already eligible for cleanup. Production watermark emission is periodic or source-driven and is calculated independently by parallel source partitions.
Protect progress across source partitions
- Generate timestamps and watermarks at the source when partition-level information is available.
- Mark truly idle inputs as idle; otherwise one silent partition can hold back the minimum downstream watermark.
- Use watermark alignment when a fast partition would otherwise force downstream operators to buffer excessive state while slower partitions catch up.
- Route too-late records to an observable side output when silent loss is unacceptable.
- Treat a late firing as an update to an earlier result. The sink must upsert, retract, version, or deduplicate that update correctly.
See Flink's official documentation for watermark generation and window lifecycle and allowed lateness.
Match the window to the business boundary
A window assigner maps each record to one or more finite groups. The choice should describe the question the product asks, not merely which API is familiar.
One fixed bucket
Tumbling window
Fixed, non-overlapping intervals place each record in one window. Use them for metrics such as orders per calendar minute when aligned boundaries have business meaning.
Overlapping view
Sliding window
A fixed-size window advances by a smaller slide. One record can contribute to several results, which improves temporal resolution but multiplies state and update work.
Activity-defined
Session window
Records with the same key merge until an inactivity gap closes the session. A late record can bridge two sessions, so downstream results must tolerate corrections.
Choose with four questions
- Which timestamp defines membership? Use event time when replay must reproduce the same grouping.
- When is the first result useful? A larger watermark delay can improve completeness while postponing emission.
- How long may a correction arrive? Allowed lateness retains window state and can create later firings.
- Can the sink revise a prior result? If not, emitting early and correcting later violates the output contract.
Aggregate incrementally when possible. A reduce or aggregate function can keep a small accumulator, while a process function that stores every record makes state grow with window traffic. Evictors also prevent pre-aggregation because the operator must retain elements for inspection.
Keep state colocated with the records that change it
Managed state is application history whose lifecycle, partitioning, and snapshots are controlled by Flink. Keyed state is available after a keyed partitioning step and is accessible only for the current record's key.
1 Source
Read a durable position
The connector emits records with identities, timestamps, and positions that can be restored after failure.
2 Keyed exchange
Partition by business key
All updates for one account, device, or order reach the same logical keyed-state owner. Choose a key that avoids one hot tenant or value dominating a subtask.
3 Stateful operator
Update bounded state
The operator updates values, timers, windows, or aggregates. State TTL and cleanup policies bound history only when their semantics permit expiration.
4 Sink
Publish with a contract
The sink appends, upserts, or commits transactionally. Output identity and replay behavior must match the business effect.
Flink organizes keyed state into key groups, the atomic units redistributed during rescaling. Set maximum parallelism with future growth in mind, assign stable operator UIDs before relying on savepoint-based upgrades, and test that schema changes can read existing serialized state.
Choose the state backend from measured constraints
Heap-resident state
HashMapStateBackend
State access operates on Java heap objects and can be fast, but the state assigned to a TaskManager must fit its heap. It supports full snapshots rather than incremental ones.
Large local state
EmbeddedRocksDBStateBackend
Serialized state can extend onto local disk and supports incremental checkpoints. Access adds serialization and storage cost, so measure throughput and latency with the real key and value shapes.
The backend controls working-state representation. Checkpoint storage separately controls where durable snapshot data is written. Production checkpoints belong on a durable filesystem or object store, not only JobManager heap or ephemeral worker disk. See the official state backend guide.
Recover from the latest completed checkpoint
A checkpoint is an automatically coordinated snapshot of managed state and stream positions. Checkpoint barriers flow with records through the operator graph; a checkpoint becomes a recovery point only after all required tasks and storage actions complete successfully.
What happens when the job fails?
Loading the checkpoint scenarios.
Loading checkpoint model
Preparing the recovery timeline.
The lab assumes one durable baseline at t=0, periodic checkpoint triggers, one active snapshot at a time, and a fixed snapshot duration no longer than the interval. Its replay span is transparent: failure time minus the barrier time of the latest completed checkpoint. That span is not a recovery-time prediction.
Exactly once has an explicit boundary
Flink checkpoints can make managed state and replayable source positions consistent as if a failure-free execution occurred. End-to-end exactly-once output additionally needs a sink that participates in checkpoint coordination, such as a supported transactional connector, or an external operation made idempotent by a stable business key.
- A retried record can execute user code more than once during recovery.
- A checkpoint that was active at failure is not a valid restore point unless it had completed.
- A transactional database call, HTTP request, email, or payment outside the supported sink protocol needs its own idempotency or reconciliation boundary.
- Checkpoint interval, duration, timeout, minimum pause, concurrency, and storage should be tuned from observed checkpoint history rather than copied from another job.
Flink's official checkpointing guide describes these controls and recommends durable checkpoint storage for production.
Separate automated recovery from planned operations
Checkpoints and savepoints use related snapshot mechanisms but have different owners and lifecycle contracts.
Automatic recovery
Checkpoint
Flink creates checkpoints as part of job execution and normally manages their cleanup. Use retained checkpoints only with an explicit storage and ownership policy.
Planned operation
Savepoint
An operator deliberately creates and owns a savepoint for upgrades, rescaling, migration, or controlled stop-and-resume. Stable operator UIDs preserve state mapping across job graph changes.
Do not use savepoints as the routine failure-recovery mechanism or assume every job change is state-compatible. Follow Flink's official checkpoint and savepoint comparison and test the exact upgrade path with production-like state.
Operate the slowest edge, not the average job
Backpressure occurs when a downstream subtask cannot accept records as quickly as upstream tasks produce them. Pressure propagates backward through bounded network buffers; it is a signal that one path is saturated, blocked, skewed, or waiting.
Diagnose in this order
- Find the backpressured and busy subtasks in the job graph rather than scaling every operator.
- Compare records in and out, busy time, backpressured time, idle time, checkpoint alignment or start delay, state size, and sink latency at subtask level.
- Inspect key distribution. Raising parallelism cannot split one hot key across keyed state owners.
- Check external rate limits, connection pools, transaction duration, and sink commit behavior before adding TaskManagers.
- Re-run failure and restore tests after changing parallelism, backend, checkpoint mode, buffer behavior, or connector guarantees.
Flink exposes backPressuredTimeMsPerSecond and related task metrics in its backpressure monitoring guide.
Production review checklist
- Define source replay guarantees, timestamp extraction, watermark behavior, and idle partition policy.
- Bound keyed and window state with aggregation, TTL, cleanup, and skew controls whose data-loss consequences are understood.
- Store checkpoints durably; alert on failed, slow, growing, and persistently in-flight checkpoints.
- Verify sink delivery semantics with forced TaskManager, JobManager, network, and destination failures.
- Give stateful operators stable UIDs and rehearse savepoint restore before upgrades.
- Track end-to-end freshness and correctness, not only records per second.