Apache Airflow
Master Apache Airflow: DAG creation, operators, scheduling, and production deployment patterns.
What is Apache Airflow?
Apache Airflow is a workflow orchestrator for finite, dependency-driven jobs. A workflow is declared as Python code, represented as a Directed Acyclic Graph (DAG), and instantiated as a separate DAG run for each scheduled or manually triggered data interval.
Airflow decides when work is ready, where it should run, and what state it reached. The heavy computation should remain in databases, data warehouses, Spark clusters, containers, or other worker environments. This separation makes pipelines observable and recoverable without turning the orchestrator into the data-processing engine.
The core invariant is interval ownership: a scheduled DAG run should read and write the data assigned to its data interval, not whatever happens to be current when the task starts. That makes reruns and backfills predictable.
Trace the Airflow control plane
The scheduler creates task instances after their dependencies and timing constraints are satisfied. The executor dispatches those instances; workers perform the task code; the metadata database preserves orchestration state for the UI and recovery tools.
From DAG definition to task result
Control state moves through Airflow while the workload executes in the selected worker environment.
Declare
DAG definition
Describe tasks, dependencies, schedule, retries, ownership, and concurrency limits as versioned Python code.
Decide
Scheduler
Create DAG runs, evaluate dependencies, and queue task instances that are ready.
Run
Executor + workers
Launch tasks in local processes, distributed workers, or isolated containers.
Record
Metadata database
Persist run state, task attempts, configuration references, and operational history.
Definition
DAG
Acyclic task dependencies describe valid execution order. The DAG is a reusable definition, not one execution.
Interval
DAG run
One instantiation of the DAG owns one data interval and has an independent lifecycle.
Attempted work
Task instance
One task inside one DAG run moves through scheduled, queued, running, retry, and terminal states.
Decide which DAG runs Airflow should create
A daily run represents one daily data interval. Choose automatic catchup or an explicit backfill, then inspect the exact intervals and execution pressure.
Catch up missed intervals
Create every missing scheduled run.
Planned run set
3 intervals will run
01 Jul
Keep success
02 Jul
Will run
03 Jul
Keep success
04 Jul
Failed exists
05 Jul
Will run
06 Jul
Keep success
07 Jul
Will run
12
4 tasks per run
2
At most 2 runs together
3
Each run owns one interval
The run set matches the selected policy
Catchup creates missing scheduled runs; it does not automatically replace an existing failed run.
Write interval-aware DAGs
A daily schedule normally creates a run after its daily interval ends. The logical date identifies that interval; it is not a promise that the task started at that wall clock time. Template paths and queries from data_interval_start and data_interval_end so a late run, retry, or backfill still addresses the same data.
1 Schedule
Define the interval
Choose a timetable and timezone that match the source data's partition boundaries.
2 Input
Read only that window
Pass interval boundaries into source queries instead of using the worker's current time.
3 Output
Write idempotently
Use a stable partition or operation key so rerunning the interval replaces or merges the intended output.
4 Commit
Publish after validation
Expose the partition only after checks succeed; keep partial output from consumers.
Test whether a retry actually makes the workflow safer
Inject a failure into the load task. Change its retry contract and watch task state, recovery delay, and external side effects diverge.
Exponential backoff
Spread retries instead of hammering the dependency.
Idempotent target write
Reuse an interval-scoped operation key.
Attempt trace
Response lost after commit
The target commits, but its acknowledgement never reaches the task.
External write observedThe task repeats a write whose first commit was hidden from Airflow.
External write observedTask-state path
Choose execution and waiting models deliberately
The executor controls where task instances run; an operator defines the work; a sensor waits for an external condition. Select each mechanism from workload isolation and resource behavior rather than from DAG size alone.
Simple deployment
Local execution
Use local processes for bounded workloads that fit one scheduler environment. It has fewer moving parts but shares one machine's capacity and failure domain.
Scale + isolation
Distributed or isolated execution
Use a distributed queue or per-task containers when workloads need horizontal worker capacity, dependency isolation, or task-specific resources.
Release worker slots
Deferrable waiting
Prefer deferrable operators or suitable sensor modes for long waits. A polling task that occupies a worker slot can starve runnable work.
Keep data products outside Airflow metadata
- Use connections and hooks to centralize service credentials and client behavior.
- Use XCom for small control values such as a partition URI, row count, or model ID.
- Put large datasets in object storage, a warehouse, or another data system; pass only their location between tasks.
- Bound task and DAG concurrency against the capacity of downstream systems, not only against available Airflow workers.
Operate DAGs as replayable production programs
Airflow is a strong fit for scheduled ETL/ELT, ML workflow orchestration, data-quality checks, and finite infrastructure jobs. It is not a stream processor, a millisecond-latency request path, or the place to perform unbounded computation inside the scheduler.
Before enabling a production schedule:
- Make every task's input and output interval explicit.
- Test a normal run, a task retry, a cleared task, and a multi-interval backfill.
- Classify failures as transient, permanent, or ambiguous before assigning retries.
- Make external writes idempotent and prevent overlapping runs from corrupting output.
- Set timeouts, retry limits, backoff, pools, and concurrency limits from dependency budgets.
- Alert on failed or late data intervals, not merely on whether scheduler processes are alive.
- Keep DAG parsing lightweight and store secrets in a managed backend rather than in DAG source.
Backfill with a small concurrency limit first. Observe warehouse load, API quotas, and task duration before increasing parallel runs.