AWS Glue
Master AWS Glue: serverless ETL, data catalog, crawlers, and data integration patterns.
What is AWS Glue?
AWS Glue is a managed data-integration service. It can catalog data sources, infer metadata with crawlers, run Apache Spark or Python jobs, process streams, evaluate data quality, and coordinate jobs and triggers without a team managing its own Spark cluster.
The service separates metadata from data processing. The Glue Data Catalog describes where a dataset lives, how it is shaped, and how engines may address it. Glue jobs read source data, transform records, and publish outputs. Crawlers can discover metadata, but they do not decide whether a schema change is compatible with downstream business contracts.
The central production invariant is: a successful run must have one reproducible input boundary and one idempotent publish boundary. Managed compute removes cluster administration; it does not remove schema drift, duplicate output, late data, skew, small files, unsafe permissions, or ambiguous retries.
Describe assets
Data Catalog
Store databases, tables, columns, partitions, locations, formats, statistics, and governance metadata shared by Glue and compatible analytics engines.
Discover metadata
Crawler
Inspect configured data stores with built-in or custom classifiers, then create or update catalog tables under an explicit schema-change policy.
Transform records
ETL job
Run managed Spark, streaming, Python-shell, or supported integration work with declared workers, code, connections, arguments, bookmarks, and observability.
Coordinate runs
Workflow and trigger
Start jobs and crawlers on schedules, events, or predecessor outcomes while preserving run identity, dependencies, retries, and failure evidence.
Trace data from landing zone to published table
A Glue pipeline should keep raw input, metadata, transformation code, and published output independently identifiable. That makes a failed or incorrect run explainable and allows a repair to write a separate version before replacing a consumer-visible table.
One governed batch pipeline
Catalog metadata helps engines find data; the publish contract decides whether transformed output is correct, repeatable, and safe for consumers.
Immutable input
Landing zone
Store source objects or records with ingestion time, source identity, encryption, retention, and a stable batch or stream position.
Metadata
Catalog contract
Describe location, schema, partitions, format, ownership, classification, and access. Review incompatible changes before making them visible to shared readers.
Bounded transform
Glue job
Read a reproducible input set, validate rows, transform with measured parallelism, and write to a versioned staging location.
Commit boundary
Published dataset
Atomically expose an approved version or stable manifest, then advance source progress and notify consumers with lineage and quality evidence.
Name every boundary
- Give each run a stable ID tied to code version, job arguments, source positions, catalog version, and target version.
- Keep raw or replayable input until the downstream recovery window closes.
- Write rejected records to a restricted quarantine with reason codes; do not silently drop or coerce them.
- Publish through a manifest, table snapshot, partition swap, or merge contract that a retry can repeat safely.
- Advance a job bookmark or streaming checkpoint only after the intended output commit is durable.
Treat catalog discovery as a proposal, not an approval
Crawlers run classifiers, connect to a source, infer schema and partitions, and write metadata to the Data Catalog. When multiple S3 folders contain unrelated schemas, use separate include paths so one inferred table does not combine incompatible data.
1 Ownership
Scope the source
Give the crawler the narrowest include path, connection, role, and schedule that match one dataset owner and one schema family.
2 Inference
Classify samples
Run ordered custom classifiers where needed, then built-in classifiers. Record which classifier and source objects produced the proposed schema.
3 Governance
Compare compatibility
Detect additions, removals, type changes, partition changes, and classification changes against the consumer contract.
4 Controlled change
Publish metadata
Add partitions or compatible columns automatically only when policy allows. Log or quarantine breaking changes for owner review.
Keep metadata useful and safe
- Use consistent database, table, column, partition, owner, and environment naming.
- Prefer explicit table definitions for high-risk shared contracts; use crawlers for discovery and partition maintenance where inference is appropriate.
- Review crawler schema-change settings instead of accepting automatic catalog updates everywhere.
- Add partition indexes only for real partition-filter patterns and include their catalog storage cost and lifecycle.
- Apply Lake Formation and IAM permissions from data classification and consumer need, not from a broad service-role template.
AWS documents the current crawler flow in Using crawlers to populate the Data Catalog and the current metadata recommendations in the Data Catalog best-practices guide.
Size compute from input shape and completion targets
Glue bills job compute from resource use and runtime. The useful planning question is not only how many gigabytes arrive; file count, compression, joins, shuffles, skew, worker memory, output shape, startup, and source/sink throughput all influence the run.
bytes + files + keys
Parallel work
Object count and skew can dominate raw input size
startup + scan + shuffle + write
Completion time
Measure stage distributions in Spark UI and logs
active DPUs x runtime
DPU-hours
Auto Scaling changes active resources across stages
per-run cost x run count
Monthly compute
Add catalog, crawler, S3, transfer, logs, and source costs
The lab uses an explicit DPU-hour reference and benchmark throughput. Prices vary by region and feature, and benchmarks vary by data, code, Glue version, and worker type. Change the workload and worker plan, then replace every assumption with measured run evidence before approval.
Loading capacity lab
Preparing Glue job assumptions...
Tune from stage evidence
- Inspect Spark UI, event logs, CloudWatch metrics, and job-run DPU-seconds where the selected Glue version and mode expose them.
- Start with representative input size, object count, key distribution, compression, and join cardinality; tiny test samples hide skew and file-open cost.
- Use Auto Scaling where supported, but set a reasoned maximum and verify how worker changes affect shuffle, source pressure, and completion time.
- Choose memory-optimized workers for measured executor-memory pressure, not as a universal fix for inefficient joins or unbounded state.
- Compact tiny inputs or outputs according to downstream scan patterns instead of adding workers to multiply file creation.
The current Auto Scaling guide lists supported Glue versions and worker types. Use the deployed region's Glue pricing page for current rates and adjacent charges.
Make incremental state and replay semantics explicit
Glue job bookmarks persist progress for supported sources and transformation contexts. They are not a transaction spanning every source, custom side effect, catalog mutation, and target. A correct incremental job still needs stable source identity and idempotent output.
Scheduled batch
Job bookmark
Track supported source progress by job and transformation context. Initialize the job, use stable contexts, and commit only after the intended run succeeds.
Continuous job
Streaming checkpoint
Track stream progress and state through checkpoints. Glue streaming jobs use checkpoints rather than batch job bookmarks.
Replay safety
Idempotent publish
Write a stable batch, partition, or record identity so a replay converges on the same logical output instead of appending another copy.
Plan repair before the first run
- Define what happens when a source object changes at the same path after processing.
- Rewind or reset bookmarks into a separate repair target until the result is compared and approved.
- Keep output keys or merge keys stable across retries and code releases.
- Record late-arrival windows and whether old partitions are reopened, versioned, or ignored.
- Separate poison-record handling from infrastructure retries so one invalid row does not create an endless job retry.
AWS documents bookmark state and commit behavior in Using job bookmarks and streaming checkpoint behavior in Streaming ETL jobs.
Observe schema drift, replay, and small-file failure
The most expensive Glue incident is often a run that succeeds technically while publishing incorrect or operationally toxic output. Inject each failure and change the control that protects its real boundary.
Loading correctness lab
Preparing pipeline incidents...
Rehearse production recovery
- Add and remove columns, change a type, alter nullability, and introduce malformed partitions; verify compatible publication and quarantine behavior.
- Fail after output files are written but before progress commit, then retry the same run and count duplicate logical records.
- Reset a bookmark and confirm replay writes a separate target or converges on stable output identities.
- Feed one hot key, many tiny files, one huge compressed file, and slow source/sink responses; inspect stage skew and memory behavior.
- Stop and restart a streaming job around checkpoint creation and verify source and sink semantics together.
Publish analytical formats for consumer queries
Columnar formats and partitions reduce downstream scans only when they match real query predicates and file sizes. Partitioning every high-cardinality field creates metadata and small-file pressure rather than useful pruning.
Parquet or ORC
Columnar files
Use typed columns, compression, statistics, and predicate projection for analytical readers. Validate logical types and schema evolution across writers.
Coarse pruning
Useful partitions
Partition by bounded dimensions frequently present in filters, such as event date and a small region set. Measure partition counts and listing behavior.
Atomic publication
Table snapshots
For supported lakehouse tables, publish a new snapshot or manifest so consumers see a coherent version instead of a directory while files are still arriving.
Keep readers and writers aligned
- Define timestamp timezone, decimal precision, nullability, nested-field, and identifier conventions before multiple jobs write the same table.
- Estimate target file count from output bytes and desired file size, then verify with the actual compression ratio.
- Keep temporary and failed-run paths outside consumer-visible table locations.
- Update catalog metadata only after files or a table snapshot are durable.
- Test Athena, Redshift, EMR, and other intended readers against the exact published schema and permissions.
Secure and operate the pipeline as a data product
Glue jobs commonly bridge high-value stores. The service role, VPC connection, KMS keys, Secrets Manager access, S3 paths, Catalog permissions, and log contents together define the data-access boundary.
Apply least privilege to each path
- Give crawlers read access only to their source scope and controlled catalog write access.
- Give jobs separate source-read, quarantine-write, staging-write, publish, log, and connection permissions.
- Use Secrets Manager or managed connections for credentials; never put credentials in scripts, arguments, catalog properties, or logs.
- Use VPC routing and security groups for private sources, and verify DNS plus egress needed by dependencies.
- Encrypt source, temporary, shuffle, output, catalog, bookmark, and log data according to classification and key policy.
Monitor technical and business outcomes
- Run duration, startup delay, DPU-seconds, worker changes, stage skew, executor loss, retries, and failed tasks.
- Input bytes and records, accepted rows, quarantined rows by reason, output rows, duplicates, and reconciliation totals.
- Source positions, bookmark or checkpoint progress, late data, replay windows, and publish version.
- Input and output file count, size distribution, partitions, catalog changes, and downstream scan cost.
- Access denials, unusual catalog changes, secret or connection failures, KMS errors, and data-quality gate failures.
The final design question is not “Can Glue run this script?” It is “Can the team prove which data the run read, which contract it applied, which version it published, and how a retry or repair changes the result?”