Apache ORC
Master Apache ORC: Optimized Row Columnar format, ACID transactions, compression, and Hive integration.
What is Apache ORC?
Apache ORC, short for Optimized Row Columnar, is a self-describing file format for typed, tabular data. It stores each column in separate encoded streams inside large horizontal row batches called stripes. An analytical reader can therefore fetch event_day and revenue without also reading a large payload field.
ORC matters because warehouse queries often scan many rows but project only a few columns. Its core invariant is: the reader starts from metadata, rejects stripes and row groups that cannot match, then decodes only required column streams from the remaining candidates. File ordering, valid statistics, pushed predicates, and reader support determine how much work is actually skipped.
ORC is a write-once file format, not a database. It does not provide a transaction manager, catalog, or in-place row updates by itself. Apache Hive can layer ACID visibility, base and delta files, and compaction around ORC files.
Warehouse scans
Strong fit
Use ORC for typed analytical data read by Hive, Spark, Trino, and other engines whose ORC implementations support the file features you write.
Point mutation
Weak fit
Do not treat one ORC file as an OLTP table. Updating a record normally means writing new file data and publishing the change through a table or transaction layer.
Reader fleet
Compatibility rule
Test every writer, reader, codec, schema change, and optional index used in production. The format contract is broader than any one engine's implementation.
Read an ORC file from the tail backward
An ORC file layout is the physical map a reader follows to find encoded values. The file starts with the ORC magic bytes, continues with independently readable stripes, and ends with the metadata, footer, postscript, and postscript-length byte that make up its tail.
ORC physical layout
A reader first obtains the file tail, then uses stripe locations and stripe footers to request the required index and data streams.
Identify
ORC header
Three magic bytes let tools recognize an ORC file.
Prune and seek
Stripe indexes
Optional row-index and Bloom-filter streams describe row groups inside a stripe.
Decode
Column streams
Typed values are separated into streams such as presence, data, length, and dictionary data. The stripe footer records their locations and encodings.
Plan
File tail
The footer defines schema, stripe locations, row count, statistics, and row-index stride. The postscript identifies compression and tail lengths.
The hierarchy has four important boundaries:
- File: one self-describing sequence of typed rows and its top-level metadata.
- Stripe: an independently readable group of whole rows containing indexes, column data streams, and a stripe footer.
- Row group: the rows represented by one row-index entry. The current ORC specification documents a writer-controlled default of 10,000 rows.
- Column stream: one physical stream for one column's presence, values, lengths, dictionary, index, or Bloom-filter data.
The stripe footer is required because stream order is not fixed. It is the directory that maps byte ranges to stream kinds, columns, and encodings. The official ORC specification defines the tail, stripes, streams, row indexes, and Bloom-filter indexes.
Make projection and pruning visible
Projection chooses which columns to read. Predicate pruning uses metadata to remove candidate row ranges. These are independent decisions and their effects compound:
- File and stripe statistics can reject whole stripes.
- Row-index min/max statistics can reject row groups inside candidate stripes.
- An optional Bloom filter can reject more equality candidates after min/max evaluation.
- The reader requests only projected column streams from groups that remain.
- The execution engine still evaluates the predicate on decoded rows.
The lab uses fixed, inspectable inputs rather than a benchmark:
- Nine row groups contain 10,000 rows each, arranged as three groups per stripe.
- Per-group compressed chunks are
0.25 MiBforevent_day,0.15 MiBforregion,0.75 MiBforcustomer_id,0.35 MiBforrevenue, and4 MiBforpayload. - The modeled file tail is
96 KiB; each stripe footer is12 KiB. - Complete row indexes add
18 KiBper group to the file. A query reads3 KiBper group in each candidate stripe for its predicate-column index. - Query selectivity is shown as a workload assumption, but it does not invent latency, throughput, or storage savings.
Plan an ORC scan
Loading the file-layout assumptions.
A metadata skip is a proof of absence
- A non-overlapping minimum and maximum proves that a stripe or group cannot match.
- An overlapping range means only "possibly matches"; it does not prove that any row qualifies.
- Bloom filters can return false positives, so a positive result keeps a group as a candidate. A valid negative result can reject it.
- Indexes accelerate selection and seeking. They do not contain enough row data to answer an arbitrary query.
- Sorting or clustering can narrow ranges, but it adds write and maintenance cost.
The Apache Hive ORC language manual also distinguishes row-group selection from final query evaluation.
Encode typed values, then compress the streams
An ORC encoding changes the representation of typed values. A compression codec then compresses stream chunks. They are separate layers, and the postscript tells a reader which general codec is needed.
Repeated values
Run-length encoding
Represent repeated or patterned integers with compact runs instead of storing every full-width value.
Small changes
Bit packing and delta
Use fewer bits for bounded integers and encode differences when nearby values are correlated.
Repeated strings
Dictionary encoding
Store distinct values once and reference them with smaller identifiers while the dictionary remains worthwhile.
Stream chunks
Generic codec
Compress encoded bytes with a reader-supported codec such as Zlib, Snappy, LZ4, or Zstandard. Codec support and results depend on the implementation and data.
Do not choose compression from a universal ratio chart. Measure representative data for output size, writer CPU, reader CPU, memory, and support across the complete reader fleet. The current ORC Java configuration reference lists supported writer settings, but an engine may expose a different subset or different defaults.
Keep file, schema, and table semantics separate
An ORC file schema describes the typed objects stored in one file. A query engine can request a reader schema and map compatible file fields, but mixed files do not become a safely evolved table merely because each file is self-describing.
Treat changes as a cross-engine contract:
- Adding a nullable field may let older files expose a missing value, but every reader still needs a tested mapping.
- Renaming or reordering fields is ambiguous when readers differ between name-based and positional evolution.
- Type widening, decimal precision changes, timestamp semantics, and case sensitivity need mixed old/new file tests.
- Removing a field from new files does not remove its historical values from old files.
Hive ACID is a separate table-level mechanism. ORC remains write-once; Hive records changes in base and delta files, supplies a committed-transaction snapshot to the reader, and compacts without mutating source files in place. See Apache ORC's ACID design and the Apache Hive transaction documentation.
Decide when metadata fallback is safe
A safe fallback reads more bytes while preserving every qualifying row. A best-effort recovery may return partial output. Those outcomes must never share the same success state.
Optional index absence is not the same as corruption. Without a Bloom filter, a reader can use min/max and final predicate evaluation. Without row-index streams, it can read candidate stripes without row-group pruning. By contrast, a damaged stripe footer removes the stream directory, and a truncated file tail removes information needed to locate stripes and interpret schema and compression.
Inject an ORC file failure
Loading metadata and recovery scenarios.
The ORC Java configuration keeps orc.skip.corrupt.data disabled by default. Enabling it can be useful for explicit salvage workflows, but a partial scan is not a complete count, sum, audit export, or training dataset. Quarantine damaged files, preserve the original bytes, identify the affected partitions, and repair from another trusted copy whenever completeness matters.
Inspect actual files before changing the layout
An ORC inspection workflow compares assumptions with file metadata and query plans. The official Java tools can print schema and stripe metadata, list data and metadata sizes, and test whether statistics or Bloom filters can reject a value.
The dependency-free example below reproduces the scan lab's range-overlap arithmetic. It reads the co-located JSON model; it does not parse an ORC binary or claim to model engine latency.
The Apache ORC Java tools reference documents meta, sizes, check, and offline recovery options. Pin the tool version to the file features and runtime readers you support.
Operate ORC as a measured storage contract
- Read-path evidence
- Capture projected columns, pushed predicates, candidate stripes, skipped row groups, bytes read, decode CPU, and final row counts.
- Verify that production plans actually invoke ORC predicate pushdown.
- Compare clustered and mixed-order files with representative query windows.
- File construction
- Keep stripes large enough for efficient sequential work but numerous enough for useful parallelism in the target engine.
- Avoid tiny files that multiply listing, footer-read, and scheduling work.
- Add Bloom filters only to equality predicates where measured pruning repays their file and memory cost.
- Compatibility and correctness
- Test old and new schemas together across every supported engine.
- Validate row counts, null counts, min/max ranges, checksums, and codec support before publishing files.
- Treat corrupt-data skipping as a labeled salvage mode, never an invisible default.
- Hive ACID operations
- Monitor base/delta file counts and compaction health.
- Keep transaction visibility and compaction ownership in the table layer.
- Do not infer transactional correctness from a healthy ORC footer alone.