Hadoop & HDFS
Master Hadoop and HDFS: distributed storage, MapReduce processing, and big data architecture patterns.
What is Hadoop HDFS?
Hadoop Distributed File System (HDFS) is a cluster filesystem for storing very large files across many machines. It splits each file into large blocks, places replicated or erasure-coded fragments on DataNodes, and uses a NameNode to coordinate the namespace and block locations.
HDFS matters when aggregate throughput, capacity, and failure recovery matter more than per-request latency or random updates. It fits write-once or append-oriented data that analytical engines scan in parallel. Core invariant: the namespace must identify enough verified block copies or fragments to reconstruct every committed file, even while machines and disks fail.
HDFS is usually paired with distributed compute such as MapReduce, Spark, Hive, or Trino. It is not a replacement for an online transactional database or a general-purpose low-latency object API.
Separate metadata coordination from the data path
HDFS uses a master-worker design, but file bytes do not flow through the NameNode. Clients ask for block locations and then stream data directly to or from DataNodes.
NameNode
Owns the directory tree, permissions, file-to-block mapping, block locations, replication decisions, leases, and namespace journal.
DataNode
Stores block files, serves reads and writes, validates checksums, reports inventory, and participates in replication pipelines.
HDFS client
Resolves metadata with the NameNode, then transfers block packets directly with DataNodes and retries alternate replicas when needed.
Standby NameNode
Tails shared edits and maintains a synchronized namespace so an HA controller can fence the old active and fail over safely.
Keep four responsibilities explicit
- Namespace durability: persist edits, checkpoint the namespace, test restore, and keep active and standby identities unambiguous.
- Block durability: maintain checksums, target copies or fragments, rack diversity, and enough repair bandwidth.
- Placement: balance locality, rack failure domains, disk utilization, storage tiers, and current cluster health.
- Client correctness: preserve write-pipeline acknowledgment, lease recovery, idempotent job output, and observable read failure.
Size raw disk, protected capacity, and namespace pressure
Raw disk is not usable data capacity. Replication consumes physical copies, operating reserve keeps recovery possible, and the file distribution determines how much metadata the NameNode must retain.
Size storage, replicas, and namespace pressure together
Change the cluster shape and file profile. The model preserves the distinction between raw disk, replicated capacity, operating reserve, and NameNode metadata.
Capacity verdict
Balanced for sequential production workloads
The reserve can absorb about 9 full nodes while the namespace remains within the illustrative 64 GiB metadata envelope.
2.30 PB
All DataNode storage
768 TB
3 physical copies
614 TB
20% held free
1.1 GiB
Illustrative block and file metadata
1.3M
4 blocks per file at 128 MB
5.0M
15.1M physical replicas tracked
9 nodes
Capacity only; rack placement and repair bandwidth still matter
Model boundary
This is a planning model, not a hardware benchmark. Validate drive throughput, rack bandwidth, NameNode heap, erasure-coding policy, ingest bursts, and re-replication time with the real cluster and file distribution.
Use a complete capacity equation
- Start with raw disk: DataNodes multiplied by installed storage per node.
- Apply protection overhead: divide by replication factor, or use the actual erasure-code data-to-parity ratio.
- Hold operating reserve: keep free space for failed-node repair, balancing, ingest bursts, and uneven placement.
- Model the file distribution: count logical blocks and files, not only total bytes, to estimate namespace memory.
- Validate the bottlenecks: measure drive throughput, rack bandwidth, NameNode heap, checkpoint time, and re-replication duration.
The small-files problem is primarily a namespace and scheduling problem. Combining many tiny objects into larger container files can reduce NameNode pressure, but the container format must still support the required read and retention patterns.
Follow the read and write paths
The client interacts with metadata and storage components differently. Understanding the path explains locality, throughput, retries, and why the NameNode does not need network capacity for every file byte.
A client streams a block through a replica pipeline
The NameNode chooses targets; packets flow through DataNodes and acknowledgments return in reverse order.
Packet source
Client
Requests a block allocation, divides data into packets, tracks checksums, and retains unacknowledged packets for retry.
Placement
NameNode
Checks the lease and namespace, selects healthy rack-aware targets, and records the block identity without carrying file bytes.
Replication
DataNode pipeline
Each target stores and verifies the packet, forwards it to the next replica, and returns an acknowledgment upstream.
Visible state
Committed block
The client closes the block and file only after the pipeline and namespace satisfy the configured durability contract.
Read path
- The client requests block locations and receives ordered replicas, usually preferring local or nearby storage.
- It streams from a DataNode, validates checksums, and switches replicas if the node or block is unhealthy.
- Compute schedulers can place work near input blocks to reduce cross-rack transfer.
Write path
- The NameNode grants a lease and chooses targets according to health, rack, capacity, and placement policy.
- The client streams packets through the selected pipeline; acknowledgments travel back only after downstream acceptance.
- A failed target causes pipeline reconstruction and continued writing when the durability policy can still be met.
HDFS supports append-oriented workflows but not arbitrary in-place file updates. Jobs commonly write a new immutable output and atomically rename it into place.
Choose replication or erasure coding deliberately
Both mechanisms make data recoverable, but they spend different combinations of storage, network, compute, and operational complexity.
Triple replication
Keeps three complete copies. Reads and repairs are straightforward, and hot data has multiple full replicas, but storage overhead is about 3x.
Erasure coding
Stores data and parity fragments. It reduces overhead for large colder datasets while increasing reconstruction, network, and compute work.
Rack-aware placement
Spreads protection across failure domains. Replica count alone is insufficient when every copy shares one rack, controller, or power source.
Storage policies
Assign hot, warm, cold, SSD, archive, replication, or erasure-code policies according to access and recovery objectives.
Prefer replication when
- Data is hot, latency-sensitive, frequently read, or operational simplicity matters most.
- Fast replica switching and uncomplicated repair justify the extra disk.
- Files are still being appended or do not fit the cluster's erasure-coding workflow.
Prefer erasure coding when
- Files are large, closed, colder, and expensive storage overhead dominates.
- The cluster has enough network and compute headroom for reconstruction.
- Operators have tested degraded reads, repair duration, and failure-domain placement.
Recover the broken invariant before optimizing the cluster
A lost DataNode, unavailable NameNode, corrupt replica, and disk imbalance require different first actions. Inject each failure, choose the response, and see how bandwidth changes the exposure window.
Loading recovery lab
Preparing failure scenarios...
Diagnose failures in a fixed order
- Protect namespace correctness. Fence uncertain NameNode ownership and stop writes if split brain or journal divergence is possible.
- Identify endangered data. Prioritize blocks with the fewest verified copies or insufficient failure-domain diversity.
- Bound repair pressure. Restore redundancy without saturating disks and rack links needed by foreground jobs.
- Verify reconstructed state. Check checksums, block counts, rack placement, namespace health, and representative client reads.
- Restore balance last. Rebalance healthy data after durability and availability are stable.
Build high availability for metadata
The NameNode is logically central, so its memory, journal, checkpoints, failover controls, and recovery procedures deserve the same rigor as the block fleet.
NameNode durability
- Persist namespace changes in the EditLog and periodically create FSImage checkpoints.
- Replicate or journal edits through a supported HA design and protect the journal from split brain.
- Back up metadata outside the active cluster and rehearse restore into an isolated environment.
- Monitor heap, garbage collection, edit growth, checkpoint duration, RPC queues, and block-report processing.
High-availability controls
- Run active and synchronized standby NameNodes with automatic health monitoring.
- Fence the old active before promotion so only one NameNode can issue authoritative namespace changes.
- Test failover with real clients, leases, writes, and dependent schedulers.
- Remember that the SecondaryNameNode creates checkpoints; it is not a production hot standby.
Federation
Federation splits independent namespace volumes across multiple NameNodes so metadata scale and organizational ownership do not depend on one namespace. DataNodes can serve blocks for several namespaces, but each namespace keeps its own metadata and failure boundary.
Secure every identity and data path
Filesystem permissions alone do not authenticate cluster services. A production HDFS design needs strong identity, transport protection, encryption boundaries, and auditable administration.
- Use Kerberos or the platform's supported strong authentication for users and services.
- Apply owner/group permissions and ACLs with least privilege; separate service identities by workload.
- Encrypt RPC and block-transfer traffic when the network is not a trusted boundary.
- Use transparent encryption zones and a protected key-management service for sensitive datasets.
- Restrict NameNode, journal, key, and administrative endpoints from general workload networks.
- Retain access and administration audit events with identities, dataset paths, policy results, and time bounds.
Operate HDFS around leading failure signals
Capacity percentage alone is a late signal. Monitor whether the cluster can still place, repair, verify, and serve blocks within its recovery objectives.
Watch continuously
- Live, dead, stale, decommissioning, and maintenance DataNodes by rack and storage type
- Missing, corrupt, under-replicated, pending-replication, and excess blocks
- NameNode heap, RPC latency, edit rate, checkpoint age, safe mode, and failover health
- Disk fullness and skew, failed volumes, checksum errors, read/write throughput, and queue time
- Cross-rack bandwidth, re-replication rate, balancer progress, and foreground-job impact
Rehearse periodically
- Active NameNode loss, journal interruption, metadata restore, and client failover
- DataNode and rack loss while the cluster is under representative read and ingest load
- Corrupt replicas, failed disks, decommissioning, storage-policy movement, and key rotation
- Capacity expansion and removal without violating reserve or failure-domain placement
A healthy HDFS cluster is not merely available now. It has enough verified copies, free disk, bandwidth, metadata headroom, and tested control-plane recovery to remain available through the next credible failure.