Hazelcast IMDG
Master Hazelcast IMDG: in-memory data grid, distributed computing, and high-performance caching.
What is Hazelcast?
Hazelcast is a distributed platform for keeping data and computation close together in memory. Applications connect to a cluster and use shared maps, caches, queues, topics, SQL, and stream-processing pipelines without assigning every key or task to a server by hand.
The central idea is partition ownership. Hazelcast hashes a key to a partition, assigns that partition to one member as its primary owner, and can place backup copies on other members. When membership changes, the cluster migrates partition ownership so the application continues to address the logical data structure rather than one host.
Hazelcast is not automatically the system of record. In-memory copies, asynchronous backups, Near Caches, split clusters, and external sinks all have distinct durability and consistency boundaries. A production design must declare which state may be stale, which operation must have one authoritative order, and how the cluster recovers after member or network failure.
Placement
Partitioned state
Map entries are assigned to partitions by key. Partition ownership is distributed across members and moves as the cluster grows, shrinks, or recovers.
Availability
Backup copies
Synchronous backups participate in mutation completion; asynchronous backups catch up later. Every backup is another full copy that consumes memory.
Consistency
AP and CP contracts
Ordinary distributed structures favor availability during some failures. The separate CP Subsystem uses Raft for locks, counters, semaphores, and references that need a single authoritative order.
Processing
Data-local compute
Entry processors, SQL, and pipelines can run work near partitioned data, reducing network transfer when the operation and partition key are designed together.
Follow one key from client to partition owner
A distributed map looks like one key-value structure to the caller, but each operation crosses a placement and ownership protocol. Stable serialization and keys are part of that protocol because they determine partition assignment, object size, and wire cost.
One distributed-map mutation
The caller addresses a logical map; Hazelcast resolves a partition owner, applies the mutation, and updates the configured backup copies.
Declare intent
Application client
Choose the map, key, timeout, operation identity, and consistency requirement. Avoid an unbounded retry that can repeat a non-idempotent mutation.
Resolve ownership
Partition service
Hash the serialized key to a partition and route the request to its current primary owner. Membership changes can move that ownership.
Apply operation
Primary owner
Execute the map operation or entry processor against the authoritative primary copy for that partition.
Protect a copy
Backup owners
Store synchronous or asynchronous replicas on different members according to the map's backup configuration.
Keep keys and work partition-aware
- Use stable, deterministic key serialization across every client and release.
- Co-locate related entries with partition-aware keys only when operations genuinely need to update or read them together.
- Prefer entry processors for small data-local mutations; sending a large object to a client, changing one field, and writing it back adds transfer and race risk.
- Treat listeners and events as notifications, not as an authoritative replay log unless the selected structure explicitly provides the retention and ordering needed.
- Measure partition skew. Evenly distributed partition counts do not prevent a hot key or hot tenant from saturating one owner.
Size primaries, backups, and recovery headroom together
One backup doubles the stored map footprint before object overhead, indexes, Near Caches, pipeline state, and platform memory. The default of one synchronous map backup is a starting point, not proof that the cluster can tolerate the intended failure while remaining below its memory and latency limits.
primary x (1 + backups)
Stored copies
Each configured backup is a complete additional copy
copies + overhead + reserve
Protected footprint
Reserve room for skew, migration, and safety restoration
primary + sync acknowledgements
Write path
Synchronous safety can add blocking latency
lost members before lost copies
Failure budget
Copies must be placed on distinct surviving members
Change members, primary data, backup count, and heap independently. The visible result distinguishes raw copies from the operational headroom required to migrate partitions and rebuild safety after failure.
Loading capacity lab
Preparing the cluster envelope...
Validate the model with real data
- Measure serialized key and value distributions instead of multiplying object count by an in-process language-object estimate.
- Include indexes, query results, pipeline snapshots, Near Caches, off-heap/native memory, and JVM or container overhead in the complete node budget.
- Remove a member at peak traffic and observe migration duration, partition-operation latency, heap pressure, and the time until every backup is restored.
- Test simultaneous failures that match the infrastructure fault domain; multiple virtual machines in one zone are not independent copies of a zonal outage.
- Benchmark synchronous and asynchronous backups against the real mutation mix. Faster acknowledgement is a weaker loss boundary, not free performance.
The current map backup documentation describes copy placement, synchronous and asynchronous behavior, and the configured backup-count limits.
Choose an AP structure or a CP primitive deliberately
Hazelcast exposes two different consistency families. They are not interchangeable implementations of the same promise.
Serve through some partitions
AP distributed structures
IMap and the ordinary structures under HazelcastInstance are designed around availability and partition tolerance. During a network split, both sides may otherwise continue and later require merge-policy reconciliation.
Preserve one order
CP Subsystem primitives
Raft-backed locks, atomic values, semaphores, latches, and references require a valid CP majority. A side without that majority stops serving the CP operation rather than creating two authoritative outcomes.
Match the model to the business invariant
- Use an AP map for disposable cache entries, rebuildable projections, sessions with a declared merge policy, or state whose temporary divergence is acceptable.
- Use a CP primitive for leader election, a fencing decision, a global permit, or a coordination value that must not have two successful owners.
- Do not use a distributed lock without fencing downstream writes. A paused lock holder can resume after its lease or session is no longer authoritative.
- Cache CP object handles rather than repeatedly fetching and destroying them; CP lifecycle operations have distributed consequences.
- Keep durable business truth in a system whose recovery and audit contract matches the domain. A CP counter is not a replacement for a ledger.
See the current distributed data structures guide for the AP and CP structure families supported by the deployed edition and version.
Observe the consequence of member loss and network partition
Backups protect copies, split-brain protection can reject selected AP operations below a minimum observed cluster size, and the CP Subsystem requires a valid majority. These controls protect different boundaries. None removes the delay before failure detection or invents a correct merge policy for conflicting business updates.
Loading failure lab
Preparing cluster incidents...
Rehearse the complete incident
- Kill one member, then another before backup restoration completes; verify the result matches the configured copy count and placement domains.
- Partition the member network while clients remain connected to both sides. Confirm which AP operations reject, which CP operations stop, and which alerts identify the consistency decision.
- Heal the network and inspect merge results at the business-record level, not only the cluster-member count.
- Restart or disconnect Near Cache clients so missed invalidations and stale local copies are visible.
- Delay failure detection and backup acknowledgement independently; instant process death is not the only production failure shape.
Hazelcast documents an eventual detection window for split-brain protection. The application must handle rejected operations and monitor that window explicitly.
Add Near Cache only with a freshness contract
A Near Cache stores selected map entries inside or beside the client. It can remove a network hop from hot reads, but it creates another copy whose invalidation and expiry behavior must match the product's tolerance for stale data.
1 Fast path
Read the local copy
The client checks its Near Cache using the same serialized key and local eviction policy. A hit avoids a cluster request.
2 Owner path
Resolve a miss
The client routes the read to the current partition owner and receives the authoritative map value for that instant.
3 Additional copy
Populate locally
The client stores the returned value under its local capacity, TTL, and maximum-idle rules.
4 Coherence
Invalidate on change
Map mutations trigger invalidation metadata. Reconnects, missed messages, and repair behavior determine the real stale-read window.
Use Near Cache where stale reads are explicitly acceptable
- Good candidates are frequently read, infrequently changed values whose temporary staleness has a named product consequence.
- Poor candidates include authorization decisions, inventory reservations, uniqueness checks, and coordination state unless the business contract independently verifies freshness.
- Track local hit rate, invalidation lag, repair activity, memory use, and stale-read incidents by client fleet.
- Do not count Near Cache memory as server-cluster capacity; it is replicated client memory with its own eviction behavior.
Run computation and streams with explicit delivery semantics
Hazelcast can execute entry processors near map data, query data with SQL, and run batch or streaming pipelines. The operation should state where its mutable state lives, how it is partitioned, and what a retry or restart may repeat.
One partitioned entry
Entry processor
Move a bounded mutation to the key owner so the application does not transfer and race an entire value. Keep processing short and avoid blocking the partition thread.
Distributed query
SQL and projections
Indexes and predicates change memory, transfer, and execution cost. Inspect plans and bound result size before exposing an ad hoc query path to user traffic.
Snapshot and sink
Streaming pipeline
At-least-once recovery can repeat records. Exactly-once support depends on the complete source, transform, snapshot, and sink contract, not only one job setting.
Keep processing recoverable
- Give external side effects stable event or operation IDs and make sinks idempotent where replay is possible.
- Bound window state, allowed lateness, snapshot interval, and backpressure so a slow sink cannot consume unbounded memory.
- Test member failure during snapshot, source reconnect, sink timeout, and deployment upgrade.
- Separate online map traffic from heavy analytical work when they compete for the same CPU, network, heap, or partition threads.
- Verify ordering assumptions. Parallel stream stages do not imply a global event order unless the pipeline explicitly establishes one.
Choose deployment and operating boundaries
Embedded members put application code and cluster ownership in one process. Client- server mode separates application clients from data members. The choice changes failure coupling, scaling, upgrade cadence, network latency, and resource isolation.
Tight locality
Embedded members
Choose embedded mode when application and data lifecycles can move together and the team can isolate partition work from application CPU, heap, and deployment churn.
Independent lifecycle
Client-server cluster
Choose clients when data members need independent sizing, upgrades, placement, and failure containment. Budget the network hop and client reconnect behavior.
Durable authority
External system of record
Use MapStore, events, or application writes only with an explicit ordering, retry, and reconciliation contract. Write-behind queues are not a transaction with the source.
Production review checklist
- Pin and test member/client versions, serialization formats, discovery configuration, and rolling-upgrade compatibility.
- Place copies across real infrastructure fault domains and rehearse degraded capacity.
- Encrypt member and client traffic where required, authenticate workloads, authorize data-structure access, and keep secrets out of map values and diagnostics.
- Monitor partition migration, backup safety, heap/native memory, garbage collection, operation latency, rejected operations, client reconnects, Near Cache repair, and pipeline snapshot health.
- Alert on user-visible consequences: stale authorization, lost session, duplicate side effect, unavailable lock, delayed event, or source overload.
- Keep Management Center and cluster discovery endpoints on restricted administrative networks with audited access.