Data Sizes

Build every estimate from explicit units

A byte count only becomes useful after it is connected to a count and a time period. Use decimal units for provider bills and network rates unless the contract says otherwise; use binary units when discussing operating-system and memory capacities. Do not silently mix them.

Keep the dimensions visible

1. Object size: bytes per item, including fields that the chosen representation stores.
2. Cardinality: initial objects plus new objects per second, day, or month.
3. Retention: how long that data remains queryable, recoverable, or legally required.
4. Copies: replicas, regional copies, snapshots, and exports that each hold data.
5. Overhead: indexes, metadata, encoding, write amplification, free-space reserve, and growth headroom.

Learn the sizes that drive first-pass decisions

Representative values are anchors for a design conversation, not universal constants. A UTF-8 character can use one to four bytes, JSON repeats field names, a database row has engine overhead, and a media format embeds its own compression choices. Measure production samples before committing capacity.

Choose a representative size deliberately

• Sample the median and p95 payload size, then size the storage plan with a stated percentile or distribution.
• Include serialized fields, primary keys, timestamps, checksums, and per-object storage metadata.
• Separate primary payloads from derivatives such as image renditions, search documents, vectors, audit logs, and exports.
• Record whether the number is compressed on the wire, compressed at rest, or uncompressed in memory. Those are different quantities.

Turn a data model into a volume and bandwidth budget

Raw volume answers how much original payload arrives and remains retained. Write throughput answers how fast the ingestion path must accept bytes. A scan or export is a third question: it needs enough sustained transfer bandwidth to finish inside its window.

The lab keeps all inputs in its model visible. Pick an object type, adjust bytes per item, initial inventory, arrival rate, retention, and the transfer window. It computes retained raw size, ingest throughput, time to move the retained dataset over the modeled link, and the minimum link rate needed for the requested window.

The displayed transfer time assumes an uninterrupted, fully usable link. In a real migration or restore, reduce available bandwidth for protocol overhead, other traffic, retries, throttling, and the read speed of the source system.

Add indexes and metadata before calling it storage

Raw size is the payload arithmetic: bytes per item x retained items. Stored size is what the selected engine reserves to serve and protect that payload. The difference is often material.

1. 1

   Payload

   Encode the object

   Serialization, field names, nulls, compression blocks, object headers, and alignment determine the logical stored bytes.

2. 2

   Indexes

   Make it queryable

   Primary, secondary, full-text, and vector indexes trade more write and storage work for faster reads. Index only the access paths the product needs.

3. 3

   Headroom

   Keep it operable

   Metadata, compaction, free space, partition skew, temporary files, and growth reserve prevent a healthy-looking plan from failing during routine operations.

Questions to ask of a storage engine

• Which fields are duplicated in every index, and are variable-length values stored inline or by reference?
• Does the engine keep multiple row versions, write-ahead logs, compaction files, or temporary spill files?
• What fraction of the active working set must fit in memory to meet the read-latency objective?
• Are partitions evenly distributed, or can a popular tenant or key create a larger-than-average shard?

Use observed storage metrics from a representative environment to replace rules of thumb. A 30% index allowance may be reasonable for one workload and badly wrong for a text or vector index.

Separate availability copies from recovery copies

Replication makes another serving copy available; it can improve availability and read scale. A backup preserves a historical recovery point; it helps recover from deletion, corruption, or an operator mistake that replication can faithfully copy.

See the footprint and cost consequences of protection choices

Compression reduces the data portion that must be stored and moved, while indexes and metadata add serving overhead. Replicas and regions multiply durable serving copies. Backups add a different kind of copy with a different recovery benefit and restore cost.

This second lab starts from a stated daily ingest rate. Change compression, index overhead, replicas, backups, regions, and retention. Compare the resulting durable footprint, one-region working set, modeled monthly storage cost, and the failure or recovery consequence. Its rates are planning assumptions, not a provider quote.

Use the output to make a decision, not to finish the estimate

• Compression: validate compression ratio and CPU cost with representative production data; incompressible media should be modeled near 0% reduction.
• Indexes: measure the exact index design and write amplification before accepting a hot working-set target.
• Replicas and regions: test the surviving capacity, replication lag, promotion path, and data-consistency promise under failure.
• Backups: time a restore at realistic scale and verify application-level invariants, not only that files copied successfully.
• Cost: include storage class, reads, writes, transfer, API operations, reserved capacity, support, and engineering effort in the final budget.

Carry uncertainty into the design review

An estimate should show a range and name what would move it. Start with a baseline, then calculate a plausible high case rather than hiding uncertainty in an unexplained multiplier.

A review-ready estimate answers

1. What object size and percentile did we use, and how was it sampled?
2. What peak and average arrival rates are assumed, and what growth horizon is covered?
3. How long is each class of data retained, and which policy or customer requirement sets that period?
4. Which serving copies, backup copies, regions, indexes, and reserve factors are included?
5. What transfer path must complete within a time limit, and what sustained throughput does it require?
6. Which measurement or load test will retire the largest uncertainty before launch?