Latency vs Throughput

The invariant: stable queues need spare capacity

For a stable service, the arrival rate must remain below the capacity of its limiting stage: arrival rate < sustainable capacity. As utilization approaches 100%, even small bursts wait behind work already in progress. Once arrivals exceed completions, a queue grows until requests time out, are rejected, or users leave.

Little's Law gives a useful first estimate: concurrency = throughput x time in system. At 200 RPS and 0.25 seconds of end-to-end latency, expect about 50 requests in flight on average. It does not say that 50 workers are sufficient; it gives a quantity to measure and investigate.

Use an end-to-end deadline, then allocate a budget to each stage. A 300 ms API target might reserve 40 ms for network, 70 ms for application work, 120 ms for a database call, and 70 ms for queueing, retries, and margin. The budget is a contract: a faster app cannot compensate for a dependency that already uses the whole deadline.

Lab 1: protect a latency budget from queue pressure

This model has a 55 ms fixed path plus one bounded service gate. Change the incoming load, the gate's effective capacity, and the user-facing target. The queue estimate becomes steep near saturation because a request must wait for prior requests to leave the same gate.

The estimate assumes a steady workload and an exponential service-time distribution. Production systems need measured service-time distributions, burst tests, and separate limits for each dependency. The invariant still holds: operating close to the limit makes tail latency fragile.

Read the right latency signal

An average can look healthy while a small but important group of users waits far too long. Record a distribution for the same user path and workload shape:

• P50: the middle request. It describes the typical experience.
• P95: a warning that contention, cache misses, or a slow dependency is affecting a meaningful minority.
• P99: the tail where retries, queueing, and shared-resource exhaustion become visible.
• Timeout and rejection rate: the work the latency graph may omit because it never completed successfully.

Pair those measurements with arrival rate, completed RPS, queue depth, connection-pool wait time, and utilization. A rise in queue depth with flat completed RPS is evidence that the service is accepting more work than its constraint can release.

Failure behavior: do not turn waiting into an outage

When a downstream dependency slows, unbounded waiting consumes the resources needed for healthy work. Design a controlled failure path:

• Set a deadline shorter than the caller's deadline, so there is time to return a useful fallback or error.
• Bound each queue and connection pool. Reject or shed low-priority work before memory, threads, and sockets are all consumed.
• Use retries sparingly.
  • Retry only failures that are transient and safe to repeat.
  • Add jitter and a cap; retrying a saturated dependency multiplies its load.
• Isolate optional work such as analytics, email, or recommendations from the checkout, payment, or write path that must remain responsive.

Lab 2: choose batching and concurrency for the workload

Batching amortizes a fixed cost such as a network round trip or disk flush across more items. It can raise throughput, but the first item waits while the batch fills and every item waits for the batch to run. Change the workload deadline, batch size, worker count, and arrival rate to see when a throughput improvement violates the latency contract.

For an interactive request, a small batch or no batch may be correct even when it costs more per item. For a background job with a minutes-long deadline, a larger batch can be an efficient choice. Pick the workload's deadline first; do not label a higher RPS number as an automatic win.

Make the trade-off explicit in architecture

Start from the user action and make one path responsible for meeting its deadline. Keep the synchronous path short: validate input, read or write the minimum durable state, and return. Put non-critical fan-out behind a durable queue with an owner, maximum queue age, and dead-letter or replay plan.

Common choices have different costs:

• A cache can remove a dependency hop and improve both latency and throughput, but must define freshness and invalidation behavior.
• A read replica can add read capacity, but replication lag may violate a read-after-write expectation.
• Horizontal scaling raises application capacity, but cannot outscale a shared database, lock, partition, or third-party API.
• Load shedding lowers accepted throughput during stress, but preserves a bounded latency and higher completion rate for the work that remains.

The next lesson, Bottleneck Analysis, shows how to prove which resource is constraining the result before changing architecture.

Operate to the boundary, not the average

Use a load test that matches the production request mix, payload size, cache state, and burst pattern. Record a baseline before a change, then compare the same workload after it.

• Alert on sustained high utilization, queue age or depth, P95/P99 latency, rejected work, and dependency-pool wait time.
• Define a capacity trigger before the SLO is breached, such as adding capacity at 70-80% sustained utilization rather than waiting for 100%.
• Review a deployment or incident with the actual arrival rate, completions, latency distribution, errors, and saturation signal together.
• Test degraded modes: a slow database, a full queue, a depleted connection pool, and a failed worker should produce bounded, understandable behavior.

Capacity is not a single server number. It is a tested promise about workload shape, latency target, error behavior, and the weakest shared resource.