Amazon SQS | System Designer

What is Amazon SQS?

Amazon Simple Queue Service (SQS) is a managed buffer between a producer that creates work and a consumer that performs it. A producer sends a message and can finish its request without waiting for the consumer. SQS stores the message until a consumer receives it, completes the work, and explicitly deletes the receipt.

SQS matters because arrival rate and processing rate rarely match perfectly. A queue absorbs short bursts, lets workers scale independently, and keeps a temporary consumer failure away from the user-facing request path. AWS operates the queue service; application teams still own message identity, processing safety, retry limits, backlog recovery, and the downstream side effect.

The core invariant is receive does not mean complete. Receiving a message only leases it through a visibility timeout. The message is complete only after the consumer's durable work succeeds and DeleteMessage succeeds with the current receipt handle.

Follow one message through the queue

SQS is a point-to-point work queue: one delivery is intended for one consumer at a time. It is not a retained event log, and one queue does not independently deliver the same message to several subscriber applications.

SQS work handoff

The queue is a durable handoff, not the business transaction. Completion happens at the consumer's side-effect boundary.

Send

Producer

Create a small, versioned command with a stable operation ID. Keep large objects in object storage and put the durable pointer in the message.

Buffer

SQS queue

Store the message across service-managed infrastructure, expose it to a poller, and hide an in-flight receipt for the configured visibility period.

Apply

Consumer

Validate the contract, perform a bounded unit of work, and use a stable idempotency key at any business-effect boundary.

Acknowledge

Delete or retry

Delete after success. If the receipt is not deleted, it becomes visible again; after a finite receive budget, a redrive policy can move it to a dead-letter queue.

Keep the message contract explicit

Identity: include a stable message ID or business operation ID that survives transport retries.
Version: name the schema version and preserve compatibility during producer and consumer rollouts.
Payload: carry only what the consumer needs; avoid secrets and large binary bodies.
Trace context: include correlation metadata so enqueue, receive, retry, side effect, and delete can be connected during an incident.

Choose Standard or FIFO from the business invariant

Queue type is not a performance preference. It determines what the consumer is allowed to assume about order and duplicate transport behavior.

Independent work at scale

Standard queue

Standard queues provide at-least-once delivery and best-effort ordering. Choose one when messages can run independently, duplicates are safe, and horizontal throughput matters more than sequence.

Make every effect idempotent.
Do not infer order from send timestamps.
Use MessageGroupId only when deliberately applying Standard-queue fair-queue behavior, not as a strict-order guarantee.

Order within a group

FIFO queue

FIFO queues preserve order within each MessageGroupId and suppress duplicate sends through deduplication IDs. Different groups can run concurrently; one hot or blocked group remains serial.

Group by the smallest entity that needs order.
Reuse a deduplication ID only for the same logical send.
Keep consumers idempotent because a crash before delete can repeat processing.

FIFO deduplication does not make an external payment, email, or database update exactly once. SQS cannot atomically combine DeleteMessage with an unrelated system. Put the idempotency record and business mutation in one destination transaction when possible.

Size for recovery, not only steady traffic

A healthy consumer pool needs spare capacity after a deployment pause, dependency outage, or worker failure. If messages arrive at rate A and consumers can sustain rate C, the backlog drains only when C > A.

Outage backlog: arrival rate x outage seconds.
Catch-up rate: consumer capacity - ongoing arrival rate.
Recovery time: outage backlog / catch-up rate.
FIFO active workers: no more than the number of currently available message groups, even if the worker pool is larger.

Loading the queue model...

Model outage backlog and catch-up time

The lab's request-pressure estimate is a planning model, not a price quote. SQS pricing and regional throughput quotas can change, so validate the modeled payload units and API pattern against the current deployment region before approval.

Treat visibility as a renewable processing lease

When SQS returns a message, the message stays in the queue but becomes temporarily invisible. The consumer receives a receipt handle for that delivery. It should delete with that handle only after owned work is durable.

1
Receive
Long-poll a bounded batch
Wait for available work instead of continuously issuing empty receives. Keep each batch small enough that every message can finish before its processing lease expires.
2
Contract
Validate before side effects
Reject malformed, unauthorized, or unsupported versions deliberately. Permanent contract errors should not consume an unlimited transient-retry budget.
3
Process
Apply with an operation key
Write the idempotency key and business effect atomically at the destination. A repeated delivery should return the first result or become a no-op.
4
Heartbeat
Extend only live work
Use ChangeMessageVisibility for legitimate long-running work. Stop extending when the worker loses ownership or cannot prove progress.
5
Complete
Delete the current receipt
Delete after success. Batch deletes can reduce request pressure, but every entry in a batch can fail independently and must be checked.

Current SQS documentation allows visibility from 0 seconds to 12 hours, with a 30-second default. Long polling can wait up to 20 seconds, and batch APIs accept up to 10 messages. Treat those as service bounds, not recommended application defaults. Confirm changes in the AWS SQS message quotas.

Inject the failure between side effect and delete

The most important SQS failure happens after the business operation succeeds but before the receipt is deleted. From SQS's perspective, that looks identical to a worker that never finished. Change the contracts below and trace what another delivery can do.

Loading the delivery trace...

Commit the idempotency key with the business effect

Quarantine poison messages without hiding incidents

A dead-letter queue (DLQ) is an isolation queue for messages that cross a configured maxReceiveCount. It protects healthy work from an endlessly failing message, but it does not diagnose, repair, or safely replay that message by itself.

Retry decision

Classify the failure

Retry timeouts, throttling, and brief dependency outages with backoff and jitter. Quarantine invalid schemas, missing required identity, and deterministic business-rule failures unless a code or data change can repair them.

Diagnosis

Preserve evidence

Keep the original body, attributes, operation ID, receive count, exception category, and consumer version. Protect sensitive payloads with the same or stronger access and retention controls as the source queue.

Recovery

Redrive deliberately

Fix the cause, test representative messages, choose a bounded redrive rate, and watch the destination's capacity. Redriven FIFO work can interleave with newly produced messages.

Set the retry budget from failure behavior

A very low receive count can quarantine work during an ordinary transient failure.
A very high receive count increases repeated work, delays diagnosis, and can hold a FIFO message group behind one poison message.
Keep DLQ retention long enough for alerting, triage, remediation, and controlled redrive.
Alarm on any DLQ arrival when the queue represents money movement, account state, or another high-integrity workflow.

Operate the user-visible freshness contract

Queue depth alone is ambiguous: ten expensive jobs can be worse than ten thousand fast ones. Track age, processing rate, and downstream outcomes together.

Age

Oldest message

How stale the next useful effect can become

Rate

Sent vs deleted

Whether consumers are keeping up with arrivals

Retries

Receive count

How often work returns without successful delete

DLQ

Quarantine arrivals

Permanent or exhausted failures requiring ownership

Build alarms around outcomes

Alert when oldest-message age crosses the product's freshness objective, not only when depth is non-zero.
Compare sent, received, deleted, empty-receive, and visibility-change rates to find polling waste or failing handlers.
Track processing latency percentiles and visibility extensions by workload type.
Correlate DLQ messages with exception class, deployment version, tenant, and message schema.
Test worker loss, dependency throttling, duplicate delivery, poison messages, and redrive under a realistic incoming load.

Secure each queue as a workload boundary

SQS removes broker administration, not authorization or data-governance work.

Give producer identities only SendMessage; give consumer identities receive, visibility-change, and delete actions only on their queues.
Restrict queue resource policies and VPC endpoints by expected account, service, and source conditions; avoid broad principals.
Use server-side encryption and grant the minimum KMS permissions when a customer managed key is required.
Keep secrets and unnecessary personal data out of bodies, attributes, queue names, logs, and DLQ inspection tools.
Set queue, DLQ, and payload-store retention from the same deletion and compliance policy.
Audit configuration changes, redrive actions, policy updates, and cross-account access.

Know when SQS is not the right primitive

One worker owns each task

Use SQS

Choose SQS for background jobs, burst absorption, decoupled service commands, and bounded retries where one consumer should complete each message.

Fan-out or event routing

Use SNS or EventBridge

Use a publisher-and-subscriber service when several independent applications each need their own delivery. Fan out to separate SQS queues when each subscriber needs buffering and retry isolation.

Replay or query

Use a stream or database

Use a retained stream when independent consumers must replay history. Use a database when the primary need is transactions, mutable state, or ad hoc queries rather than asynchronous ownership transfer.

The design question is not "Can SQS carry this message?" It is "Does one bounded, idempotent consumer own this work, and can the system detect when that ownership is no longer making progress?"

No quiz questions available

Could not load questions file