📬SQS Basics

Fully managed AWS queues for decoupling services without running your own broker.

📌 Key IdeaSQS is a serverless message queue on AWS. Producers send messages; consumers poll and delete after processing. Standard queues maximize throughput; FIFO queues provide ordering and deduplication per message group. Design for at-least-once delivery — idempotent handlers, visibility timeout tuning, and DLQ are essential.

🧩

Problem it solves

Microservices on AWS need async decoupling without operating Kafka or RabbitMQ clusters.
Traffic spikes and slow consumers can be buffered in a durable queue instead of timing out callers.
Background jobs (webhooks, image processing, email sends) need reliable handoff from API to worker.
Lambda, ECS, and EC2 workers need a scalable trigger source with built-in retry semantics.

📖

Definition

Amazon SQS (Simple Queue Service) is a fully managed message queue. Producers call SendMessage; consumers call ReceiveMessage (poll), process the payload, then DeleteMessage using a receipt handle. Messages are hidden from other consumers during a visibility timeout. Failed processing returns messages after timeout for retry; poison messages route to a Dead Letter Queue (DLQ).

✨

Why It Matters

Zero broker ops — no patching, clustering, or capacity planning for the queue itself.
Native integration with Lambda (event source mapping), SNS (fan-out), and CloudWatch (metrics/alarms).
Absorbs load spikes — API returns fast after enqueue; workers process at their pace.
Foundation pattern for event-driven AWS architectures alongside SNS and EventBridge.

🖼

Visual

📬 SQS core model

Producer -> Queue -> Consumer (poll)

Standard queue: best-effort ordering, at-least-once
FIFO queue: strict order + dedup per message group
Visibility timeout: hides message while processing
Receipt handle: token required to delete/extend
DLQ: poison messages after maxReceiveCount

Fan-out: SNS topic -> multiple SQS queues

✅

When to Use

Decouple microservices on AWS without self-hosting a message broker.
Buffer async jobs, webhooks, or background tasks with automatic scaling.
Trigger Lambda, ECS, or EC2 workers via queue event source mapping.
Fan-out work via SNS → multiple SQS queues (one per consumer type).
FIFO queue when strict ordering or deduplication is needed within a message group.

🚫

When NOT to Use

Event replay or multi-subscriber log consumption — use Kafka or Kinesis.
Sub-millisecond latency or push delivery — SQS is poll-based (seconds-scale with long polling).
Complex in-broker routing — use SNS filters, EventBridge rules, or RabbitMQ exchanges.
Cross-cloud messaging where AWS vendor lock-in is unacceptable.

💻

Code Variations

1️⃣ Send + receive with long pollingjava

sqs.sendMessage(SendMessageRequest.builder()
    .queueUrl(queueUrl)
    .messageBody(payload)
    .build());

ReceiveMessageResponse resp = sqs.receiveMessage(
    ReceiveMessageRequest.builder()
        .queueUrl(queueUrl)
        .maxNumberOfMessages(10)
        .waitTimeSeconds(20)  // long poll
        .build());

Long polling (up to 20s) reduces empty receives and API cost vs short poll.

2️⃣ Process + delete (at-least-once safe)java

for (Message msg : resp.messages()) {
  try {
    processIdempotently(msg.body(), msg.messageId());
    sqs.deleteMessage(DeleteMessageRequest.builder()
        .queueUrl(queueUrl)
        .receiptHandle(msg.receiptHandle())
        .build());
  } catch (Exception e) {
    // don't delete — visibility timeout will redrive
    log.warn("processing failed, will retry", e);
  }
}

Delete only after success. Failed messages reappear after visibility timeout.

3️⃣ FIFO send with message group + dedupjava

sqs.sendMessage(SendMessageRequest.builder()
    .queueUrl(fifoQueueUrl)
    .messageBody(payload)
    .messageGroupId(orderId)           // order within group
    .messageDeduplicationId(eventId)   // 5-min dedup window
    .build());

MessageGroupId enables parallel processing across groups, ordering within group.

4️⃣ Extend visibility for long-running workjava

sqs.changeMessageVisibility(ChangeMessageVisibilityRequest.builder()
    .queueUrl(queueUrl)
    .receiptHandle(msg.receiptHandle())
    .visibilityTimeout(300)  // extend 5 more minutes
    .build());

Heartbeat pattern: extend visibility periodically while processing long jobs.

5️⃣ SQS production checklisttext

- Long polling: waitTimeSeconds=20
- Visibility timeout > p99 processing time
- DLQ with maxReceiveCount (e.g. 3–5)
- Idempotent handler (messageId or business key)
- CloudWatch alarms: ApproximateAgeOfOldestMessage
- Batch receive (max 10) for throughput
- Encrypt at rest (SSE-SQS or KMS) if sensitive

📊

Standard vs FIFO queue

Aspect	Standard Queue	FIFO Queue
Throughput	Nearly unlimited	~300 TPS default (3,000 with high-throughput mode + batching)
Ordering	Best-effort (may arrive out of order)	Strict FIFO within message group
Duplicates	At-least-once — duplicates possible	Content-based deduplication (5-min window) or deduplication ID
Queue name	Any name	Must end with `.fifo`
Use case	High volume, order not critical	Per-user/order sequencing, exactly-once processing intent
Message group	N/A	`MessageGroupId` — parallel groups, ordered within group

⚖️

Key SQS trade-offs

Area	Strength	Caveat	Mitigation
Managed ops	No cluster to run or patch	AWS-specific, less portable	Abstract queue behind interface if multi-cloud needed
Standard throughput	Nearly unlimited scale	Best-effort ordering, duplicates possible	Idempotent consumers; FIFO if order matters
FIFO ordering	Strict order per message group	300 TPS default throughput cap	Shard across multiple MessageGroupIds
Visibility timeout	Prevents double-processing in flight	Too short → duplicate; too long → lag on crash	Tune timeout; use ChangeMessageVisibility heartbeat
Polling model	Simple API; long polling reduces cost	Latency floor vs push brokers	Long poll 20s; batch receive up to 10 messages
Retention	Up to 14 days	Not a replayable event log	Archive to S3/Kinesis if replay needed

🔀

Common Confusions Cleared Up

Visibility timeout lifecycle

After receive, message is invisible to other consumers for visibility timeout duration. If not deleted before timeout expires, it becomes visible again → redelivery. Too short: duplicate processing while still in progress. Too long: slow recovery after worker crash.

Receipt handle is single-use

Each receive returns a unique receipt handle. You must delete or extend visibility with that exact handle. Processing twice requires receiving twice (after visibility timeout) — design handlers idempotently.

SQS vs Kafka mental model

SQS is a task queue — message deleted after processing, no replay. Kafka is a durable log — messages retained, multiple consumer groups replay independently. Pick queue for jobs; pick log for events.

🧪

Variations / Implementations

1) Standard queue + idempotent worker

Default for high throughput. Accept best-effort ordering and possible duplicates. Worker uses idempotency key or dedupe table.

2) FIFO queue + message group

Set MessageGroupId = userId or orderId for per-entity ordering. Different groups process in parallel; same group is strictly ordered.

3) DLQ + redrive

Configure redrive policy: after maxReceiveCount failures, message moves to DLQ. Inspect, fix, and redrive back to source queue for replay.

4) SNS → SQS fan-out

SNS topic publishes to multiple SQS queues. Each subscriber queue gets a copy — decoupled fan-out without custom routing code.

⚠️

Common Mistakes / Anti-Patterns

Short polling in a tight loop — wastes API calls and money.
Deleting message before processing completes.
Visibility timeout shorter than max processing time — duplicate work.
No DLQ — poison messages retry forever, blocking queue.
Non-idempotent handler with at-least-once delivery.
Using Standard queue when strict ordering is required.

🎤

Interview Q&A

🔮

Likely Follow-up Questions

How does Lambda event source mapping handle batch failures?
When would you use SNS + SQS vs EventBridge?
How do you monitor SQS queue health in production?

🧾

Quick Cheat Sheet

DeliveryAt-least-once — idempotent handlers required.

Queue typesStandard (scale) vs FIFO (order + dedup).

PollingLong poll waitTimeSeconds=20.

VisibilityTune > p99 process time; extend if needed.

FailuresDLQ + maxReceiveCount for poison messages.

Fan-outSNS topic → multiple SQS queues.

vs KafkaSQS = task queue; Kafka = replayable log.

🔗