Queue Systems Comparison - Redis, RabbitMQ, AWS SQS, and More

queues redis rabbitmq sqs message-broker job-queue

Queue Systems Comparison

Choosing the right queue system depends on your requirements: scale, reliability, complexity, and budget. This guide compares popular queue systems from simple in-memory queues to enterprise message brokers.

Queue System Categories

1. Job Queues

Focus: Background job processing

Examples: BullMQ (Redis), Sidekiq (Ruby), Celery (Python)

// Add job to queue
await queue.add('send-email', {
  to: '[email protected]',
  subject: 'Welcome!',
});

// Worker processes job
queue.process('send-email', async (job) => {
  await sendEmail(job.data);
});

Best for: Background tasks in a single application

2. Message Brokers

Focus: Communication between services

Examples: RabbitMQ, Apache Kafka, AWS SNS/SQS

// Publish message
await broker.publish('orders', {
  type: 'order.created',
  orderId: '123',
});

// Multiple services subscribe
await broker.subscribe('orders', handleOrder);

Best for: Microservices, event-driven architecture

3. Cloud Queue Services

Focus: Managed, scalable queuing

Examples: AWS SQS, Google Cloud Tasks, Azure Queue Storage

// Send message to SQS
await sqs.sendMessage({
  QueueUrl: queueUrl,
  MessageBody: JSON.stringify({ orderId: '123' }),
});

Best for: Cloud-native applications, no ops burden

Redis-Based Queues (BullMQ)

Overview

BullMQ is a Node.js queue library backed by Redis. Most popular choice for JavaScript/TypeScript applications.

Architecture:

Producer → Redis → Worker(s)

Setup

# Install
npm install bullmq ioredis

# Requires Redis server
docker run -d -p 6379:6379 redis:7

import { Queue, Worker } from 'bullmq';

// Create queue
const emailQueue = new Queue('emails', {
  connection: {
    host: 'localhost',
    port: 6379,
  },
});

// Add jobs
await emailQueue.add('send-welcome', {
  email: '[email protected]',
  name: 'John',
});

// Process jobs
const worker = new Worker('emails', async (job) => {
  console.log(`Processing ${job.name}`, job.data);
  await sendEmail(job.data);
}, {
  connection: { host: 'localhost', port: 6379 },
});

worker.on('completed', (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on('failed', (job, err) => {
  console.log(`Job ${job.id} failed:`, err);
});

Features

Job Management:

// Delayed jobs
await queue.add('reminder', { userId: 123 }, {
  delay: 3600000,  // 1 hour
});

// Scheduled jobs (recurring)
await queue.add('daily-report', {}, {
  repeat: {
    cron: '0 9 * * *',  // Every day at 9am
  },
});

// Priority
await queue.add('urgent', data, { priority: 10 });
await queue.add('normal', data, { priority: 5 });

// Retries with backoff
await queue.add('flaky-api', data, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 2000,
  },
});

Concurrency:

// Process 10 jobs at once
const worker = new Worker('emails', processJob, {
  concurrency: 10,
});

Rate Limiting:

// Max 100 jobs per minute
const worker = new Worker('api-calls', processJob, {
  limiter: {
    max: 100,
    duration: 60000,
  },
});

Pros

✅ Fast (in-memory Redis)
✅ Simple setup
✅ Rich features (delays, repeats, priorities)
✅ Great for Node.js/TypeScript
✅ Good dashboard (Bull Board)
✅ Mature, widely used

Cons

❌ Requires Redis (additional infrastructure)
❌ Limited to single Redis instance (or cluster)
❌ Not true message broker (no pub/sub routing)
❌ Jobs lost if Redis crashes (without persistence)
❌ No cross-language support (Node.js only)

When to Use

✅ Node.js/TypeScript applications
✅ Background job processing
✅ Need delays, retries, priorities
✅ Single application (not microservices)
✅ <1M jobs/day

Pricing

Self-hosted Redis:
- Free (local dev)
- $15-50/mo (managed Redis, small)
- $100-500/mo (managed Redis, production)

Redis Cloud:
- Free: 30MB
- $5/mo: 250MB
- $28/mo: 1GB

RabbitMQ

Overview

Full-featured message broker with advanced routing, supports multiple protocols (AMQP, MQTT, STOMP).

Architecture:

Producer → Exchange → Queue → Consumer

Setup

# Docker
docker run -d -p 5672:5672 -p 15672:15672 rabbitmq:3-management

# Or use CloudAMQP (managed)

import amqplib from 'amqplib';

// Connect
const connection = await amqplib.connect('amqp://localhost');
const channel = await connection.createChannel();

// Create queue
await channel.assertQueue('tasks', { durable: true });

// Publish message
channel.sendToQueue('tasks', Buffer.from(JSON.stringify({
  type: 'send-email',
  email: '[email protected]',
})), {
  persistent: true,
});

// Consume messages
await channel.consume('tasks', async (msg) => {
  if (msg) {
    const data = JSON.parse(msg.content.toString());
    await processTask(data);
    channel.ack(msg);
  }
});

Advanced Features

Exchanges and Routing:

// Topic exchange (pattern matching)
await channel.assertExchange('logs', 'topic');

// Route by pattern
await channel.publish('logs', 'error.payment', Buffer.from(msg));
await channel.publish('logs', 'info.auth', Buffer.from(msg));

// Consumer subscribes to patterns
await channel.bindQueue(queue, 'logs', 'error.*');  // All errors
await channel.bindQueue(queue, 'logs', '*.payment'); // All payment logs

Dead Letter Queues:

// Failed messages go to DLQ
await channel.assertQueue('tasks', {
  deadLetterExchange: 'dlx',
  deadLetterRoutingKey: 'failed',
});

Message TTL:

// Message expires after 1 hour
channel.sendToQueue('tasks', msg, {
  expiration: '3600000',
});

Pros

✅ True message broker (pub/sub, routing)
✅ Multiple protocols (AMQP, MQTT, STOMP)
✅ Advanced routing (topics, headers)
✅ Strong reliability guarantees
✅ Language-agnostic (client libraries for all languages)
✅ Mature and battle-tested
✅ Clustering and federation

Cons

❌ More complex than job queues
❌ Requires learning AMQP concepts
❌ Higher operational overhead
❌ Slower than Redis for simple jobs
❌ More memory-intensive

When to Use

✅ Microservices architecture
✅ Complex routing requirements
✅ Need message durability guarantees
✅ Multi-language services
✅ Event-driven architecture
✅ >1M messages/day

Pricing

Self-hosted:
- Free (local dev)
- $50-200/mo (VPS, small)
- $500-2000/mo (production cluster)

CloudAMQP (managed):
- Free: 1M messages/mo, 3 connections
- $19/mo: 20M messages/mo
- $99/mo: 100M messages/mo
- $399/mo: Dedicated instance

Apache Kafka

Overview

Distributed streaming platform designed for high-throughput, real-time data pipelines.

Architecture:

Producer → Topic (Partitions) → Consumer Group

Setup

# Docker Compose
docker-compose up -d  # Zookeeper + Kafka

# Or use Confluent Cloud, AWS MSK

import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'],
});

// Producer
const producer = kafka.producer();
await producer.connect();

await producer.send({
  topic: 'orders',
  messages: [
    {
      key: 'order-123',
      value: JSON.stringify({ orderId: '123', amount: 99.99 }),
    },
  ],
});

// Consumer
const consumer = kafka.consumer({ groupId: 'order-processor' });
await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: true });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const data = JSON.parse(message.value.toString());
    await processOrder(data);
  },
});

Features

Partitions (Scalability):

Topic: orders
├── Partition 0: [msg1, msg4, msg7]
├── Partition 1: [msg2, msg5, msg8]
└── Partition 2: [msg3, msg6, msg9]

3 consumers can process in parallel

Consumer Groups:

// Multiple consumers share load
// Each message processed by one consumer in group

const consumer1 = kafka.consumer({ groupId: 'processors' });
const consumer2 = kafka.consumer({ groupId: 'processors' });
const consumer3 = kafka.consumer({ groupId: 'processors' });

// Load balanced across 3 consumers

Stream Processing:

// Process events in real-time
await consumer.run({
  eachMessage: async ({ message }) => {
    const event = JSON.parse(message.value);

    // Real-time analytics
    await updateMetrics(event);
    await detectFraud(event);
    await triggerAlerts(event);
  },
});

Pros

✅ Extremely high throughput (millions/sec)
✅ Horizontally scalable (partitions)
✅ Message replay (retain messages)
✅ Stream processing (real-time analytics)
✅ Strong ordering guarantees (per partition)
✅ Battle-tested at scale (LinkedIn, Netflix, Uber)

Cons

❌ Very complex (Zookeeper, brokers, partitions)
❌ Heavy operational burden
❌ Overkill for simple job queues
❌ Not designed for job scheduling
❌ Requires learning Kafka concepts
❌ Expensive (hardware, managed services)

When to Use

✅ Event streaming (logs, metrics, user activity)
✅ Real-time analytics
✅ Extremely high throughput (>100k/sec)
✅ Need message replay
✅ Building data pipelines
✅ Large-scale microservices

Pricing

Self-hosted:
- $500-5000+/mo (cluster, depends on scale)

Confluent Cloud:
- $1/mo base + $0.11/GB ingress + $0.05/GB egress
- Typical: $100-1000+/mo

AWS MSK (Managed Streaming for Kafka):
- $73/mo minimum (2 brokers)
- $300-1000+/mo typical production

AWS SQS (Simple Queue Service)

Overview

Fully managed queue service from AWS. No servers to manage, pay per use.

Types:

Standard Queue: Unlimited throughput, at-least-once delivery
FIFO Queue: Ordered, exactly-once delivery, 300 msg/sec

Setup

import { SQSClient, SendMessageCommand, ReceiveMessageCommand } from '@aws-sdk/client-sqs';

const client = new SQSClient({ region: 'us-east-1' });

// Send message
await client.send(new SendMessageCommand({
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789/my-queue',
  MessageBody: JSON.stringify({
    type: 'order-created',
    orderId: '123',
  }),
  DelaySeconds: 0,
}));

// Receive messages
const response = await client.send(new ReceiveMessageCommand({
  QueueUrl: queueUrl,
  MaxNumberOfMessages: 10,
  WaitTimeSeconds: 20,  // Long polling
}));

for (const message of response.Messages || []) {
  const data = JSON.parse(message.Body);
  await processMessage(data);

  // Delete after processing
  await client.send(new DeleteMessageCommand({
    QueueUrl: queueUrl,
    ReceiptHandle: message.ReceiptHandle,
  }));
}

Features

Delay Queues:

// Message available after 15 minutes
await client.send(new SendMessageCommand({
  QueueUrl: queueUrl,
  MessageBody: JSON.stringify(data),
  DelaySeconds: 900,
}));

Dead Letter Queues:

// After 3 failed attempts, send to DLQ
// Configure in AWS Console or CloudFormation

FIFO Queues:

// Guaranteed order, exactly-once delivery
await client.send(new SendMessageCommand({
  QueueUrl: fifoQueueUrl,
  MessageBody: JSON.stringify(data),
  MessageGroupId: 'order-processing',  // Ensures order
  MessageDeduplicationId: 'unique-id',  // Prevents duplicates
}));

Pros

✅ Fully managed (no servers)
✅ Unlimited scale
✅ Pay per use ($0.40 per million requests)
✅ Integrates with AWS services (Lambda, SNS, etc.)
✅ No operational burden
✅ High availability (replicated)

Cons

❌ AWS lock-in
❌ Limited to 15-minute delay max
❌ No advanced routing (use SNS for fan-out)
❌ 256KB message size limit
❌ Polling-based (not push)
❌ FIFO limited to 300 msg/sec

When to Use

✅ AWS-based applications
✅ Want fully managed service
✅ Need unlimited scale
✅ Microservices on AWS (Lambda, ECS)
✅ Don't want to manage infrastructure

Pricing

Standard Queue:
- First 1M requests/mo: Free
- $0.40 per million requests after

FIFO Queue:
- First 1M requests/mo: Free
- $0.50 per million requests after

Example (1M messages/day):
- 30M requests/mo = $12/mo
- Plus data transfer costs

Google Cloud Tasks / Pub/Sub

Cloud Tasks

Task queue for asynchronous execution.

import { CloudTasksClient } from '@google-cloud/tasks';

const client = new CloudTasksClient();

// Create task
await client.createTask({
  parent: queuePath,
  task: {
    httpRequest: {
      url: 'https://my-service.com/process',
      body: Buffer.from(JSON.stringify(data)).toString('base64'),
    },
    scheduleTime: {
      seconds: Date.now() / 1000 + 3600,  // 1 hour from now
    },
  },
});

Best for: Scheduled tasks, HTTP-based processing

Pub/Sub

Messaging service for event-driven systems.

import { PubSub } from '@google-cloud/pubsub';

const pubsub = new PubSub();

// Publish
await pubsub.topic('orders').publishMessage({
  data: Buffer.from(JSON.stringify({ orderId: '123' })),
});

// Subscribe
const subscription = pubsub.subscription('order-processor');
subscription.on('message', async (message) => {
  const data = JSON.parse(message.data.toString());
  await processOrder(data);
  message.ack();
});

Best for: Event-driven architectures, pub/sub patterns

Pricing

Cloud Tasks:
- Free: 1M operations/mo
- $0.40 per million after

Pub/Sub:
- Free: 10GB/mo
- $40/TB after
- Typical: $10-100/mo

Azure Queue Storage / Service Bus

Queue Storage

Simple queue service.

import { QueueServiceClient } from '@azure/storage-queue';

const client = QueueServiceClient.fromConnectionString(connectionString);
const queueClient = client.getQueueClient('tasks');

// Send message
await queueClient.sendMessage(JSON.stringify({ task: 'process' }));

// Receive message
const messages = await queueClient.receiveMessages({ numberOfMessages: 10 });
for (const message of messages.receivedMessageItems) {
  await processMessage(message);
  await queueClient.deleteMessage(message.messageId, message.popReceipt);
}

Service Bus

Enterprise messaging service (similar to RabbitMQ).

Best for: Complex routing, topics, subscriptions

Pricing

Queue Storage:
- $0.05 per million operations
- Very cheap

Service Bus:
- Basic: $0.05 per million operations
- Standard: $10/mo + $0.80 per million
- Premium: $677/mo (dedicated resources)

Decision Matrix

By Use Case

Use Case	Best Choice	Why
Background jobs (Node.js)	BullMQ	Simple, feature-rich
Microservices (Docker)	RabbitMQ	Language-agnostic, routing
Event streaming	Kafka	High throughput, replay
AWS application	SQS	Managed, integrates with AWS
GCP application	Cloud Tasks / Pub/Sub	Managed, integrates with GCP
Scheduled tasks	BullMQ or Cloud Tasks	Built-in scheduling
High-throughput (>1M/sec)	Kafka	Designed for scale
Simple job queue	BullMQ	Easiest setup

By Scale

Scale	Recommendation
<1k jobs/day	BullMQ (simple)
1k-100k jobs/day	BullMQ or SQS
100k-1M jobs/day	RabbitMQ or SQS
1M-10M jobs/day	RabbitMQ or Kafka
>10M jobs/day	Kafka

By Architecture

Architecture	Recommendation
Monolith	BullMQ
Microservices (same language)	BullMQ or SQS
Microservices (multiple languages)	RabbitMQ
Event-driven	RabbitMQ or Kafka
Serverless (AWS)	SQS + Lambda
Real-time streaming	Kafka

Cost Comparison (100k messages/day)

Service	Monthly Cost	Notes
BullMQ (Redis)	$15-50	Managed Redis (small)
RabbitMQ (CloudAMQP)	$19	20M messages/mo tier
AWS SQS	$1.20	3M requests * $0.40/M
Google Cloud Tasks	$0.80	3M operations * $0.40/M
Kafka (self-hosted)	$100-500	VPS/cluster costs
Kafka (Confluent Cloud)	$100+	Varies by throughput

Key Takeaways

Start simple:

Small app → BullMQ (Redis)
AWS app → SQS
GCP app → Cloud Tasks

Scale up:

Microservices → RabbitMQ
High throughput → Kafka
Need reliability → RabbitMQ with persistence

Consider:

Complexity: BullMQ < SQS < RabbitMQ < Kafka
Cost: SQS < BullMQ < RabbitMQ < Kafka
Scale: BullMQ < RabbitMQ < SQS < Kafka
Features: SQS < BullMQ < RabbitMQ < Kafka

Most common path: Start with BullMQ, migrate to RabbitMQ if you outgrow it or need cross-language support.

Resources

BullMQ: docs.bullmq.io
RabbitMQ: rabbitmq.com/documentation
Apache Kafka: kafka.apache.org/documentation
AWS SQS: docs.aws.amazon.com/sqs
Message Queue Patterns: Enterprise Integration Patterns