Event-Driven Microservices in Node.js: Apache Kafka vs RabbitMQ Architectural Guide

Architectural Comparison: Apache Kafka vs RabbitMQ

Preventing Dual-Write Inconsistency: The Transactional Outbox

When a user places an order, saving the order to PostgreSQL and publishing a Kafka message in two separate network calls can leave your system out of sync if the server crashes between them. Always write the event into an outbox_events table within the exact same database transaction, then use a background publisher or Debezium Change Data Capture (CDC) to publish events to Kafka reliably.

Production Node.js Kafka Consumer & DLQ Handler

Below is a battle-tested consumer implementation utilizing KafkaJS and Redis for automatic deduplication:

// kafka-consumer.ts - Production Node.js Kafka Consumer with DLQ & Idempotency
import { Kafka, EachMessagePayload, logLevel } from "kafkajs";
import Redis from "ioredis";

const kafka = new Kafka({
  clientId: "order-billing-service",
  brokers: (process.env.KAFKA_BROKERS || "localhost:9092").split(","),
  logLevel: logLevel.ERROR,
});

const redis = new Redis(process.env.REDIS_URL);
const consumer = kafka.consumer({ groupId: "billing-processors-v1" });
const producer = kafka.producer();

export async function startEventConsumer() {
  await consumer.connect();
  await producer.connect();
  await consumer.subscribe({ topic: "orders.events.v1", fromBeginning: false });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }: EachMessagePayload) => {
      const eventKey = message.key?.toString() || "";
      const eventValue = message.value ? JSON.parse(message.value.toString()) : {};
      const idempotencyKey = `processed_event:${eventKey}:${eventValue.eventId}`;

      // 1. Idempotency Check: Prevent duplicate billing on network retries
      const alreadyProcessed = await redis.set(idempotencyKey, "1", "EX", 86400 * 7, "NX");
      if (!alreadyProcessed) {
        console.log(`Skipping duplicate event ${eventValue.eventId}`);
        return;
      }

      try {
        await processOrderBilling(eventValue);
      } catch (error) {
        console.error(`Billing failed for order ${eventKey}, forwarding to DLQ`, error);

        // 2. Dead Letter Queue (DLQ) Forwarding
        await producer.send({
          topic: "orders.events.dlq",
          messages: [
            {
              key: eventKey,
              value: JSON.stringify({
                ...eventValue,
                error: error instanceof Error ? error.message : "Unknown",
                failedAt: new Date().toISOString(),
                originalTopic: topic,
                partition,
              }),
            },
          ],
        });
      }
    },
  });
}

Designing Idempotent Consumers

In distributed systems, networks experience transient timeouts, leading to at-least-once delivery semantics. Your consumers will receive duplicate messages. Guarantee idempotency by:

• Unique Event IDs: Embed a UUIDv7 in the message header.
• Redis Deduplication Lock: Store processed_event:{id} with a 7-day TTL before executing business logic.
• Database Upserts: Use INSERT ... ON CONFLICT DO UPDATE with version checks.

Dead-Letter Queues (DLQ) & Poison Pill Mitigation

A malformed payload or unhandled exception must never block an entire partition stream. Catch fatal errors, wrap the payload with metadata (error trace, timestamp, retry count), and publish it to anorders.events.dlq topic while alerting on-call engineers via PagerDuty.

Event-Driven Microservices Checklist

Frequently Asked Questions

How many partitions should a Kafka topic have?

A good rule of thumb is: Partition Count = Max Expected Throughput / Consumer Processing Rate. Having 6 to 12 partitions per topic allows balanced parallelism across consumer pods without excessive broker metadata overhead.

Can RabbitMQ handle event streaming?

RabbitMQ has introduced RabbitMQ Streams (an append-only log plugin), but Kafka remains the gold standard for large-scale persistent stream processing and event sourcing.

What is consumer lag and why does it matter?

Consumer lag is the delta between the latest published offset and the consumer's current committed offset. Growing lag indicates that your consumers are overwhelmed and you need to scale horizontal worker instances.