As systems grow and evolve, the traditional synchronous request-response model (think standard REST APIs) often reveals its fundamental flaws. When service A calls Service B synchronously, Service A is tightly coupled to Service B’s uptime, latency, and throughput.

In high-throughput environments—like the Ad-Tech platforms I’ve worked on at Getir or large-scale integrations at Definex—this tight coupling becomes a recipe for cascading failures. This is where Event-Driven Architecture (EDA), and specifically Apache Kafka, becomes invaluable.

In this post, I’ll share practical lessons and patterns I’ve learned while transitioning systems from synchronous monoliths to asynchronous, event-driven microservices using Kafka.

1. Demystifying Kafka: It’s a Log, Not a Traditional Queue

Coming from a background of using RabbitMQ or Amazon SQS, my first mistake with Kafka was treating it like a standard message broker. It is not.

Kafka is essentially a distributed commit log. When you write a message (an event) to a Kafka topic, you are appending it to the end of a log file. Consumers read from this log sequentially.

Why this matters:

  • Persistence: Unlike SQS, where a message is deleted once processed, Kafka retains messages for a configured duration (e.g., 7 days). This allows different consumers to read the same data at different times.
  • Replayability: If a bug in your consumer corrupts your database, you can simply fix the bug, reset your Kafka consumer group offset, and “replay” the events to rebuild the correct state. This is a superpower in production environments.

2. The Power of “Dumb Pipes, Smart Endpoints”

In traditional Enterprise Service Bus (ESB) architectures, the middleware was “smart.” It handled routing, transformation, and complex logic. This often led to unmaintainable middleware monoliths.

Kafka flipped this paradigm to “dumb pipes, smart endpoints.” Kafka’s only job is to get bytes from point A to point B as fast and reliably as possible. All the business logic, filtering, and transformation live in the producing and consuming microservices.

// Spring Boot Kafka Producer Example
@Service
public class OrderEventPublisher {
    
    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;
    
    public void publishOrderCreated(Order order) {
        OrderCreatedEvent event = new OrderCreatedEvent(
            order.getId(), 
            order.getCustomerId(), 
            order.getTotalAmount()
        );
        
        // The key (customerId) ensures all orders for the same customer 
        // go to the same partition, guaranteeing order.
        kafkaTemplate.send("orders-topic", order.getCustomerId(), event);
    }
}

3. Partitioning: The Key to Horizontal Scaling

If you want to understand how Kafka scales, you must understand partitions. A topic is divided into partitions, which can be spread across multiple broker nodes.

The Golden Rule: The number of partitions in a topic dictates your maximum consumer concurrency.

If orders-topic has 4 partitions, you can have a maximum of 4 consumer instances in the same consumer group processing messages in parallel. If you spin up a 5th instance, it will sit idle.

The Partition Key Strategy

When publishing an event, you provide a Key. Kafka hashes this key to determine the partition.

  • Using customerId as the key ensures that all events for Customer A always land in Partition 1, and events for Customer B land in Partition 2.
  • Because consumers read partitions sequentially, this guarantees that Customer A’s events are processed in the exact order they occurred relative to each other.

If you don’t provide a key, Kafka distributes events round-robin, which provides excellent load balancing but zero ordering guarantees. Choosing the right partition key is critical to your system design.

4. Designing Idempotent Consumers

In distributed systems, “exactly-once” delivery is notoriously difficult. Network blips happen, consumer nodes crash mid-process, and timeouts occur. Because of this, Kafka at its core provides “at-least-once” delivery semantics.

This means your consumer will occasionally process the same event twice.

Therefore, every single Kafka consumer you write must be idempotent. Processing the same event once or one hundred times must leave the system in the exact same state.

@Service
public class InventoryConsumer {

    @KafkaListener(topics = "orders-topic", groupId = "inventory-service")
    @Transactional
    public void consumeOrder(OrderCreatedEvent event) {
        // 1. Idempotency Check
        if (processedEventRepository.existsById(event.getOrderId())) {
            log.info("Duplicate event detected, ignoring order: {}", event.getOrderId());
            return; 
        }

        // 2. Process Business Logic
        inventoryService.reserveStock(event);

        // 3. Record processed state (in the same transaction)
        processedEventRepository.save(new ProcessedEvent(event.getOrderId()));
    }
}

Note: We check a database table (or a Redis cache) to see if we’ve seen this orderId before.

5. The Transactional Outbox Pattern

One of the hardest problems in microservices is atomically updating the database and publishing an event to Kafka.

// BAD PRACTICE
@Transactional
public void createOrder(OrderRequest request) {
    Order order = db.save(new Order(request)); // Local DB tx
    kafkaTemplate.send("orders", order); // Network call inside a DB tx! 
    // What if Kafka is down? The DB rolls back.
    // What if the DB commits, but the Kafka call fails? You lose the event.
}

The gold standard solution is the Transactional Outbox Pattern:

  1. You create an outbox table in your relational database.
  2. In a single local transaction, you save your business entity (e.g., Order) AND insert a record into the outbox table containing the event payload.
  3. A separate asynchronous process (like Debezium CDC or a polling worker) reads the outbox table and publishes the messages to Kafka.

This absolutely guarantees that if the business transaction succeeds, the event will eventually be published.

Conclusion

Shifting to an Event-Driven Architecture with Kafka requires a paradigm shift in how you think about data state and system interaction. It introduces new complexities around ultimate consistency, dead-letter queues, and schema evolution (which is a topic for another day!).

However, the benefits—massive scalability, deep decoupling of services, and the ability to absorb massive traffic spikes without bringing down edge APIs—make it an essential pattern for modern enterprise software development.

Are you currently working on migrating a system to an event-driven model? What challenges have you run into with Kafka? Connect with me on LinkedIn and let’s discuss your architecture!