Event Streaming #

Behind the large systems we use every day — social media feeds that are always current, credit card fraud detection happening in milliseconds, analytics dashboards moving in real time — there’s one shared foundation: a continuous flow of events being processed relentlessly. Event streaming isn’t just a way to send messages from one service to another. It’s a shift in how you think about data — from something static waiting to be queried, to something moving and constantly flowing. This article covers event streaming from its foundational principles, its fundamental differences from message queues and traditional databases, the anatomy of modern streaming platforms, four real industry use cases, the technical challenges that are often underestimated, and the best practices that separate implementations that last from ones that become operational problems.

What Is Event Streaming? #

Event streaming is a paradigm where events are produced, stored, and consumed continuously by one or many systems. The keyword that sets it apart from other paradigms is continuity — a stream has no defined beginning or end like a file or a batch query. It keeps flowing as long as the system runs.

The most fundamental shift in thinking is about what gets stored:

Traditional database — stores current state:
  orders table: {id: 1, status: "shipped", updated_at: "2026-04-17"}

  ✓ "What is order #1's status now?" → "shipped"
  ✗ "When did its status change from paid to shipped?" → can't be answered

Event streaming — stores the sequence of changes:
  order-events stream:
    offset 0: OrderCreated    {order_id: 1, total: 250000, at: "09:00"}
    offset 1: PaymentReceived {order_id: 1, payment_id: "PAY-001", at: "09:05"}
    offset 2: OrderPacked     {order_id: 1, warehouse: "JKT-1", at: "10:30"}
    offset 3: OrderShipped    {order_id: 1, tracking: "JNE-123", at: "11:00"}

  ✓ "What is order #1's status now?" → replay all events → "shipped"
  ✓ "When did it transition to shipped?" → offset 3, at 11:00
  ✓ "How long from payment to shipped?" → 1 hour 55 minutes
  ✓ "Order #1's state at 10:00?" → replay up to offset 1 → "paid"

Event streaming doesn’t focus on the final state, but on the journey of state changes itself. This is what unlocks capabilities like a complete audit trail, time-travel debugging, and rebuilding state from any point in history.

flowchart LR
    subgraph DB["Traditional Database"]
        W1[Write] --> T1[("(Table:\ncurrent state)")]
        T1 --> R1["Read\ncurrent state"]
    end
    subgraph ES["Event Streaming"]
        W2[Write Event] --> L[("(Append-only\nEvent Log)")]
        L --> R2["Read\ncurrent state\nreplay from start"]
        L --> R3["Read\nhistorical state\nreplay to T"]
        L --> R4["Read\nby multiple\nconsumers"]
    end

Event Streaming vs Message Queue vs Database #

These three paradigms are often mixed up. Understanding the differences is a prerequisite for choosing the right tool — and for not over-engineering a simple system with Kafka just because it sounds advanced.

The easiest analogy for understanding message queue vs event streaming:

• Message Queue is like a factory conveyor belt — items are picked one by one by workers, and once picked, they’re gone from the belt.
• Event Streaming is like a video recording — it can be watched many times by many viewers, new viewers can start from the beginning, and old viewers aren’t affected.

flowchart TD
    subgraph MQ["Message Queue — Consume & Delete"]
        P1[Producer] --> Q["(Queue)"]
        Q -->|"take msg1\n→ msg1 gone"| C1[Consumer A]
        Q -->|"take msg2\n→ msg2 gone"| C2[Consumer B]
    end
    subgraph EV["Event Streaming — Persistent Log"]
        P2[Producer] --> L[("(Event Log\noffset: 0,1,2,3...)")]
        L -->|"read from offset 0\nevents stay"| C3["Consumer A\nEmail Service"]
        L -->|"read from offset 0\nindependent"| C4["Consumer B\nPayment Service"]
        L -->|"read from offset 2\ncan choose where to start"| C5["Consumer C\nAnalytics"]
    end

Anatomy of an Event Streaming Platform #

Topics and Partitions #

A topic is a named stream — a logical channel where similar events are collected. order-events, user-events, payment-events are examples of topics.

A partition is the physical unit inside a topic. One topic is divided into several partitions, each an independent, ordered log. Partitions are the key to horizontal scalability.

flowchart LR
    subgraph Topic["Topic: order-events (3 partitions)"]
        subgraph P0["Partition 0"]
            E0["offset 0\nOrderCreated\norder#1"] --> E3["offset 1\nOrderPaid\norder#1"] --> E6["offset 2\nOrderShipped\norder#1"]
        end
        subgraph P1["Partition 1"]
            E1["offset 0\nOrderCreated\norder#2"] --> E4["offset 1\nOrderPaid\norder#2"] --> E7["offset 2\nOrderShipped\norder#2"]
        end
        subgraph P2["Partition 2"]
            E2["offset 0\nOrderCreated\norder#3"] --> E5["offset 1\nOrderPaid\norder#3"]
        end
    end
    PROD[Producer] -->|"key=order#1"| P0
    PROD -->|"key=order#2"| P1
    PROD -->|"key=order#3"| P2

Choosing the partition key is a critical design decision — it determines ordering and load distribution:

// ANTI-PATTERN: no partition key — events for the same order
// can land in different partitions, ordering isn't guaranteed
producer.SendMessage(&sarama.ProducerMessage{
    Topic: "order-events",
    Value: sarama.ByteEncoder(payload),
    // no Key!
})

// CORRECT: use the entity ID as the partition key
// All events for the same order always land in the same partition
producer.SendMessage(&sarama.ProducerMessage{
    Topic: "order-events",
    Key:   sarama.StringEncoder(event.OrderID), // ← per-order ordering guaranteed
    Value: sarama.ByteEncoder(payload),
})

// Impact: consumers always receive the correct order:
// OrderCreated → PaymentReceived → OrderShipped
// not a random order because events landed in different partitions

Offsets and Consumer Groups #

An offset is the sequence number of each event within a partition. Consumers control which offset has been read themselves — this is what makes replay native.

A consumer group is the horizontal scaling mechanism. Each consumer in a group gets a different subset of partitions — the load is distributed evenly. Two different groups can consume the same topic independently.

flowchart TD
    subgraph T["Topic: order-events (6 partitions)"]
        P0[P0] 
        P1[P1] 
        P2[P2] 
        P3[P3] 
        P4[P4] 
        P5[P5]
    end
    subgraph CG1["Consumer Group: payment-service (3 instances)"]
        I1["Instance A\nP0, P1"]
        I2["Instance B\nP2, P3"]
        I3["Instance C\nP4, P5"]
    end
    subgraph CG2["Consumer Group: email-service (2 instances)"]
        I4["Instance X\nP0, P1, P2"]
        I5["Instance Y\nP3, P4, P5"]
    end
    P0 --> I1
    P1 --> I1
    P2 --> I2
    P3 --> I2
    P4 --> I3
    P5 --> I3
    P0 --> I4
    P1 --> I4
    P2 --> I4
    P3 --> I5
    P4 --> I5
    P5 --> I5

The number of consumer instances in one group can’t exceed the number of partitions. With 6 partitions and 8 consumer instances, 2 instances will sit idle. The partition count must be designed with future scaling needs in mind — partitions can be added but not reduced without rebalancing.

Retention and Log Compaction #

Unlike a message queue that deletes messages after consumption, event streaming keeps events for a configured retention period.

// Log compaction: only the latest event per key is kept
// Good for: the "current state" of each entity
// Before compaction:
// offset 0: UserUpdated {user_id: "123", name: "Umar"}
// offset 5: UserUpdated {user_id: "123", name: "Unis"}
// offset 9: UserUpdated {user_id: "123", name: "Unis Badri"}
//
// After compaction:
// offset 9: UserUpdated {user_id: "123", name: "Unis Badri"}
// offsets 0 and 5 are removed because a newer one exists for key "123"

Four Main Use Cases in Industry #

1. Microservices Communication #

Event streaming as the communication backbone between microservices — replacing synchronous API calls for workflows that don’t need an immediate response.

flowchart TD
    subgraph Before["❌ Synchronous Chain — Vulnerable to Cascading Failures"]
        OS1[Order Service] -->|POST| PS1[Payment Service]
        PS1 -->|POST| IS1[Inventory Service]
        IS1 -->|POST| ES1[Email Service]
        ES1 -- "down!" --> IS1
        IS1 -- "error!" --> PS1
        PS1 -- "error!" --> OS1
    end
    subgraph After["✅ Event Streaming — Fault Isolated"]
        OS2[Order Service] -->|publish\norder.created| K["(Kafka)"]
        K --> PS2[Payment Service]
        K --> IS2[Inventory Service]
        K --> ES2["Email Service\ndown? events queue up\nprocessed when it recovers"]
    end

// ANTI-PATTERN: the order service calls all downstream services directly
func (s *OrderService) CreateOrder(req CreateOrderRequest) error {
    order := s.db.CreateOrder(req)
    s.paymentSvc.InitiatePayment(order.ID)   // if it fails → order fails
    s.inventorySvc.ReserveStock(order.Items) // if it fails → order fails
    s.emailSvc.SendConfirmation(order.UserID) // if it fails → order fails
    return nil
}

// CORRECT: publish one event, each service reacts independently
func (s *OrderService) CreateOrder(req CreateOrderRequest) error {
    order := s.db.CreateOrder(req)
    
    s.kafka.Publish("order.created", OrderCreatedEvent{
        EventID:   uuid.New().String(),
        OrderID:   order.ID,
        UserID:    order.UserID,
        Items:     order.Items,
        CreatedAt: time.Now(),
    })
    return nil // done — doesn't care who consumes it
}

2. Real-Time Analytics and Stream Processing #

Event streaming enables analytics computed directly from the stream, instead of batch queries to a database.

// Concept: compute revenue per minute using stream processing
func buildRevenueStream(orderEvents KStream) KTable {
    return orderEvents.
        Filter(func(e OrderEvent) bool {
            return e.Type == "order.paid"
        }).
        GroupBy(func(e OrderEvent) string {
            return e.OccurredAt.Truncate(time.Minute).Format(time.RFC3339)
        }).
        Aggregate(
            func() int64 { return 0 },
            func(key string, event OrderEvent, agg int64) int64 {
                return agg + event.Amount
            },
        )
    // Result: a {minute → total_revenue} table updated in real time
    // without waiting for an hourly batch job
}

Real use cases using this pattern: real-time fraud detection (anomalous transaction patterns within a 5-minute window), live leaderboards, real-time API latency monitoring, product recommendations updated as users browse.

3. Change Data Capture (CDC) #

CDC is a technique for capturing every change in a database (INSERT, UPDATE, DELETE) and publishing it as events to a stream — without polling or triggers that burden the primary database.

flowchart LR
    DB[("(PostgreSQL\nDatabase)")] -->|"WAL / binlog\nreading"| D["Debezium\nConnector"]
    D -->|"publish change events"| K["(Kafka)"]
    K --> ES["Elasticsearch\nupdate search index"]
    K --> RC["Redis\ninvalidate cache"]
    K --> DW["Data Warehouse\nnew record"]
    K --> AU["Audit Service\nlog changes"]

// CDC event produced by Debezium for every UPDATE in the database
type CDCEvent struct {
    Before  map[string]interface{} `json:"before"` // state before
    After   map[string]interface{} `json:"after"`  // state after
    Op      string                 `json:"op"`     // "c"=create, "u"=update, "d"=delete
    Source  CDCSource              `json:"source"`
}

// Consumer uses CDC to invalidate cache automatically
func (c *CacheConsumer) HandleCDC(event CDCEvent) error {
    if event.Op == "u" || event.Op == "d" {
        userID := event.Before["id"].(string)
        return c.redis.Del(ctx, "user:"+userID)
    }
    return nil
}

4. Event Sourcing as the Source of Truth #

In event sourcing, the event stream IS the primary database — not an add-on. Application state is reconstructed by replaying events from the start.

// ANTI-PATTERN: storing only the final state — losing the history
func (s *AccountService) Debit(accountID string, amount int64) error {
    account := s.db.FindAccount(accountID)
    account.Balance -= amount
    return s.db.Save(account)
    // The question "why is the balance this amount?" → can't be answered
}

// CORRECT: store events, rebuild state from replay
func (s *AccountService) Debit(accountID string, amount int64) error {
    event := AccountDebitedEvent{
        AccountID: accountID,
        Amount:    amount,
        OccurredAt: time.Now(),
    }
    return s.eventStore.Append(accountID, event)
}

func (s *AccountService) GetBalance(accountID string) int64 {
    events := s.eventStore.LoadEvents(accountID)
    account := Account{Balance: 0}
    for _, e := range events {
        switch ev := e.(type) {
        case AccountCreditedEvent:
            account.Balance += ev.Amount
        case AccountDebitedEvent:
            account.Balance -= ev.Amount
        }
    }
    return account.Balance
    // Bonus: you can query "what was the balance on date X?" by filtering timestamps
}

Technical Challenges Often Underestimated #

Ordering Is Only Guaranteed Per Partition #

This is one of the most common misconceptions. Event streaming doesn’t guarantee global ordering across the whole topic. Ordering is only guaranteed within a single partition.

// ANTI-PATTERN: no partition key — related events
// can land in different partitions, consumers see the wrong order
//
// Partition 0: OrderCreated(id=1), OrderShipped(id=2)
// Partition 1: PaymentReceived(id=1), OrderShipped(id=1)
//
// A consumer might see: OrderShipped(id=1) before OrderCreated(id=1)!

// CORRECT: partition key = entity ID
//
// Partition 0: OrderCreated(id=1), PaymentReceived(id=1), OrderShipped(id=1)
// Partition 1: OrderCreated(id=2), PaymentReceived(id=2), OrderShipped(id=2)
//
// All events for order #1 are always in Partition 0 → ordering guaranteed

producer.SendMessage(&sarama.ProducerMessage{
    Topic: "order-events",
    Key:   sarama.StringEncoder(event.OrderID), // ← the ordering key
    Value: sarama.ByteEncoder(payload),
})

Exactly-Once Semantics #

Exactly-once is the hardest and most expensive guarantee. In practice, at-least-once + idempotent consumers is a far more realistic solution.

// ANTI-PATTERN: committing the offset before processing — events can be lost
msg := consumer.Receive()
consumer.CommitOffset(msg.Offset) // ← committed too early
processEvent(msg)                  // crash here = event lost forever

// CORRECT: commit the offset AFTER successful processing
func (c *Consumer) processLoop() {
    for msg := range c.messages {
        if err := c.processEvent(msg); err != nil {
            // Don't commit — the message will be redelivered
            log.Errorf("processing failed for offset %d: %v", msg.Offset, err)
            continue
        }
        // Commit only after success
        c.session.MarkMessage(msg, "")
    }
}

// Idempotent consumer — safe even if the event arrives 2x
func (c *Consumer) processEvent(msg *sarama.ConsumerMessage) error {
    var event OrderEvent
    json.Unmarshal(msg.Value, &event)
    
    if processed, _ := c.idempotencyStore.Exists(ctx, event.EventID); processed {
        return nil // already processed — skip safely
    }
    
    if err := c.businessLogic(event); err != nil {
        return err
    }
    
    c.idempotencyStore.Set(ctx, event.EventID, time.Now())
    return nil
}

Schema Evolution with a Schema Registry #

When many producers and consumers operate on the same events, schema changes can break consumers that haven’t been redeployed.

flowchart TD
    subgraph Without["❌ Without a Schema Registry"]
        P1["Producer\nsends new format\namount_cents"] --> K1["(Kafka)"]
        K1 --> C1["Old Consumer\nexpects amount\n💥 panic / error"]
    end
    subgraph With["✅ With a Schema Registry"]
        P2["Producer\nregisters new schema"] --> SR[("(Schema\nRegistry)")]
        SR -->|"check compatibility"| OK{Compatible?}
        OK -->|Yes| K2["(Kafka)"]
        OK -->|No| REJ["❌ Reject\nbreaking change"]
        K2 --> C2["Consumer\ndeserializes with\nthe correct schema"]
    end

// ANTI-PATTERN: renaming a field directly — breaking change without warning!
// Before: {"user_id": "123", "amount": 250000}
// After:  {"customer_id": "123", "amount_cents": 25000000} ← old consumers crash!

// CORRECT: backward-compatible evolution — add new fields, don't remove old ones
// Schema v1
type OrderPaidEventV1 struct {
    EventID string  `json:"event_id"`
    OrderID string  `json:"order_id"`
    UserID  string  `json:"user_id"`  // stays
    Amount  float64 `json:"amount"`   // stays
}

// Schema v2 — backward compatible
type OrderPaidEventV2 struct {
    EventID    string  `json:"event_id"`
    OrderID    string  `json:"order_id"`
    UserID     string  `json:"user_id"`       // stays for compatibility
    CustomerID string  `json:"customer_id"`   // new field (alias)
    Amount     float64 `json:"amount"`        // stays for compatibility
    AmountCents int64  `json:"amount_cents"`  // new field (alias)
    PromoCode  *string `json:"promo_code,omitempty"` // new field, optional
}
// v1 consumers can still read v2 events because the old fields remain
// v2 consumers read the richer new fields

Never change or delete fields that already exist in an event schema being consumed in production. Use a Confluent Schema Registry or protocols like Protobuf / Avro that have built-in compatibility rules. Uncontrolled schema evolution is one of the biggest causes of incidents in Kafka-based systems.

Event Streaming Design Best Practices #

Events Are Public Contracts #

Once an event topic is published and consumers depend on it, it becomes a contract that must be maintained like a public API.

Design principles for events as public contracts:

  □ Event names based on facts (past-tense verbs): OrderCreated, not CreateOrder
  □ All fields must be backward-compatible — no deletion or type changes
  □ New fields are always optional with safe default values
  □ Breaking change → create a new versioned event type (order.created.v2)
  □ Run old and new versions in parallel during the transition
  □ Document in a schema registry or internal wiki
  □ Don't publish internal implementation details to topics consumed by other services

Partition Key Strategy #

For most business use cases, Entity ID is the best choice because it provides the most meaningful semantic ordering.

Consumer Lag Monitoring Is Mandatory #

Consumer lag is the most important metric in event streaming — the difference between the newest produced offset and the newest consumed offset.

Consumer lag = latest_offset - consumer_committed_offset

Lag = 0         → consumer is real-time, no backlog ✓
Lag = 1.000     → 1000 events not yet processed
Lag keeps rising → consumer can't keep up with the producer
                → needs consumer scale-out or processing logic optimization

// CORRECT: monitor consumer lag and alert when it exceeds thresholds
func monitorConsumerLag(client sarama.Client, group, topic string) {
    ticker := time.NewTicker(30 * time.Second)
    for range ticker.C {
        lag, err := calculateTotalLag(client, group, topic)
        if err != nil {
            log.Errorf("failed to calculate consumer lag: %v", err)
            continue
        }

        // Send to the metrics system
        metrics.Gauge("kafka.consumer.lag", float64(lag), map[string]string{
            "group": group,
            "topic": topic,
        })

        // Tiered alerting
        if lag > 100_000 {
            alert.Critical(fmt.Sprintf("[KAFKA] critical consumer lag: group=%s topic=%s lag=%d", group, topic, lag))
        } else if lag > 10_000 {
            alert.Warning(fmt.Sprintf("[KAFKA] high consumer lag: group=%s topic=%s lag=%d", group, topic, lag))
        }
    }
}

Anti-Patterns to Avoid #

// ✗ Events too granular — high communication overhead, hard to manage
kafka.Publish("user.first-name.updated", event)
kafka.Publish("user.last-name.updated", event)
kafka.Publish("user.email.updated", event)
// ✓ Events at a meaningful business level
kafka.Publish("user.profile.updated", event) // one event, one intent

// ✗ Events containing commands, not facts
kafka.Publish("send-welcome-email", event)  // command!
kafka.Publish("process-payment", event)     // command!
// ✓ Events containing historical facts
kafka.Publish("user.registered", event)         // fact
kafka.Publish("order.payment.received", event)  // fact

// ✗ Committing the offset before processing — events can be lost on crash
session.MarkMessage(msg, "") // commit first
processEvent(msg)             // crash here = event lost
// ✓ Commit AFTER successful processing
if err := processEvent(msg); err != nil { return err }
session.MarkMessage(msg, "") // commit after success

// ✗ Non-idempotent consumer — the same event processed 2x
func handle(event Event) { db.Insert(event.Data) } // duplicate on redelivery!
// ✓ Check event_id first
func handle(event Event) {
    if idempotencyStore.Exists(event.EventID) { return nil }
    db.Insert(event.Data)
    idempotencyStore.Set(event.EventID)
}

// ✗ No consumer lag monitoring — learning about problems from user complaints
// ✓ Automatic alerts when lag exceeds the threshold, before users feel the impact

// ✗ Schema changes without versioning — old consumers suddenly crash
// ✓ Schema Registry + backward-compatible evolution + versioned event types

Event Streaming Implementation Checklist #

TOPIC DESIGN:
  □ Topic names reflect the domain, not the implementation (order-events, not kafka-order-svc)
  □ Partition count planned with scaling needs in mind
  □ Retention policy set according to needs (7 days, 30 days, infinite)
  □ Log compaction enabled if the topic is used as a state store

PRODUCER:
  □ Partition keys set for all events that need ordering
  □ Event schemas documented in a schema registry
  □ All events have: event_id, event_type, occurred_at, correlation_id

CONSUMER:
  □ Offsets committed AFTER successful processing, not before
  □ Consumers idempotent — check event_id before processing
  □ DLQ configured for events that exceed max retries
  □ Descriptive consumer group names (payment-service, not consumer-1)

SCHEMA:
  □ Schema registry used (Confluent Schema Registry / Protobuf)
  □ Backward-compatible evolution for minor changes
  □ Versioned event types for breaking changes
  □ No field deletion from schemas already in production

OBSERVABILITY:
  □ Consumer lag monitoring active for all consumer groups
  □ Consumer lag alerts above the established threshold
  □ Correlation IDs propagated from events to consumer logs
  □ Distributed tracing covers the flow from producer to consumer

Summary #

• Event streaming stores the sequence of changes, not current state — this unlocks complete audit trails, time-travel debugging, and rebuilding state from any point.
• Not a message queue: event streaming keeps events after consumption (retention), supports many independent consumers, and native replay; message queues delete messages after consumption.
• Topics and partitions: topics are logical channels, partitions are the physical units enabling horizontal scaling — the partition count is the upper bound of consumer group parallelism.
• Use the entity ID as the partition key — ordering is only guaranteed per partition; without the right key, related events can be processed out of order.
• Consumer groups: each group receives all events independently; instances within a group share partition load; instances must never exceed the partition count.
• At-least-once + idempotent consumers is the most pragmatic trade-off — exactly-once is very expensive and rarely truly needed.
• Commit offsets AFTER processing — committing before processing is a recipe for event loss when a consumer crashes.
• A Schema Registry is mandatory in multi-team environments — it prevents accidental breaking schema changes from destroying production consumers.
• Consumer lag is the most important metric — monitoring and alerting must exist before problems are detected by users, not after.
• Events are public contracts — treat them like public APIs: backward-compatible for minor evolution, versioning for breaking changes.

← Previous: Event-Driven   Next: Inversion of Control →