Microservices #

Microservices is an architectural approach where an application is built as a collection of small independent services — each running in its own process, communicating through well-defined APIs, and deployable separately. Each service is responsible for one specific business domain.

What’s often not realized: microservices aren’t about size (not “small” in terms of lines of code), but about independence. Services that can be developed, tested, deployed, and scaled independently of other services — that’s what makes this architecture valuable.

And what’s even less often realized: microservices bring very significant complexity. Teams not ready to face distributed systems will find themselves with all the weaknesses of monoliths plus all the weaknesses of distributed systems — without the benefits of either. Martin Fowler calls this the “distributed monolith” — the worst anti-pattern of both worlds.

Monoliths First, Microservices Later #

One of the most common mistakes in software engineering is starting directly with microservices. It feels like an ambitious, forward-thinking decision, but it almost always leads to problems.

Why starting with a monolith is wiser:

  Project beginnings:
  → Business domains aren't fully understood yet
  → Wrong boundaries are very expensive to change in microservices
  → Small teams don't need the deployment independence microservices claim
  → Operational overhead doesn't justify the benefits

  Good monoliths (modular monoliths):
  → Code structured with clear module boundaries
  → Modules only communicate through defined interfaces
  → Easy to refactor into microservices later
  → Far easier to develop, test, and debug

  When microservices start making sense:
  → Teams are large enough (usually 15+ engineers)
  → Clear scaling bottlenecks specific to domains
  → High deployment frequency where different teams need independent deployments
  → Business domains well understood (not assumed)
  → Teams experienced with distributed systems

graph LR
    subgraph Monolith["Modular Monolith — Start here"]
        A[User Module] --> B[Order Module]
        A --> C[Payment Module]
        B --> C
        B --> D[Inventory Module]
    end

    subgraph Microservices["Microservices — Evolve when clear needs exist"]
        E[User Service] -->|API| F[Order Service]
        F -->|API| G[Payment Service]
        F -->|API| H[Inventory Service]
        E -->|API| G
    end

    Monolith -->|Evolve when real needs appear| Microservices

Defining Correct Service Boundaries #

Wrong service boundaries are the root of most problems in microservice architectures. Overly granular boundaries create chatty, interdependent services. Overly wide boundaries lose the independence benefits.

Domain-Driven Design (DDD) provides a framework for defining correct boundaries through the concept of Bounded Contexts — each service should correspond to one bounded context, where the domain model has consistent, unambiguous meaning.

Service boundary definition principles:

  ✓ Aligned with business domains, not technical layers
    Don't: UserDatabase Service, OrderRepository Service
    Do:    User Service (everything about users), Order Service (everything about orders)

  ✓ High cohesion, loose coupling
    High cohesion: everything in this service is closely related
    Loose coupling: services don't need to know other services' implementation details

  ✓ Services own their own databases (database per service)
    No database sharing between services
    This is what enables true deployment independence

  ✓ Services deployable without changing other services
    If changing one service always requires changing another → wrong boundary

  ✗ Signs of wrong boundaries:
    → Service A is always called together with Service B (should be one service)
    → Changing Service A's domain model always requires changing Service B
    → One user request needs 10+ service calls to complete (too granular)
    → Services without their own state (only orchestrators without domain logic)

Inter-Service Communication: Synchronous vs Asynchronous #

The choice of how services communicate with each other is one of the most important decisions in microservices.

Synchronous Communication (REST/gRPC) #

// Service A calls Service B synchronously.
// HTTP client with timeouts and retries.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

// getUserProfile calls the User Service, retrying with
// exponential backoff on failure.
func getUserProfile(ctx context.Context, userID int) (map[string]interface{}, error) {
    client := &http.Client{Timeout: 5 * time.Second}

    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet,
            fmt.Sprintf("http://user-service/users/%d", userID), nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("X-Request-ID", getRequestID()) // propagate trace context

        resp, err := client.Do(req)
        if err == nil && resp.StatusCode < 500 {
            defer resp.Body.Close()
            var result map[string]interface{}
            if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
                return nil, err
            }
            return result, nil
        }
        if resp != nil {
            resp.Body.Close()
        }
        lastErr = err

        // Exponential backoff: 1s, 2s, 4s (capped at 10s)
        wait := time.Duration(1<<uint(attempt)) * time.Second
        if wait > 10*time.Second {
            wait = 10 * time.Second
        }
        select {
        case <-time.After(wait):
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }
    return nil, fmt.Errorf("getUserProfile failed after retries: %w", lastErr)
}

// getInventory adds a circuit breaker so a failing service
// doesn't cascade failures to the rest of the system.
func getInventory(ctx context.Context, productID int) (map[string]interface{}, error) {
    cb := newCircuitBreaker(5, 30*time.Second) // pseudo circuit breaker

    if !cb.allow() {
        return nil, fmt.Errorf("circuit is OPEN")
    }

    client := &http.Client{Timeout: 3 * time.Second}
    resp, err := client.Get(fmt.Sprintf("http://inventory-service/products/%d/stock", productID))
    if err != nil {
        cb.failure()
        return nil, err
    }
    defer resp.Body.Close()

    cb.success()
    var result map[string]interface{}
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return nil, err
    }
    return result, nil
}

When to use synchronous communication:
  ✓ Responses needed before continuing (data queries)
  ✓ Users waiting for direct responses (read operations)
  ✓ Operations requiring immediate acknowledgements

Synchronous communication risks:
  → Temporal coupling: if Service B is down, Service A also fails
  → Cascade failures: one service's failure spreads to others
  → Latency accumulation: latency = the sum of all called services' latencies
  → Mandatory: circuit breakers + timeouts + retries with exponential backoff

Asynchronous Communication (Message Queues) #

// Communication via message queues — loose coupling, resilient.

package main

import (
    "context"
    "encoding/json"
    "os"
)

// OrderCreatedEvent is published when an order is created.
type OrderCreatedEvent struct {
    EventType   string  `json:"event_type"`
    OrderID     string  `json:"order_id"`
    UserID      int     `json:"user_id"`
    TotalAmount float64 `json:"total_amount"`
    Items       []Item  `json:"items"`
}

// PublishOrderCreated publishes an event to a queue — the Order
// Service doesn't need to know who will consume this event.
func PublishOrderCreated(orderID string, userID int, total float64, items []Item) error {
    event := OrderCreatedEvent{
        EventType:   "order.created",
        OrderID:     orderID,
        UserID:      userID,
        TotalAmount: total,
        Items:       items,
    }

    payload, err := json.Marshal(event)
    if err != nil {
        return err
    }

    // aws-sdk-go-v2 style SQS call:
    _, err = sqsClient.SendMessage(context.Background(), &sqsv2.SendMessageInput{
        QueueUrl:    aws.String(os.Getenv("ORDER_EVENTS_QUEUE_URL")),
        MessageBody: aws.String(string(payload)),
    })
    return err
}

// HandleOrderCreated — the Inventory Service consumes the order.created
// event. If the Inventory Service is down, messages stay in the queue
// until processed.
func HandleOrderCreated(eventData map[string]interface{}) {
    orderID := eventData["order_id"].(string)
    items := eventData["items"].([]interface{})

    for _, item := range items {
        m := item.(map[string]interface{})
        reserveInventory(m["product_id"].(string), m["quantity"].(int))
    }

    logger.Info("Inventory reserved for order %s", orderID)
}

When to use asynchronous communication:
  ✓ Operations not needing immediate responses (writes/actions)
  ✓ Fanout: one event consumed by many services
  ✓ Bufferable workloads (emails, notifications, report generation)
  ✓ Resilience: consumers can be down and catch up later

Useful event patterns:
  Event Notification: "Something happened" (minimal data)
    → order.created { order_id: "123" }
    → Consumers fetch details themselves if needed

  Event-Carried State Transfer: events contain all data
    → order.created { order_id: "123", user_id: 42, items: [...] }
    → Consumers don't need to call back to other services

  Event Sourcing: all state changes as events
    → Rebuild state by replaying all events
    → More complex but very powerful for audit trails

API Gateways: The Single Entry Point #

An API Gateway is a component sitting in front of all microservices, providing a single entry point for clients.

API Gateway responsibilities:

  Routing:
  → GET /users/* → User Service
  → GET /orders/* → Order Service
  → POST /payments/* → Payment Service

  Cross-cutting concerns:
  → Authentication & Authorization (JWT validation before forwarding requests)
  → Rate limiting
  → SSL termination
  → Request/response logging
  → Correlation ID injection (for distributed tracing)

  Client-specific aggregation:
  → BFF (Backend for Frontend): one gateway per client type
    - Mobile BFFs: responses optimized for low bandwidth
    - Web BFFs: richer responses for desktop browsers

  Benefits:
  → Clients don't need to know how many services exist or where they are
  → Internal service changes don't affect exposed client APIs
  → A single place for cross-cutting concerns

# Kong API Gateway configuration (example)

services:
  - name: user-service
    url: http://user-service:8080

  - name: order-service
    url: http://order-service:8080

routes:
  - name: users-route
    service: user-service
    paths:
      - /api/users

  - name: orders-route
    service: order-service
    paths:
      - /api/orders

plugins:
  - name: jwt          # Authentication
    service: user-service
    config:
      key_claim_name: kid

  - name: rate-limiting
    config:
      minute: 100      # 100 requests per minute per consumer
      policy: local

  - name: correlation-id
    config:
      header_name: X-Correlation-ID
      generator: uuid

Service Discovery #

In microservices, service instances can run on various hosts and ports that change over time (especially in Kubernetes). Service discovery lets services find each other without hardcoded addresses.

Two service discovery approaches:

  Client-side discovery:
  → Clients query a Service Registry to find instances
  → Clients do their own load balancing
  → Examples: Netflix Eureka, Consul

  Server-side discovery:
  → Clients only know one endpoint (a load balancer)
  → The load balancer knows where instances are
  → Examples: AWS ALB, Kubernetes Services

  In Kubernetes: built-in service discovery
  → Every Service gets a DNS name
  → http://user-service.default.svc.cluster.local
  → Or simply: http://user-service (in the same namespace)
  → kube-proxy handles load balancing to healthy pods

# Kubernetes Service definition — automatic service discovery

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: default
spec:
  selector:
    app: user-service      # route to pods with this label
  ports:
    - protocol: TCP
      port: 80             # exposed port
      targetPort: 8080     # port inside the pod

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3              # 3 instances for HA
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
        - name: user-service
          image: myregistry/user-service:v1.2.3
          ports:
            - containerPort: 8080
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: user-service-secrets
                  key: database_url
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30

Distributed Tracing #

When one user request passes through 5 services, debugging becomes very difficult without distributed tracing. Distributed tracing connects all logs and spans from all services into one end-to-end trace.

// OpenTelemetry — the standard for distributed tracing.

package main

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

// setupTracing configures the tracer — done once at startup.
func setupTracing(serviceName string) error {
    exporter, err := otlptracegrpc.New(context.Background(),
        otlptracegrpc.WithEndpoint("jaeger:4317"))
    if err != nil {
        return err
    }

    provider := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )
    otel.SetTracerProvider(provider)
    return nil
}

// tracer for manual instrumentation.
var tracer = otel.Tracer("orders")

// processOrder records a span for each step of the flow.
// All child spans connect to the existing parent span.
func processOrder(ctx context.Context, orderData map[string]interface{}) error {
    ctx, span := tracer.Start(ctx, "process_order")
    defer span.End()
    span.SetAttributes(
        attribute.String("order.id", orderData["id"].(string)),
        attribute.Float64("order.total", orderData["total"].(float64)),
    )

    ctx, child := tracer.Start(ctx, "validate_payment")
    if err := validatePayment(ctx, orderData); err != nil {
        child.RecordError(err)
        child.End()
        return err
    }
    child.End()

    ctx, child = tracer.Start(ctx, "reserve_inventory")
    if err := reserveInventory(ctx, orderData["items"]); err != nil {
        child.RecordError(err)
        child.End()
        return err
    }
    child.End()

    _, child = tracer.Start(ctx, "notify_fulfillment")
    if err := publishOrderCreated(ctx, orderData); err != nil {
        child.RecordError(err)
        child.End()
        return err
    }
    child.End()

    return nil
}

Distributed tracing provides:

  Traces: the entire journey of one request end to end
  Spans: one operation within a trace (one service call, one DB query)
  Context propagation: trace IDs sent via HTTP headers to all services

  What can be seen:
  → How much time is spent in each service
  → Where bottlenecks occur
  → Which service errors happen in
  → How services call each other

  Tools:
  → Jaeger (open source)
  → Zipkin (open source)
  → AWS X-Ray
  → Datadog APM

Eventual Consistency: The Data Reality in Microservices #

Because each service owns its own database, no ACID transaction can span multiple services. Data between services can only achieve eventual consistency — consistent in the end, but with a window where states differ.

Managing eventual consistency:

  The problem:
  An order is created in the Order Service (status: "pending")
  The order.created event is sent to the Payment Service
  The Payment Service is down → not yet processed
  The user sees the order status: "pending" — but payment hasn't been processed
  → Data temporarily inconsistent, but eventually consistent

  Strategies:

  1. The Saga Pattern — long-running transactions with compensating actions
     Step 1: Order Service → create order (status: pending)
     Step 2: Payment Service → charge payment
     Step 3: Inventory Service → reserve items
     Step 4: Order Service → update status to confirmed

     If Step 3 fails:
     Compensate Step 2: refund the payment
     Compensate Step 1: cancel the order

  2. The Outbox Pattern — avoiding the dual-write problem
     Instead of writing to the DB and publishing events separately:
     → Write to the DB and the outbox table in one transaction
     → A background process publishes events from the outbox table
     → If publishing fails, the event stays in the outbox → retryable

// The Outbox Pattern for reliable event publishing.

package main

import (
    "database/sql"
    "encoding/json"
    "log"
    "time"
)

// OrderRepository creates orders and records the event
// in the same database transaction.
type OrderRepository struct {
    db *sql.DB
}

// CreateOrderWithEvent creates the order AND records the event
// in one database transaction — no possibility of the order being
// created but the event not published.
func (r *OrderRepository) CreateOrderWithEvent(orderData map[string]interface{}) (int64, error) {
    tx, err := r.db.Begin()
    if err != nil {
        return 0, err
    }
    defer tx.Rollback() // no-op after Commit

    // 1. Create the order
    result, err := tx.Exec(
        `INSERT INTO orders (user_id, total) VALUES (?, ?)`,
        orderData["user_id"], orderData["total"])
    if err != nil {
        return 0, err
    }
    orderID, err := result.LastInsertId() // generate order.id
    if err != nil {
        return 0, err
    }

    // 2. Save the event to the outbox table in the same transaction
    payload, err := json.Marshal(map[string]interface{}{
        "order_id": orderID,
        "user_id":  orderData["user_id"],
        "total":    orderData["total"],
    })
    if err != nil {
        return 0, err
    }
    if _, err := tx.Exec(
        `INSERT INTO outbox (event_type, aggregate_id, payload, published)
         VALUES ('order.created', ?, ?, false)`,
        orderID, payload); err != nil {
        return 0, err
    }

    if err := tx.Commit(); err != nil {
        return 0, err
    }
    return orderID, nil
}

// OutboxPublisher runs as a separate background process,
// publishing events from the outbox table.
func (r *OrderRepository) OutboxPublisher() {
    for {
        rows, err := r.db.Query(
            `SELECT id, event_type, payload FROM outbox
             WHERE published = false ORDER BY created_at LIMIT 100`)
        if err == nil {
            for rows.Next() {
                var (
                    id        int64
                    eventType string
                    payload   string
                )
                if err := rows.Scan(&id, &eventType, &payload); err != nil {
                    continue
                }
                if err := publishToQueue(eventType, payload); err == nil {
                    r.db.Exec(
                        `UPDATE outbox SET published = true, published_at = ? WHERE id = ?`,
                        time.Now(), id)
                } else {
                    log.Printf("Failed to publish event %d: %v", id, err)
                    // published stays false → retried in the next iteration
                }
            }
            rows.Close()
        }

        time.Sleep(5 * time.Second) // check every 5 seconds
    }
}

When Microservices, When Monoliths #

A decision framework:

  Questions to answer:

  1. How large is the team?
     < 10 engineers: monoliths are almost always better
     10-30 engineers: modular monoliths, evaluate needs
     30+ engineers: microservices start making sense

  2. Are there scaling requirements differing per domain?
     All domains grow proportionally → monoliths are fine
     Search needs 10x more scale than checkout → microservices make sense

  3. How mature is the business domain?
     Just starting, lots of uncertainty → monoliths (easy to refactor)
     Stable, well-understood domains → microservices are safer

  4. Is the team experienced with distributed systems?
     No → start with a monolith, learn distributed systems first
     Yes → microservices can be considered

  5. Is there an independent deployment need?
     All features deployed together → monoliths are fine
     Different teams need to release anytime → microservices

  The common answer: start with a modular monolith.
  Evolve to microservices based on real needs, not anticipation.

Anti-Patterns to Avoid #

✗ Anti-pattern 1: Distributed Monoliths
  Tightly coupled interdependent services
  Changing Service A always requires changing Services B and C
  Deployments must happen together
  → All the bad of monoliths + all the bad of distributed systems
  ✓ Solution: fix boundaries, reduce inter-service coupling

✗ Anti-pattern 2: Too many services too early
  "Nano-services": every function becomes its own service
  Every request needs 15 service calls
  Network overhead dominates latency
  ✓ Solution: start with larger services, split only when clear reasons exist

✗ Anti-pattern 3: Shared databases between services
  The User Service and Order Service use the same database
  No independence — schema changes in one place affect the other
  ✓ Solution: database per service, communication via APIs or events

✗ Anti-pattern 4: Long synchronous chains
  Request → A → B → C → D → E → response
  Accumulative latency + any failure fails everything
  ✓ Solution: redesign to async events, or shorten chains by merging services

✗ Anti-pattern 5: No distributed tracing
  Bugs in production, no idea which services requests pass through or where they fail
  Per-service logs aren't enough for distributed bug investigations
  ✓ Solution: OpenTelemetry + Jaeger/Zipkin from the start

✗ Anti-pattern 6: Ignoring eventual consistency
  Assuming data is always consistent like in monoliths
  Strange bugs because states are out of sync between services
  ✓ Solution: design for eventual consistency, use the Outbox Pattern

Microservices Checklist #

ARCHITECTURE:
  □ Each service has a clear, cohesive business domain
  □ Services deployable independently without coordinating with other services
  □ Each service owns its own database (no sharing)
  □ Service boundaries validated with event storming or domain analysis

COMMUNICATION:
  □ Synchronous (REST/gRPC) only for needs requiring direct responses
  □ Asynchronous (message queues) for operations not needing direct responses
  □ Circuit breakers on all synchronous calls to other services
  □ Timeouts configured for all external calls
  □ Retries with exponential backoff for transient failures

RELIABILITY:
  □ Health check endpoints in every service (/health or /ready)
  □ Graceful shutdowns — finishing in-flight requests
  □ Dead letter queues for failed messages
  □ Outbox patterns for reliable event publishing

OBSERVABILITY:
  □ Distributed tracing with OpenTelemetry
  □ Correlation IDs propagated to all services
  □ Structured logging with trace IDs in every log entry
  □ Per-service metrics (request rates, error rates, latencies)
  □ Centralized log aggregation (ELK, Loki)

API GATEWAYS:
  □ Single entry points for all clients
  □ Authentication/Authorization at the gateway layer
  □ Rate limiting at gateways
  □ Clear API versioning strategies

DEPLOYMENT:
  □ Each service has its own CI/CD pipeline
  □ Immutable container images (tagged with git SHAs)
  □ Blue-green or canary deployments for zero downtime
  □ Fast, tested rollbacks

DATA CONSISTENCY:
  □ Eventual consistency understood and well-designed
  □ Saga patterns for long-running transactions
  □ Idempotent consumers for message processing

Summary #

• Start with monoliths, evolve to microservices — don’t start new projects with microservices. Modular monoliths are easier to develop, test, and debug, and easier to refactor into microservices when real needs emerge.
• Correct boundaries are the foundation of everything — wrong boundaries create distributed monoliths worse than either option. Use DDD Bounded Contexts as guidance.
• Database per service is mandatory — database sharing removes the independence that’s the main reason for microservices. Without it, all services stay tightly coupled.
• Choose communication per needs — synchronous for queries needing direct responses, asynchronous for bufferable actions. Don’t default to synchronous for everything.
• Circuit breakers prevent cascade failures — without circuit breakers, one service’s failure spreads across the entire system. This is the difference between partial and total outages.
• Distributed tracing is a need, not an option — debugging microservices without tracing is like debugging blind. Install OpenTelemetry from day one.
• Eventual consistency is a reality, not a bug — with database-per-service, there’s no strong consistency between services. Design systems to work with this, not against it.
• The Outbox Pattern prevents the dual-write problem — writing to the database and publishing events must happen in one atomic operation. The outbox pattern ensures events aren’t lost on failures.
• Operational complexity is much higher — more services mean more things that can fail, more to monitor, more deployments to coordinate. Make sure the team is ready.
• Microservices aren’t about technology — they’re about organization — Conway’s Law: system architecture mirrors organizational communication structures. Microservices work best when each service is owned by a dedicated team.

#

← Previous: Serverless   Next: Micro Frontend