System Integration #

Almost no modern system stands alone. Backend services call payment gateways, notification services listen to events from order services, mobile apps consume REST APIs, schedulers pull data from legacy ERPs. Every one of these connection points is an integration — and every integration not designed correctly is a weak point that can become a source of data inconsistency, security breaches, or cascading failures. The integration challenge isn’t just technical (“the API can be called”) but design (“what if the payload changes?”, “what if the downstream goes down?”, “what if the same request arrives twice?”). This article covers system integration from its five main types, three communication patterns with their trade-offs, critical best practices from authentication to observability, concrete webhook and HMAC verification implementations, and a complete end-to-end case study.

Five Types of System Integration #

flowchart TD
    SI[System Integration] --> P2P["Point-to-Point\nDirect connection"]
    SI --> API["API-Based\nREST/gRPC/GraphQL contract"]
    SI --> ED["Event-Driven\nMessage broker"]
    SI --> FB["File-Based\nCSV/XML/Parquet batches"]
    SI --> DB["Database-Level\n❌ Anti-pattern"]

1. Point-to-Point Integration #

Each system connects directly to other systems without a mediator.

flowchart LR
    A[System A] --> B[System B]
    A --> C[System C]
    D[System D] --> B

It feels simple at first, but as systems grow, the number of connections grows quadratically: N systems produce up to N(N-1)/2 connections.

Every connection is a dependency that must be managed, monitored, and tested. When it’s still acceptable: prototypes, temporary integration between two small systems, or when mediator complexity isn’t worth it.

2. API-Based Integration #

Systems communicate through defined API contracts — REST, gRPC, or GraphQL.

flowchart LR
    M[Mobile App] -- "REST" --> B[Backend API]
    B -- "gRPC" --> I[Internal Service]
    B -- "REST" --> P[Payment Gateway]

This is the most common and most flexible pattern. The key often missed: an API isn’t just an endpoint, it’s a contract. Request/response schemas that change without versioning are a source of breaking changes that break all consumers at once.

3. Event-Driven Integration #

Systems communicate through asynchronous events via a message broker — Kafka, RabbitMQ, AWS SNS/SQS, or Google Pub/Sub.

flowchart TD
    OS[Order Service] -- "publish" --> K["(Kafka: order-events)"]
    K --> PS["Payment Service\nsubscribe"]
    K --> IS["Inventory Service\nsubscribe"]
    K --> AS["Analytics Service\nsubscribe"]

Advantages: true loose coupling — the Order Service doesn’t know who’s listening to its events. Adding a new consumer requires no changes to the producer. The downsides: more complex debugging, and the system operates with eventual consistency.

4. File-Based Integration #

Data exchange through files (CSV, XML, JSON, Parquet) placed in an agreed location.

flowchart LR
    LE[Legacy ERP] -- "export batch\nfile.csv" --> SF[("(Shared Storage\nS3/SFTP)")]
    SF -- "scheduled\nimport job" --> NS[New System]

Most common in integrations with legacy systems (ERP, mainframe) that don’t support modern APIs, or for high-volume batch jobs. The risks: high latency (new data only available after the batch finishes), difficult error handling (a corrupted file mid-process), and no real-time confirmation.

5. Database-Level Integration — The Anti-Pattern #

One system accesses another system’s database directly.

// ANTI-PATTERN: Payment Service directly queries the Order Service's DB
func (p *PaymentService) getOrderTotal(orderID string) float64 {
    row := orderDB.QueryRow(  // ← accessing another system's database!
        "SELECT total FROM orders WHERE id = ?", orderID)
    // ...
}

flowchart LR
    subgraph Bad["❌ Database-Level Integration"]
        PS1[Payment Service] -->|"direct SQL query"| ODB[("(Order DB\ninternal schema)")]
        OS1[Order Service] --> ODB
    end
    subgraph Good["✅ API/Event as the Interface"]
        PS2[Payment Service] -->|"GET /orders/:id"| OAPI[Order Service API]
        OAPI --> ODB2[("(Order DB\nencapsulated)")]
    end

This completely violates encapsulation: the database schema becomes a public API that can’t be changed without breaking consumers. The Payment Service now depends on the Order Service’s internal implementation details. There’s no way to evolve the database schema without coordinating with every party accessing it. Always use APIs or events as the interface.

Three Communication Patterns #

Synchronous Request-Response #

The client waits for the response before continuing. Suitable when the result is needed immediately.

sequenceDiagram
    participant C as Client
    participant V as Payment Validator
    participant G as Payment Gateway

    C->>V: POST /validate
    V-->>C: {valid: true}
    C->>G: POST /charge
    G-->>C: {txn_id: "abc"}

When to use sync: data validation that determines whether the flow can continue, queries whose results are shown directly to users, operations with strict latency SLAs.

The risk: cascading failures — if the downstream is slow or down, the caller is also slowed or fails. Must always be paired with timeouts and circuit breakers.

Asynchronous Messaging #

The producer sends a message to the broker and finishes immediately. Consumers process it at a different time.

sequenceDiagram
    participant O as Order Service
    participant Q as Queue: send-confirmation-email
    participant E as Email Service

    O->>Q: publish
    O-->>O: return 202 (done immediately)
    Q->>E: consume (background processing)

When to use async: operations that don’t need an immediate result (sending emails, generating reports, syncing to CRM), long-running processes, operations where eventual consistency is acceptable.

Orchestration vs Choreography #

Two different approaches to coordinating many services in a workflow.

Orchestration — a central coordinator directs the workflow:

flowchart TD
    OC[Order Orchestrator] -->|"1. Validate inventory"| INV[Inventory Service]
    OC -->|"2. Process payment"| PAY[Payment Service]
    OC -->|"3. Create shipment"| SHIP[Shipping Service]
    OC -->|"4. Send notification"| NOTIF[Notification Service]

Choreography — each service reacts to events independently:

flowchart TD
    OS[Order Service] -->|publish| EV[OrderCreated]
    EV --> PS["Payment Service\nprocesses, publishes PaymentDone"]
    EV --> IS["Inventory Service\nreduces stock, publishes StockReduced"]
    EV --> SS["Shipping Service\ncreates label after PaymentDone"]

Critical Best Practices #

Authentication — Don’t Trust Requests from Other Systems #

Every integration must be authenticated, including internal service-to-service calls. “Internal network” isn’t a security guarantee.

// Machine-to-machine auth with OAuth 2.0 Client Credentials
type ServiceClient struct {
    tokenURL     string
    clientID     string
    clientSecret string
    httpClient   *http.Client
    tokenCache   *TokenCache
}

func (c *ServiceClient) GetAccessToken(ctx context.Context) (string, error) {
    // Check the cache first — the token is valid 5 minutes before expiry
    if token, ok := c.tokenCache.Get(); ok {
        return token, nil
    }

    resp, err := c.httpClient.PostForm(c.tokenURL, url.Values{
        "grant_type":    {"client_credentials"},
        "client_id":     {c.clientID},
        "client_secret": {c.clientSecret},
        "scope":         {"internal:read internal:write"},
    })
    // parse the token and cache with TTL = expires_in - 5 minutes buffer
    token := parseToken(resp)
    c.tokenCache.Set(token, token.ExpiresIn-5*time.Minute)
    return token.AccessToken, nil
}

Authentication options by context:

HMAC Webhook Signature Verification #

Webhooks from payment gateways, GitHub, Stripe, and similar services usually send an HMAC signature to prove the request came from them and the payload wasn’t modified.

sequenceDiagram
    participant PG as Payment Gateway
    participant W as Webhook Handler

    PG->>W: POST /webhook\nX-Signature: sha256=...\nX-Timestamp: ...
    W->>W: 1. Read the raw body
    W->>W: 2. Compute HMAC from the raw body
    W->>W: 3. Compare with X-Signature\n(constant-time)
    W->>W: 4. Check timestamp freshness\n(< 5 minutes)
    alt Valid
        W-->>PG: 200 OK (immediately)
        W->>W: Process async in the background
    else Invalid
        W-->>PG: 401 Unauthorized
    end

// Webhook handler with HMAC signature verification
func (h *PaymentWebhookHandler) HandleWebhook(w http.ResponseWriter, r *http.Request) {
    // Read the raw body BEFORE parsing — the signature is computed from raw bytes
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Failed to read body", http.StatusBadRequest)
        return
    }

    // Verify the signature
    if !h.verifySignature(body, r.Header.Get("X-Webhook-Signature")) {
        log.Warnf("invalid webhook signature from %s", r.RemoteAddr)
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }

    // Check the timestamp to prevent replay attacks
    timestamp := r.Header.Get("X-Webhook-Timestamp")
    if !h.isTimestampFresh(timestamp, 5*time.Minute) {
        log.Warnf("stale webhook timestamp: %s", timestamp)
        http.Error(w, "Request expired", http.StatusUnauthorized)
        return
    }

    // Return 200 IMMEDIATELY — process async in the background
    // Webhook providers usually time out in 5-30 seconds
    w.WriteHeader(http.StatusOK)

    // Process in a separate goroutine to avoid timeouts
    go h.processWebhookAsync(body)
}

func (h *PaymentWebhookHandler) verifySignature(body []byte, signature string) bool {
    mac := hmac.New(sha256.New, []byte(h.webhookSecret))
    mac.Write(body)
    expected := "sha256=" + hex.EncodeToString(mac.Sum(nil))

    // Use hmac.Equal for constant-time comparison
    // — prevents timing attacks
    return hmac.Equal([]byte(signature), []byte(expected))
}

Always use hmac.Equal (constant-time comparison), not == or plain bytes.Equal, to compare signatures. A normal comparison can leak information through timing attacks — an attacker can guess the signature byte by byte by measuring response times.

Data Contracts and Versioning #

Schemas are contracts. Non-backward-compatible changes break all consumers without notice.

// CORRECT: versioned API response — add fields, don't remove
type OrderResponseV1 struct {
    OrderID   string    `json:"order_id"`
    Status    string    `json:"status"`
    Total     float64   `json:"total"`
    CreatedAt time.Time `json:"created_at"`
}

// V2: new fields added as optional — V1 consumers aren't broken
type OrderResponseV2 struct {
    OrderID        string    `json:"order_id"`
    Status         string    `json:"status"`
    Total          float64   `json:"total"`
    CreatedAt      time.Time `json:"created_at"`
    // New fields — omitempty so it's not breaking for old consumers
    TrackingNumber *string   `json:"tracking_number,omitempty"`
    PromotionCode  *string   `json:"promotion_code,omitempty"`
}

API versioning strategies:

Idempotency Keys for Critical Operations #

Every operation that isn’t naturally idempotent (create, charge) must be made idempotent with an idempotency key.

// The client includes an idempotency key in every critical request
func (c *PaymentClient) CreateCharge(ctx context.Context, req ChargeRequest) (*Charge, error) {
    // Generate the idempotency key from intent — not random
    // The same key for the same request ensures deduplication works
    idempotencyKey := fmt.Sprintf("charge:%s:%s:%d",
        req.OrderID,
        req.UserID,
        req.AmountCents,
    )

    httpReq, _ := http.NewRequestWithContext(ctx, "POST",
        c.baseURL+"/charges",
        encodeJSON(req),
    )
    httpReq.Header.Set("Idempotency-Key", idempotencyKey)
    httpReq.Header.Set("X-Correlation-ID", getCorrelationID(ctx))

    // The server will deduplicate: if the key already exists, return the old result
    return c.do(httpReq)
}

Observability — Correlation IDs Across the Whole System #

Without distributed tracing, debugging a request passing through 5 services is a nightmare.

// Middleware that propagates the correlation ID to all outbound requests
type CorrelationMiddleware struct {
    next http.RoundTripper
}

func (m *CorrelationMiddleware) RoundTrip(req *http.Request) (*http.Response, error) {
    // Take it from the context, or generate if absent
    correlationID := getOrGenerateCorrelationID(req.Context())

    // Propagate to the downstream
    req.Header.Set("X-Correlation-ID", correlationID)
    req.Header.Set("X-Request-ID", uuid.New().String())

    return m.next.RoundTrip(req)
}

// All logs must include the correlation ID
func logRequest(ctx context.Context, msg string, fields ...interface{}) {
    cid := ctx.Value("correlation_id")
    log.WithField("correlation_id", cid).Info(append([]interface{}{msg}, fields...)...)
}

Case Study: Order → Payment → Notification #

Here’s an end-to-end flow applying all the best practices discussed above.

sequenceDiagram
    participant U as User
    participant OS as Order Service
    participant K as Kafka
    participant PS as Payment Service
    participant IS as Inventory Service
    participant PG as Payment Gateway
    participant NS as Notification Service

    U->>OS: POST /checkout
    OS->>OS: 1. Create order (PENDING)
    OS->>K: 2. Publish OrderCreated
    OS-->>U: 3. {order_id, status: "pending"}

    K->>PS: consume OrderCreated
    K->>IS: consume OrderCreated
    IS->>IS: reserve stock

    PS->>PG: 4. Call payment gateway\n(with idempotency key)
    PG->>PS: webhook callback
    PS->>PS: 5. Verify HMAC signature
    PS->>PS: 6. Check idempotency
    PS->>PS: 7. Update order to PAID
    PS->>K: 8. Publish PaymentCompleted

    K->>NS: consume PaymentCompleted
    NS->>NS: send email + push notification

// Payment Service — handler for the gateway webhook
func (s *PaymentService) HandleGatewayWebhook(ctx context.Context,
    payload WebhookPayload, signature string) error {

    // Verify the signature
    if !verifyHMAC(payload, signature, s.webhookSecret) {
        return ErrInvalidSignature
    }

    // Idempotency check — webhooks can be redelivered
    if processed, _ := s.processedWebhooks.Exists(ctx, payload.WebhookID); processed {
        log.Infof("webhook %s already processed, skipping", payload.WebhookID)
        return nil
    }

    // Process in a single transaction
    return s.db.Transaction(func(tx *gorm.DB) error {
        // Update the order status
        if err := s.orderRepo.UpdateStatus(ctx, tx,
            payload.OrderID, "PAID"); err != nil {
            return err
        }

        // Publish the event
        if err := s.eventBus.Publish(ctx, PaymentCompletedEvent{
            OrderID:       payload.OrderID,
            TransactionID: payload.TransactionID,
            Amount:        payload.Amount,
            PaidAt:        time.Now(),
            CorrelationID: getCorrelationID(ctx),
        }); err != nil {
            return err
        }

        // Mark the webhook as processed
        return s.processedWebhooks.Set(ctx, tx, payload.WebhookID)
    })
}

Anti-Patterns to Avoid #

// ✗ Trusting internal requests without authentication
func handleInternalRequest(r *http.Request) {
    // "It's from internal, must be safe"
    processRequest(r) // no auth check
}
// ✓ Verify JWT or mTLS even for internal calls

// ✗ No timeout on outbound calls
resp, err := http.Get("https://external-api.com/data")
// If the external API hangs, this goroutine hangs forever
// ✓ Always set a timeout
client := &http.Client{Timeout: 10 * time.Second}

// ✗ Swallowing downstream errors without logging
result, err := externalService.Call(req)
if err != nil {
    return nil // error disappears without a trace
}
// ✓ Log with context, propagate or wrap with enough information

// ✗ Hardcoded credentials
const apiKey = "«redacted:sk_live_…»" // ← in Git, exposed to every developer
// ✓ Read from a secret manager or secure environment variable

// ✗ Webhooks processed synchronously without a queue
func handleWebhook(w http.ResponseWriter, r *http.Request) {
    processHeavyOperation() // could time out before returning 200
    w.WriteHeader(200)
}
// ✓ Return 200 immediately, process async in the background

Production-Ready Integration Checklist #

SECURITY:
  □ All integrations authenticated (no "trusted by default")
  □ Tokens/secrets stored in a secret manager (not hardcoded)
  □ Short-lived tokens with rotation
  □ Webhook signatures verified with HMAC
  □ Timestamp freshness checks to prevent replay attacks
  □ TLS for all communication

DATA CONTRACT:
  □ API schemas documented (OpenAPI/Protobuf)
  □ Versioning strategy defined
  □ Backward-compatible changes (add fields, don't remove)
  □ Breaking changes handled with new major versions

RELIABILITY:
  □ Timeouts configured for all outbound calls
  □ Retries with exponential backoff + jitter
  □ Circuit breakers for frequently flaky downstreams
  □ Idempotency keys for operations that aren't naturally idempotent
  □ DLQ for events that fail to process

OBSERVABILITY:
  □ Correlation IDs generated at the entry point and propagated to all downstreams
  □ Every request/response logged with the correlation ID
  □ Metrics: error rate, latency, throughput per integration point
  □ Alerts for error rates exceeding thresholds

Summary #

• System integration connects systems so they can exchange data and collaborate — not just “APIs calling each other”, but involving contracts, security, and long-term reliability.
• Five types: point-to-point (simple but not scalable), API-based (standard and flexible), event-driven (loose coupling but eventual consistency), file-based (for legacy/batch), and database-level (anti-pattern — avoid).
• Orchestration provides clear flow visibility with a central coordinator; choreography provides more independence but flows are harder to trace — choose based on your needs.
• Authentication is mandatory even for internal services — use OAuth 2.0 Client Credentials or mTLS, not “trusted by default”.
• HMAC signature verification for webhooks — including constant-time comparison and freshness checks to prevent timing and replay attacks.
• Idempotency keys are mandatory for operations that aren’t naturally idempotent — payments, order creation, actions with financial effects.
• Return 200 immediately from webhook endpoints, process async in the background — webhook providers usually time out in 5–30 seconds.
• APIs are contracts — schema changes must be backward-compatible; breaking changes need new major versions.
• Correlation IDs must be generated at the entry point and propagated to all downstreams — without this, debugging requests spanning many services is very hard.

← Previous: Jitter   Next: SOLID →