Cheat SheetsSystem DesignObservability & Operations

Observability & Operations — Cheat Sheet

System Design · 5 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org
Observability & Operations
System Design5 topicsQuick revision reference
1

Logging Best Practices

Structured, centralised logging with correlation IDs is essential for debugging distributed systems. Log at appropriate levels, include context, and aggregate logs in a searchable platform (ELK, Loki).

  • ✓Structured JSON logs with consistent fields are searchable and machine-parseable.
  • ✓Correlation IDs (traceId) propagated across services link all logs for a single request.
  • ✓Centralise logs in ELK or Grafana Loki — never rely on SSH-ing into individual servers.
  • ✓Use appropriate log levels: ERROR for failures, INFO for business events, DEBUG off in production.
  • ✓Never log passwords, tokens, credit card numbers, or unmasked PII.
JSON + XML + Java — structured logging
// Unstructured log (bad — hard to parse and search)
2025-03-29 10:00:01 INFO OrderService - Order 123 placed by user 42

// Structured log (good — JSON, searchable)
{
  "timestamp": "2025-03-29T10:00:01.234Z",
  "level": "INFO",
  "service": "order-service",
  "traceId": "abc-123-def-456",
  "spanId": "span-789",
  "userId": "u-42",
  "orderId": "order-123",
  "message": "Order placed successfully",
  "total": 99.99,
  "itemCount": 3,
  "durationMs": 45
}

// Spring Boot + Logback JSON encoder
// logback-spring.xml
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>traceId</includeMdcKeyName>
    <includeMdcKeyName>userId</includeMdcKeyName>
  </encoder>
</appender>

// Using MDC for context propagation
MDC.put("userId", currentUser.getId());
MDC.put("orderId", order.getId());
log.info("Order placed successfully");
2

Monitoring & Alerting

Monitoring collects metrics (CPU, latency, error rates, business KPIs) and alerting notifies teams when thresholds are breached. Prometheus + Grafana is the standard open-source stack.

  • ✓RED (Rate, Errors, Duration) for service health; USE (Utilisation, Saturation, Errors) for infrastructure.
  • ✓Prometheus scrapes metrics; Grafana visualises; Alertmanager routes alerts.
  • ✓Track latency percentiles (p50, p95, p99) — averages hide tail latency problems.
  • ✓Good alerts are actionable, have runbooks, use severity levels, and avoid alert fatigue.
  • ✓Business metrics (orders/min, revenue/hour) are as important as infrastructure metrics.
Conceptual + Java + YAML — RED/USE methods with Micrometer
// RED method — for every service
// Rate:     requests per second
// Errors:   error rate (5xx / total requests)
// Duration: latency percentiles (p50, p95, p99)

// USE method — for every resource (CPU, memory, disk, network)
// Utilisation: % of resource in use (CPU at 70%)
// Saturation:  work queued (thread pool queue depth)
// Errors:      error count (disk I/O errors)

// Spring Boot + Micrometer metrics (auto-exposed for Prometheus)
// application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,info
  metrics:
    tags:
      application: order-service
    distribution:
      percentiles-histogram:
        http.server.requests: true  # exposes p50, p95, p99

// Custom business metric
@Component
public class OrderMetrics {
    private final Counter ordersPlaced;
    private final Timer orderProcessingTime;

    public OrderMetrics(MeterRegistry registry) {
        this.ordersPlaced = Counter.builder("orders.placed")
            .tag("payment_method", "card")
            .register(registry);
        this.orderProcessingTime = Timer.builder("orders.processing.time")
            .register(registry);
    }
}
3

Distributed Tracing

Distributed tracing tracks a request as it flows through multiple microservices, creating a visual trace of the entire call chain with timing data. It enables pinpointing latency bottlenecks and failure points.

  • ✓A trace tracks a request across multiple services; each service creates spans (timed operations).
  • ✓Context propagation (traceId + spanId) via HTTP headers links spans into a complete trace.
  • ✓OpenTelemetry (OTel) is the industry standard for instrumentation — vendor-neutral.
  • ✓Sample 1-10% of requests in production to balance cost and visibility; always sample errors.
  • ✓Tracing pinpoints latency bottlenecks: "the bank API call in payment service takes 120ms."
Conceptual — trace structure and context propagation
// Trace structure
//
// Trace ID: abc-123 (entire request)
//
// ┌─ Span 1: API Gateway (10ms) ─────────────────────────────────┐
// │  ┌─ Span 2: Order Service (200ms) ──────────────────────────┐│
// │  │  ┌─ Span 3: Payment Service (150ms) ───────────────────┐ ││
// │  │  │  ┌─ Span 4: Bank API call (120ms) ────────────────┐ │ ││
// │  │  │  └─────────────────────────────────────────────────┘ │ ││
// │  │  └─────────────────────────────────────────────────────┘ ││
// │  │  ┌─ Span 5: Inventory Service (90ms) ──────────────────┐ ││
// │  │  └─────────────────────────────────────────────────────┘ ││
// │  └──────────────────────────────────────────────────────────┘│
// └──────────────────────────────────────────────────────────────┘
// Total: 450ms — bottleneck is Bank API call (120ms)

// Context propagation via HTTP headers (W3C Trace Context)
// traceparent: 00-abc123def456-span789-01
// tracestate: vendor=value

// Spring Boot auto-propagation (Micrometer Tracing)
// Trace context automatically added to:
// - Outgoing HTTP requests (RestTemplate, WebClient, Feign)
// - Kafka messages (trace headers in Kafka record headers)
// - JDBC queries (as spans)
4

Blue-Green & Canary Deployments

Blue-green deployment maintains two identical environments and switches traffic instantly. Canary deployment gradually routes a small percentage of traffic to the new version. Both enable zero-downtime releases with easy rollback.

  • ✓Blue-green: two identical envs, instant switch, instant rollback — simple but doubles infrastructure.
  • ✓Canary: gradual traffic shift (1% → 10% → 50% → 100%) — less risky, needs good monitoring.
  • ✓Both require automated monitoring and rollback based on error rate, latency, and business metrics.
  • ✓Kubernetes + Istio + Argo Rollouts enable automated canary with traffic splitting and analysis.
  • ✓Always have a rollback plan — "what happens if the new version breaks?"
Conceptual + Terraform — blue-green switch
// Blue-Green deployment flow
//
// Step 1: Blue (v1) is live, Green is idle
//   Load Balancer → Blue (v1) [100% traffic]
//                   Green (idle)
//
// Step 2: Deploy v2 to Green, run smoke tests
//   Load Balancer → Blue (v1) [100% traffic]
//                   Green (v2) [smoke tests pass ✅]
//
// Step 3: Switch traffic to Green
//   Load Balancer → Green (v2) [100% traffic]
//                   Blue (v1) [idle, ready for rollback]
//
// Step 4: If problems → instant rollback to Blue
//   Load Balancer → Blue (v1) [100% traffic]

// AWS: Route 53 weighted routing for blue-green
resource "aws_route53_record" "api" {
  zone_id        = var.zone_id
  name           = "api.example.com"
  type           = "A"
  set_identifier = "green"
  weighted_routing_policy { weight = 100 }  # all traffic to green
  alias { name = aws_lb.green.dns_name }
}
5

Chaos Engineering

Chaos engineering intentionally injects failures (kill pods, introduce latency, partition networks) into production-like systems to verify that the system handles failures gracefully and build confidence in resilience.

  • ✓Chaos engineering intentionally injects failures to verify system resilience before real failures occur.
  • ✓Process: define steady state → hypothesise → inject failure → observe → learn and fix.
  • ✓Start small (staging, non-critical) and gradually move to production with blast radius controls.
  • ✓Common experiments: pod kill, network latency, DNS failure, disk pressure, zone failure.
  • ✓Tools: Chaos Monkey (Netflix), Litmus Chaos (K8s), Gremlin (commercial), AWS FIS.
Conceptual — chaos engineering lifecycle
// Chaos engineering process
//
// 1. Define steady state
//    "Order API returns 200 in < 200ms for 99.9% of requests"
//
// 2. Hypothesise
//    "If we kill 2 of 5 order-service pods, Kubernetes reschedules
//     them within 30s and latency stays below 200ms because the
//     remaining 3 pods absorb the traffic"
//
// 3. Run experiment
//    Kill 2 pods at 10:00 AM
//    Monitor: request rate, error rate, p99 latency
//
// 4. Observe results
//    ✅ Pods rescheduled in 15s
//    ⚠️ Latency spiked to 350ms for 20 seconds (unexpected)
//    ✅ No 5xx errors
//
// 5. Learn and improve
//    Finding: HPA takes 30s to scale up → pre-warm extra pods
//    Action: Set minReplicas from 3 to 5

// Blast radius controls:
// Start in staging → move to production
// Start with non-critical services → then critical
// Set maximum impact: "kill at most 1 pod per service"
// Have an abort switch: stop experiment immediately if SLO breached
Learn this free with Aria, your AI tutor → AiCanCode.org/learn/system-design