Cheat SheetsMicroservicesResilience

Resilience — Cheat Sheet

Microservices · 5 topics. Download the PDF or the Instagram carousel and share it.

Cheat Sheet · AiCanCode.org
Resilience
Microservices5 topicsQuick revision reference
1

Circuit Breaker Pattern

When failure rate exceeds a threshold the circuit "opens", short-circuiting calls and returning a fallback; after a wait it goes half-open to probe recovery.

  • ✓Circuit breaker has three states: CLOSED (normal), OPEN (fast-fail), HALF-OPEN (recovery probe).
  • ✓The circuit opens when the failure rate in a sliding window (count or time-based) exceeds the configured threshold.
  • ✓Always provide a fallback method — a cached response, a default value, or a graceful error — so the caller never waits for a timeout.
  • ✓waitDurationInOpenState prevents hammering a recovering service; set it to at least the expected recovery time of the downstream service.
  • ✓Circuit breakers prevent cascading failures: an open circuit stops thread exhaustion in the upstream service.
  • ✓Combine with Retry (for transient failures) and Bulkhead (for resource isolation) to build a comprehensive resilience strategy.
application.yml
// Resilience4j Circuit Breaker — configuration
resilience4j:
  circuitbreaker:
    instances:
      paymentService:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 10            # evaluate last 10 calls
        failureRateThreshold: 50         # open if >=50% fail
        waitDurationInOpenState: 30s     # stay open for 30 seconds
        permittedNumberOfCallsInHalfOpenState: 3
        recordExceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
2

Resilience4j Circuit Breaker

Resilience4j is a lightweight fault-tolerance library providing CircuitBreaker, RateLimiter, Retry, Bulkhead, and TimeLimiter decorators for functional or reactive code.

  • ✓CircuitBreaker has three states: CLOSED (normal), OPEN (blocked), HALF_OPEN (probing)
  • ✓failureRateThreshold triggers OPEN; waitDurationInOpenState controls recovery pause
  • ✓Fallback method must have same signature as the decorated method plus a Throwable parameter
  • ✓Decorator order: Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead
  • ✓@Retry uses exponential backoff + jitter to avoid thundering herd on recovery
  • ✓Expose circuit breaker state via Spring Actuator /actuator/circuitbreakers endpoint
Spring Boot — resilience4j CircuitBreaker config
# application.yml
resilience4j:
  circuitbreaker:
    instances:
      payment-service:
        slidingWindowType: COUNT_BASED     # or TIME_BASED
        slidingWindowSize: 10              # last 10 calls
        failureRateThreshold: 50           # open if ≥50% fail
        slowCallRateThreshold: 80          # open if ≥80% are slow
        slowCallDurationThreshold: 2000ms
        waitDurationInOpenState: 10s       # stay open 10s before HALF_OPEN
        permittedNumberOfCallsInHalfOpenState: 3
        minimumNumberOfCalls: 5            # need at least 5 calls before evaluating

# Spring Boot Actuator exposes circuit breaker state:
# GET /actuator/circuitbreakers
# GET /actuator/circuitbreakerevents
3

Retry Pattern

Transient failures are retried with exponential back-off and jitter; combine with idempotency on the server side to avoid duplicate effects.

  • ✓Only retry idempotent operations — retrying POST /payments without an idempotency key causes double-charging.
  • ✓Exponential backoff + jitter prevents thundering-herd: all clients retrying at exactly the same interval.
  • ✓Never retry 4xx client errors — they indicate a problem with your request, not a transient server issue.
  • ✓Combine @Retry with @CircuitBreaker: the circuit opens after exhausted retries, preventing retry storms.
  • ✓Set max-attempts conservatively (2–3) — aggressive retries amplify load on already-degraded services.
  • ✓Monitor retry rate as a metric — a sudden spike indicates upstream instability, not normal operation.
Properties + Java — Resilience4j retry with exponential jitter
# application.properties
resilience4j.retry.instances.inventoryService.max-attempts=3
resilience4j.retry.instances.inventoryService.wait-duration=500ms
resilience4j.retry.instances.inventoryService.enable-exponential-backoff=true
resilience4j.retry.instances.inventoryService.exponential-backoff-multiplier=2
resilience4j.retry.instances.inventoryService.randomized-wait-factor=0.5
# wait times: ~500ms, ~1s, ~2s (randomised by ±50%)
resilience4j.retry.instances.inventoryService.retry-exceptions=  java.net.SocketTimeoutException,  org.springframework.web.client.HttpServerErrorException$ServiceUnavailable

@Service
public class OrderService {

    @Retry(name = "inventoryService", fallbackMethod = "inventoryFallback")
    @CircuitBreaker(name = "inventoryService")  // combine with CB
    public InventoryResponse checkInventory(String sku) {
        return inventoryClient.checkStock(sku);  // retried on transient errors
    }

    // Fallback only called after all retries are exhausted
    private InventoryResponse inventoryFallback(String sku, Exception ex) {
        log.warn("Inventory check failed after retries for {}: {}", sku, ex.getMessage());
        return InventoryResponse.unavailable(sku);
    }
}
4

Bulkhead Pattern

Isolates failure by limiting concurrent calls to a downstream dependency (semaphore or thread-pool bulkhead), preventing one slow service from exhausting shared resources.

  • ✓Bulkhead prevents thread pool exhaustion: one slow service cannot starve all other endpoints.
  • ✓Semaphore bulkhead is lightweight (counts permits); thread-pool bulkhead provides true pool isolation.
  • ✓Thread-pool bulkhead requires CompletableFuture return type — it offloads the call to its own pool.
  • ✓Combine: Bulkhead (concurrency limit) + TimeLimiter (timeout) + CircuitBreaker (failure threshold).
  • ✓Set max-wait-duration=0ms to fail fast when the bulkhead is full — blocking is worse than fast failure.
  • ✓Size bulkheads based on expected concurrency and downstream SLAs — too small = false positives; too large = no protection.
Properties + Java — Resilience4j semaphore bulkhead
# application.properties — semaphore bulkhead
resilience4j.bulkhead.instances.paymentService.max-concurrent-calls=10
resilience4j.bulkhead.instances.paymentService.max-wait-duration=0ms
# max-wait-duration=0: reject immediately when full (fail-fast)

@Service
public class CheckoutService {

    @Bulkhead(name = "paymentService",
              type = Bulkhead.Type.SEMAPHORE,
              fallbackMethod = "paymentFallback")
    public PaymentResult processPayment(PaymentRequest req) {
        return paymentClient.charge(req);  // max 10 concurrent calls
    }

    // Called when bulkhead is full (BulkheadFullException)
    private PaymentResult paymentFallback(PaymentRequest req,
                                           BulkheadFullException ex) {
        log.warn("Payment bulkhead full — circuit protecting main thread pool");
        return PaymentResult.rejected("Payment service temporarily unavailable");
    }
}

// Monitor: Resilience4j metrics exposed via Micrometer
// resilience4j.bulkhead.available.concurrent.calls{name="paymentService"}
// Alert when available calls drops to 0 consistently → bulkhead too small
5

Timeouts & Deadline Propagation

Always set read/connect timeouts on HTTP clients; propagate deadlines through call chains so that downstream work is abandoned when the upstream caller has given up.

  • ✓Always set both connect timeout (TCP establishment) and read timeout (response wait) on every HTTP client
  • ✓Base read timeouts on actual p99 latency + safety margin — not arbitrary large values like 30 s
  • ✓Deadlines are absolute timestamps that shrink across call chains; timeouts reset at each hop
  • ✓gRPC propagates deadlines automatically; HTTP services need explicit header forwarding (e.g. X-Request-Deadline)
  • ✓Combine timeouts with circuit breakers — timeouts prevent individual call hangs, circuit breakers prevent repeated waits
  • ✓Timeout budget each leg proportionally: in A→B→C with a 5 s SLA, each leg timeout must fit within 5 s total
Java — RestClient, Feign, and WebClient timeout configuration
// Spring Boot 3 — RestClient with Apache HttpClient 5 timeouts
@Bean
public RestClient inventoryClient() {
    CloseableHttpClient httpClient = HttpClients.custom()
        .setConnectionRequestTimeout(Timeout.ofSeconds(2))   // connect timeout
        .setResponseTimeout(Timeout.ofSeconds(5))            // read timeout
        .build();

    return RestClient.builder()
        .baseUrl("http://inventory-service")
        .requestFactory(new HttpComponentsClientHttpRequestFactory(httpClient))
        .build();
}

// Feign client — application.yml
// spring:
//   cloud:
//     openfeign:
//       client:
//         config:
//           inventory-service:
//             connect-timeout: 2000   # ms
//             read-timeout: 5000      # ms

// WebClient with Netty reactor timeout
@Bean
public WebClient webClient() {
    HttpClient nettyClient = HttpClient.create()
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2_000)
        .responseTimeout(Duration.ofSeconds(5));
    return WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(nettyClient))
        .build();
}
Learn this free with Aria, your AI tutor → AiCanCode.org/learn/microservices