Bulkhead Pattern

The Bulkhead pattern isolates the resources (thread pools, connection pools, semaphore slots) allocated to each downstream dependency. When one dependency degrades or fails, it can only exhaust its own isolated resource slice — leaving all other dependencies unaffected and all other traffic flowing normally.

The name comes from the watertight compartments (bulkheads) built into a ship's hull: if one compartment is breached and floods, the flooding is contained locally. The ship does not sink. Without bulkheads, one breach floods the entire hull.

---

1. The Problem Without Bulkheads

In a typical microservice without resource isolation, all outbound service calls share the same application thread pool (Tomcat's default, usually 200 threads).

Bulkhead Isolation vs. Shared Thread Pool

Isolated Compartments (Healthy)

With bulkheads, Payment API is isolated to a 10-thread compartment. When Payment slowness occurs, only its 10 threads block. Catalog and User profiles continue serving normal traffic.

System Impact:

Click button to simulate Payment API slowness.

---

2. Thread Pool vs Semaphore Bulkheads — Choosing the Right Model

These are two fundamentally different isolation mechanisms. The correct choice depends on whether your execution model is blocking or non-blocking.

Bulkhead Types — Thread Pool vs. Semaphore

Thread Pool Bulkhead

Submits task to a dedicated ThreadPoolExecutor. The calling thread is freed immediately after submission!

✅ Calling Thread Freed: Zero caller thread blocking.

⚡ Ideal For: Synchronous blocking I/O (RestTemplate, JDBC).

⚠️ Overhead: Thread stack memory (~1MB/thread) + context switches.

---

3. Configuration with Resilience4j and Spring Boot

Dependency

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<!-- For metrics export to Prometheus -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Complete application.yml Configuration

resilience4j:

  # ── THREAD POOL BULKHEAD ────────────────────────────────────────────────────
  thread-pool-bulkhead:
    instances:

      # Payment service: critical, slow dependency — maximum isolation
      paymentService:
        maxThreadPoolSize: 10      # Hard cap: max 10 threads executing Payment calls
        coreThreadPoolSize: 5      # Always-warm threads (not evicted when idle)
        queueCapacity: 20          # Queue for tasks when all 10 threads are busy
                                   # Keep SMALL — large queues = latency spike before rejection
        keepAliveDuration: 20ms    # Idle thread beyond core evicted after 20ms

      # Shipping service: non-critical, slow — tight constraints to fail fast
      shippingService:
        maxThreadPoolSize: 5
        coreThreadPoolSize: 2
        queueCapacity: 10          # Small queue → fail fast when shipping is slow
        keepAliveDuration: 30ms

      # Inventory service: fast, high-volume — larger pool, minimal queue
      inventoryService:
        maxThreadPoolSize: 20
        coreThreadPoolSize: 10
        queueCapacity: 50
        keepAliveDuration: 60ms

  # ── SEMAPHORE BULKHEAD ───────────────────────────────────────────────────────
  bulkhead:
    instances:

      # User service: non-blocking reactive client — semaphore is appropriate
      userService:
        maxConcurrentCalls: 30     # Max in-flight concurrent calls
        maxWaitDuration: 50ms      # How long to wait for a slot before rejecting
                                   # Keep SHORT — long waits add latency under load

      # Catalog service: read-heavy, fast — generous limit
      catalogService:
        maxConcurrentCalls: 100
        maxWaitDuration: 10ms      # Fail very fast — catalog is not critical path

      # Analytics service: best-effort, non-critical
      analyticsService:
        maxConcurrentCalls: 5      # Tight limit — analytics should never starve other services
        maxWaitDuration: 0ms       # Reject immediately (0ms wait) if full

---

4. Implementation Patterns

Thread Pool Bulkhead — Blocking Service Call

@Service
@RequiredArgsConstructor
@Slf4j
public class PaymentService {

    private final PaymentClient paymentClient;
    private final MeterRegistry meterRegistry;

    // Thread Pool Bulkhead: paymentClient.charge() runs on a dedicated pool thread
    // Calling thread (Tomcat) returns immediately with a CompletableFuture
    @Bulkhead(name = "paymentService",
              type = Bulkhead.Type.THREADPOOL,
              fallbackMethod = "paymentFallback")
    public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
        return CompletableFuture.supplyAsync(() -> {
            log.debug("Processing payment {} on thread {}", request.getId(), Thread.currentThread().getName());
            return paymentClient.charge(request);
            // If this blocks for 10 seconds: the pool thread blocks, not the Tomcat thread
        });
    }

    // Fallback: called when pool is full (BulkheadFullException)
    // or when the wrapped call fails with any other exception
    public CompletableFuture<PaymentResult> paymentFallback(
            PaymentRequest request, BulkheadFullException ex) {
        log.warn("Payment bulkhead full for request {}. Pool exhausted.", request.getId());
        meterRegistry.counter("bulkhead.payment.rejected").increment();
        return CompletableFuture.completedFuture(
            PaymentResult.rejected("Service temporarily at capacity — please retry")
        );
    }

    public CompletableFuture<PaymentResult> paymentFallback(
            PaymentRequest request, Throwable ex) {
        log.error("Payment call failed: {}", ex.getMessage());
        return CompletableFuture.completedFuture(
            PaymentResult.failed("Payment unavailable: " + ex.getMessage())
        );
    }
}

Semaphore Bulkhead — Reactive / Non-Blocking Call

@Service
@RequiredArgsConstructor
@Slf4j
public class UserProfileService {

    private final UserServiceClient userClient;

    // Semaphore Bulkhead: maxConcurrentCalls=30 concurrent calls allowed
    // Runs on calling thread — appropriate for non-blocking reactive clients
    @Bulkhead(name = "userService",
              type = Bulkhead.Type.SEMAPHORE,
              fallbackMethod = "userProfileFallback")
    public UserProfile getUserProfile(String userId) {
        return userClient.getProfile(userId);
        // If 30 calls are already in-flight, the 31st waits maxWaitDuration (50ms)
        // Then throws BulkheadFullException → fallback
    }

    public UserProfile userProfileFallback(String userId, BulkheadFullException ex) {
        log.warn("User service semaphore full for userId={}. Returning cached profile.", userId);
        return UserProfile.anonymous(userId); // degraded response
    }

    public UserProfile userProfileFallback(String userId, Throwable ex) {
        log.error("User profile fetch failed for userId={}: {}", userId, ex.getMessage());
        return UserProfile.anonymous(userId);
    }
}

Programmatic Bulkhead (Without Annotations)

For dynamic configuration or when annotation-based AOP is unavailable:

@Service
@RequiredArgsConstructor
public class InventoryService {

    private final BulkheadRegistry bulkheadRegistry;
    private final ThreadPoolBulkheadRegistry threadPoolBulkheadRegistry;
    private final InventoryClient inventoryClient;

    // Programmatic thread pool bulkhead
    public CompletableFuture<StockLevel> getStockLevel(Long productId) {
        ThreadPoolBulkhead bulkhead = threadPoolBulkheadRegistry.bulkhead("inventoryService");

        return bulkhead.executeSupplier(() -> inventoryClient.getStock(productId))
            .toCompletableFuture()
            .exceptionally(ex -> {
                if (ex.getCause() instanceof BulkheadFullException) {
                    log.warn("Inventory bulkhead full for product {}", productId);
                    return StockLevel.unknown(productId);
                }
                log.error("Inventory call failed: {}", ex.getMessage());
                return StockLevel.unavailable(productId);
            });
    }

    // Programmatic semaphore bulkhead with event listeners
    public void registerBulkheadMetrics() {
        Bulkhead bulkhead = bulkheadRegistry.bulkhead("inventoryService");

        bulkhead.getEventPublisher()
            .onCallRejected(event ->
                log.warn("Inventory call rejected — semaphore full. Available permits: {}",
                    bulkhead.getMetrics().getAvailableConcurrentCalls()))
            .onCallPermitted(event ->
                log.debug("Inventory call permitted. Active: {}",
                    bulkhead.getMetrics().getMaxAllowedConcurrentCalls()
                    - bulkhead.getMetrics().getAvailableConcurrentCalls()))
            .onCallFinished(event ->
                log.debug("Inventory call finished: {}", event.getEventType()));
    }
}

Combining Bulkhead with Circuit Breaker and Retry

Bulkhead, Circuit Breaker, and Retry are complementary and most effective when combined. Each operates at a different timescale and protects against a different failure mode:

Resilience Stack — Bulkhead + Circuit Breaker + Retry

EXECUTION STACK (OUTER TO INNER):

@Bulkhead Position

The Bulkhead wraps the outside. If the thread pool or semaphore queue is full, the request is rejected immediately WITHOUT acquiring thread permits or triggering retries.

Critical Exception Handling Rule:

ignoreExceptions:
 – BulkheadFullException # NEVER retry pool rejections!

// Combined: Retry (inner) + CircuitBreaker (middle) + Bulkhead (outer)
// Order matters: Bulkhead rejection should NOT trigger retry
// Circuit Breaker open should NOT trigger retry
// Only transient execution failures should trigger retry
@Bulkhead(name = "paymentService", type = Bulkhead.Type.THREADPOOL, fallbackMethod = "paymentFallback")
@CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
@Retry(name = "paymentService", fallbackMethod = "paymentFallback")
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
    return CompletableFuture.supplyAsync(() -> paymentClient.charge(request));
}

resilience4j:
  thread-pool-bulkhead:
    instances:
      paymentService:
        maxThreadPoolSize: 10
        coreThreadPoolSize: 5
        queueCapacity: 20

  circuitbreaker:
    instances:
      paymentService:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        failureRateThreshold: 50          # Open if >50% of last 20 calls fail
        waitDurationInOpenState: 30s      # Wait 30s before trying half-open
        permittedNumberOfCallsInHalfOpenState: 5
        slowCallRateThreshold: 80         # Also open if >80% of calls are slow
        slowCallDurationThreshold: 3s     # "Slow" = takes more than 3 seconds

  retry:
    instances:
      paymentService:
        maxAttempts: 3
        waitDuration: 200ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2.0
        retryExceptions:
          - java.net.ConnectException
          - java.net.SocketTimeoutException
        ignoreExceptions:
          - io.github.resilience4j.bulkhead.BulkheadFullException   # Don't retry rejections
          - io.github.resilience4j.circuitbreaker.CallNotPermittedException

---

5. Pool Sizing — The Most Critical Configuration Decision

Incorrectly sized pools are the primary cause of bulkhead misconfiguration in production. Too small → unnecessary rejections that hurt availability. Too large → resource waste and context-switch overhead that defeats the purpose.

Little's Law Pool Sizing Calculator (L = λ × W)

Request Rate (λ):100 RPS

P99 Response Time (W):200ms (0.2s)

Safety Multiplier:1.5x

Calculated Bulkhead Sizing

Little's Law Base (L = λ × W):20 threads

maxThreadPoolSize:30 threads

coreThreadPoolSize:15 threads

queueCapacity:60 slots

Queue Sizing Golden Rule: Keep queueCapacity small (2× maxThreads). A giant queue converts fast failures into 30s slow failures!

Sizing Heuristics by Dependency Criticality

Dependency Type	Max Pool Size	Queue Capacity	Max Wait (Semaphore)	Rationale
Critical synchronous (payment, auth)	15–30	10–20	50–100ms	Larger pool for availability; fail fast on overload
Non-critical synchronous (notification, analytics)	5–10	5–10	0–20ms	Small pool; shed load early to protect critical paths
High-volume read (catalog, product)	20–50	50	10ms	High concurrency; fast calls need large semaphore
Slow batch/reporting	3–5	5	0ms	Extreme isolation; never let batch starve real-time traffic
Third-party external API	5–10	10	0–10ms	Rate limits apply; tight pool prevents exceeding quotas

---

6. Observability — Metrics, Alerting, and Diagnostics

Key Metrics to Track

Resilience4j exports metrics via Micrometer. With the Prometheus registry on the classpath, these are automatically available at /actuator/prometheus:

Metric	Description	Alert Threshold
resilience4j_thread_pool_bulkhead_queue_depth	Current tasks in queue	> 80% of queueCapacity
resilience4j_thread_pool_bulkhead_thread_pool_size	Active threads in pool	Sustained at maxThreadPoolSize
resilience4j_bulkhead_available_concurrent_calls	Available semaphore permits	Consistently near 0
resilience4j_bulkhead_max_allowed_concurrent_calls	Configured limit	(configuration baseline)
resilience4j_bulkhead_calls_total{kind="rejected"}	Total rejected calls	Any non-zero rate → investigate
resilience4j_bulkhead_calls_total{kind="successful"}	Total successful calls	Drop in rate → possible issue

Prometheus Alerting Rules

groups:
  - name: bulkhead-alerts
    rules:
      # Critical: high rejection rate — bulkhead pool is continuously saturated
      - alert: BulkheadRejectionRateHigh
        expr: >
          rate(resilience4j_bulkhead_calls_total{kind="rejected"}[5m]) /
          rate(resilience4j_bulkhead_calls_total[5m]) > 0.10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Bulkhead {{ $labels.name }} rejecting >10% of calls"
          description: "Rejection rate {{ $value | humanizePercentage }}. Check downstream {{ $labels.name }} for slowness."

      # Warning: thread pool queue depth growing — heading toward rejection
      - alert: BulkheadQueueDepthHigh
        expr: >
          resilience4j_thread_pool_bulkhead_queue_depth /
          resilience4j_thread_pool_bulkhead_queue_capacity > 0.80
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Bulkhead {{ $labels.name }} queue at {{ $value | humanizePercentage }}"

      # Warning: all pool threads active — approaching saturation
      - alert: BulkheadPoolSaturated
        expr: >
          resilience4j_thread_pool_bulkhead_thread_pool_size ==
          resilience4j_thread_pool_bulkhead_max_thread_pool_size
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Bulkhead {{ $labels.name }} pool fully saturated for 3m"

Grafana Dashboard Queries

# Rejection rate per bulkhead (% of total calls rejected)
rate(resilience4j_bulkhead_calls_total{kind="rejected"}[5m]) /
rate(resilience4j_bulkhead_calls_total[5m]) * 100

# Thread pool utilization % (for thread pool bulkheads)
resilience4j_thread_pool_bulkhead_thread_pool_size /
resilience4j_thread_pool_bulkhead_max_thread_pool_size * 100

# Queue depth trend (leading indicator for upcoming rejections)
resilience4j_thread_pool_bulkhead_queue_depth

# Semaphore available permits (lower = more concurrency pressure)
resilience4j_bulkhead_available_concurrent_calls

Structured Logging for Bulkhead Events

@Configuration
public class BulkheadObservabilityConfig {

    @Bean
    public ApplicationRunner bulkheadEventListeners(
            ThreadPoolBulkheadRegistry tpRegistry,
            BulkheadRegistry semaphoreRegistry,
            MeterRegistry meterRegistry) {

        return args -> {
            // Thread Pool Bulkhead events
            tpRegistry.getAllBulkheads().forEach(bulkhead ->
                bulkhead.getEventPublisher()
                    .onCallRejected(event -> {
                        log.warn("ThreadPool bulkhead [{}] rejected call. Active threads: {}/{}, Queue: {}/{}",
                            bulkhead.getName(),
                            bulkhead.getMetrics().getThreadPoolSize(),
                            bulkhead.getMetrics().getMaximumThreadPoolSize(),
                            bulkhead.getMetrics().getQueueDepth(),
                            bulkhead.getMetrics().getQueueCapacity());
                        meterRegistry.counter("bulkhead.threadpool.rejected",
                            "name", bulkhead.getName()).increment();
                    })
            );

            // Semaphore Bulkhead events
            semaphoreRegistry.getAllBulkheads().forEach(bulkhead ->
                bulkhead.getEventPublisher()
                    .onCallRejected(event -> {
                        log.warn("Semaphore bulkhead [{}] rejected call. Available permits: {}/{}",
                            bulkhead.getName(),
                            bulkhead.getMetrics().getAvailableConcurrentCalls(),
                            bulkhead.getMetrics().getMaxAllowedConcurrentCalls());
                        meterRegistry.counter("bulkhead.semaphore.rejected",
                            "name", bulkhead.getName()).increment();
                    })
                    .onCallPermitted(event ->
                        meterRegistry.counter("bulkhead.semaphore.permitted",
                            "name", bulkhead.getName()).increment())
            );
        };
    }
}

---

7. Bulkhead for Database Connection Pools

Bulkheading is not limited to thread pools for outbound HTTP calls — the same pattern applies to database connection pools (HikariCP). Multiple services or query types sharing one connection pool can cause identical starvation behavior.

Problem: Reporting Queries Starving Transactional Queries

HikariCP pool (20 connections):
  5 long-running reporting queries grab 5 connections (30 second each)
  OLTP queries queue for a connection
  → timeout after 30s → user-facing errors
  → reporting is starving transactions

Solution: Separate DataSources per Workload Type

@Configuration
public class MultiDataSourceConfig {

    // Primary DataSource: OLTP — small pool, strict timeout
    @Bean
    @Primary
    @ConfigurationProperties("spring.datasource.oltp")
    public DataSource oltpDataSource() {
        HikariDataSource ds = new HikariDataSource();
        ds.setPoolName("HikariPool-OLTP");
        ds.setMaximumPoolSize(15);
        ds.setMinimumIdle(5);
        ds.setConnectionTimeout(5_000);    // Fail fast if no OLTP connection in 5s
        ds.setMaxLifetime(1_800_000);
        return ds;
    }

    // Reporting DataSource: separate pool for heavy queries
    @Bean
    @ConfigurationProperties("spring.datasource.reporting")
    public DataSource reportingDataSource() {
        HikariDataSource ds = new HikariDataSource();
        ds.setPoolName("HikariPool-Reporting");
        ds.setMaximumPoolSize(5);          // Tight cap — reporting should never starve OLTP
        ds.setMinimumIdle(1);
        ds.setConnectionTimeout(30_000);   // Can wait longer — reporting is not user-facing
        ds.setMaxLifetime(1_800_000);
        return ds;
    }
}

// Route repositories to correct DataSource via Spring's AbstractRoutingDataSource
// or by using separate EntityManagerFactory instances per DataSource

---

8. Common Failure Modes and Anti-Patterns

Anti-Pattern 1 — Queue Too Large (Hidden Latency Spike)

# ❌ Queue of 1000 — effectively disables fast-fail behavior
thread-pool-bulkhead:
  instances:
    paymentService:
      maxThreadPoolSize: 10
      queueCapacity: 1000    # 1000 requests queue behind 10 threads
                              # At 200ms/request: last request waits 20 seconds before failing
                              # Client times out anyway; queue just adds latency

# ✅ Small queue — fail fast and let the caller retry or degrade
      queueCapacity: 20      # At most 20 requests queue; rest rejected immediately with clear error

Anti-Pattern 2 — All Dependencies in One Bulkhead (The "Shared Silo")

// ❌ All external calls share one bulkhead — defeats the purpose
// If payment is slow, ALL external calls are affected

@Bulkhead(name = "externalServices")  // One bulkhead for payment, shipping, catalog, user
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) { ... }

@Bulkhead(name = "externalServices")  // Shares pool with payment — catalog now affected by payment slowness
public CompletableFuture<CatalogItem> getCatalogItem(Long id) { ... }

// ✅ One bulkhead per logical dependency group
@Bulkhead(name = "paymentService")
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) { ... }

@Bulkhead(name = "catalogService")    // Completely isolated from payment
public CompletableFuture<CatalogItem> getCatalogItem(Long id) { ... }

Anti-Pattern 3 — Double Threading (Thread Pool Waste)

// ❌ Thread Pool Bulkhead wrapping an already-async @Async method
// Call path: Tomcat thread → Bulkhead pool thread → @Async pool thread = 3 threads per request

@Bulkhead(name = "paymentService", type = THREADPOOL)
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
    return paymentClient.chargeAsync(request); // chargeAsync is already @Async — uses another pool!
    // Three thread contexts: Tomcat → bulkhead pool → async pool
    // Wasted threads, wasted context switches
}

// ✅ Use Semaphore Bulkhead when the call is already async/non-blocking
@Bulkhead(name = "paymentService", type = SEMAPHORE)
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
    return paymentClient.chargeAsync(request);
}

Anti-Pattern 4 — Mixing Different SLA Dependencies in One Pool

// ❌ Fast critical endpoint and slow non-critical endpoint share one bulkhead
@Bulkhead(name = "dataServices")
public UserProfile getProfile(String userId) { ... }   // 50ms, critical

@Bulkhead(name = "dataServices")  // Shares pool with getProfile!
public AnalyticsReport generateReport(ReportRequest req) { ... }  // 5000ms, non-critical
// Report fills the pool → profile calls rejected → critical path broken

// ✅ Separate by criticality and SLA
@Bulkhead(name = "userService")           // Fast, generous pool
public UserProfile getProfile(String userId) { ... }

@Bulkhead(name = "analyticsService")      // Small, tight pool — analytics shed load before user
public AnalyticsReport generateReport(ReportRequest req) { ... }

Anti-Pattern 5 — No Fallback Strategy

// ❌ BulkheadFullException propagates to caller — unhandled exception
@Bulkhead(name = "paymentService", type = THREADPOOL)
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
    return CompletableFuture.supplyAsync(() -> paymentClient.charge(request));
    // No fallback — caller gets BulkheadFullException, HTTP 500 to user
}

// ✅ Defined fallback for each exception type the bulkhead can throw
@Bulkhead(name = "paymentService", type = THREADPOOL, fallbackMethod = "paymentFallback")
public CompletableFuture<PaymentResult> processPayment(PaymentRequest request) {
    return CompletableFuture.supplyAsync(() -> paymentClient.charge(request));
}

// Fallback for pool exhaustion
public CompletableFuture<PaymentResult> paymentFallback(
        PaymentRequest request, BulkheadFullException ex) {
    return CompletableFuture.completedFuture(
        PaymentResult.queued("Payment queued — will process shortly")
    );
}

// Fallback for other errors
public CompletableFuture<PaymentResult> paymentFallback(
        PaymentRequest request, Throwable ex) {
    return CompletableFuture.completedFuture(
        PaymentResult.failed("Payment service unavailable")
    );
}

---

9. Adaptive Bulkhead Sizing (Advanced)

Static pool sizes are adequate for stable traffic patterns, but real production traffic fluctuates. Adaptive sizing adjusts pool limits based on observed metrics:

@Service
@RequiredArgsConstructor
@Slf4j
public class AdaptiveBulkheadManager {

    private final ThreadPoolBulkheadRegistry bulkheadRegistry;
    private final MeterRegistry meterRegistry;

    // Periodically evaluate and adjust pool sizes based on rejection rate and queue depth
    @Scheduled(fixedDelay = 30_000)  // Evaluate every 30 seconds
    public void adjustBulkheadSizes() {
        bulkheadRegistry.getAllBulkheads().forEach(this::evaluateAndAdjust);
    }

    private void evaluateAndAdjust(ThreadPoolBulkhead bulkhead) {
        ThreadPoolBulkhead.Metrics metrics = bulkhead.getMetrics();

        double queueUtilization = (double) metrics.getQueueDepth() / metrics.getQueueCapacity();
        double threadUtilization = (double) metrics.getThreadPoolSize() / metrics.getMaximumThreadPoolSize();

        // High utilization → increase pool size (up to a configured maximum)
        if (queueUtilization > 0.80 && threadUtilization > 0.90) {
            log.warn("Bulkhead [{}] under pressure — queue {}%, threads {}%. Consider increasing pool size.",
                bulkhead.getName(),
                (int)(queueUtilization * 100),
                (int)(threadUtilization * 100));
            meterRegistry.counter("bulkhead.sizing.pressure_detected",
                "name", bulkhead.getName()).increment();
        }

        // Very low utilization → log for potential downsizing
        if (queueUtilization < 0.10 && threadUtilization < 0.20) {
            log.debug("Bulkhead [{}] lightly utilized — queue {}%, threads {}%. Pool may be oversized.",
                bulkhead.getName(),
                (int)(queueUtilization * 100),
                (int)(threadUtilization * 100));
        }
    }
}

---

10. Interview Questions

Q1: What is the Bulkhead pattern and why is it necessary?

The Bulkhead pattern isolates resources allocated to each downstream dependency so that the failure or degradation of one dependency cannot consume resources needed by others.

• Without bulkheading: All service calls share a single application thread pool (e.g. Tomcat's 200 threads). A single slow downstream dependency blocks all threads, making the entire microservice unresponsive to all traffic.
• With bulkheading: Each dependency gets a dedicated resource slice (thread pool or semaphore). Slowness in one dependency is strictly contained to its isolated slice.

---

Q2: When do you choose Thread Pool Bulkhead vs. Semaphore Bulkhead?

• Thread Pool Bulkhead is built for blocking I/O (RestTemplate, JDBC, legacy HTTP clients). The call executes on a separate pool worker thread, completely freeing the calling thread immediately.
• Semaphore Bulkhead is built for non-blocking / reactive calls (WebFlux, R2DBC). It limits concurrency via an atomic counter without allocating separate thread stacks, since non-blocking threads are not held during I/O anyway.

---

Q3: How do you size a thread pool for a Bulkhead?

Use Little's Law: $\text{Pool Size } (L) = \text{Request Rate } (\lambda) \times \text{Average Latency } (W)$

• For 100 RPS with 200ms P99 latency: $100 \times 0.2 = 20$ threads base.
• Add a safety multiplier of 1.5–2.0 for burst traffic $\rightarrow 30$ threads.
• For non-critical dependencies, intentionally cap the pool below the calculated size to enforce early load shedding.

---

Q4: Why should the queue capacity be kept small?

A large queue converts fast failures into slow failures.

If all pool threads are busy on a 3-second timeout, a queue capacity of 200 means requests wait 3 seconds in line before failing anyway. A small queue (e.g. $2 \times \text{maxPoolSize}$) rejects excess requests immediately with a fast 0ms failure, preserving user experience and allowing immediate fallback.

---

Q5: How does Bulkhead interact with Circuit Breaker?

They protect against different failure dimensions:

• Bulkhead limits concurrency at any single moment (prevents resource starvation).
• Circuit Breaker tracks failure rates over time (trips OPEN during sustained degradation).

Wrapping order: The @Bulkhead wraps the outside (rejecting over-concurrency), while @CircuitBreaker sits inside (stopping calls when unhealthy). Bulkhead rejections (BulkheadFullException) must be ignored by @Retry so rejections are never retried.

---

See Also

• Circuit Breaker Pattern
• Retry Pattern
• Rate Limiting Algorithms
• Database Connection Pooling