Distributed Tracing

In a monolithic application, tracing a request is straightforward since all calls execute on a single execution stack. In a microservices environment, a single user request can propagate through dozens of services across network boundaries. Distributed Tracing provides visibility into the complete journey of a request as it crosses process boundaries.

How It Works: Spans and Traces

Distributed tracing coordinates two key data concepts:

Trace (Global Request Journey - Trace ID: abc123xyz)
├─ Span A: Gateway (Client Request Received) [Duration: 100ms]
│    ├─ Span B: Order Service (Create Order DB) [Duration: 40ms]
│    └─ Span C: Payment Service (Charge Call) [Duration: 50ms]
│         └─ Span D: Stripe API (External Network Hop) [Duration: 30ms]

Trace: The complete end-to-end journey of a request. It is represented by a unique Trace ID (traceId) generated by the first service that intercepts the request (typically the API Gateway).
Span: A single logical unit of work (e.g., an HTTP request, a database query, or a message publish). Each span has a Span ID, a parent Span ID, a timestamp, and a duration.
SpanContext / Baggage: In addition to traceId/spanId, OpenTelemetry propagates Baggage — arbitrary key-value pairs (e.g., tenant.id, user.tier) that travel with the request and can be read by any downstream service without re-querying a database. Baggage is propagated via the baggage header, separately from traceparent.
Trace Context Propagation: To pass the traceId and active spanId across network borders, HTTP and Kafka calls inject metadata headers (most commonly using the W3C TraceContext standard: traceparent header).

Architecture: Where Spans Actually Go

A production tracing pipeline is rarely "app → Jaeger" directly. The standard pattern uses the OpenTelemetry Collector as an intermediary:

Distributed Tracing Architecture & Span Waterfall

Zero Coupling:

Application pods send OTLP to a local DaemonSet collector. If Jaeger goes down, application traffic is never impacted.

Centralized PII Redaction:

The collector scrubs credit cards, tokens, and social security numbers from span tags before persisting to disk.

Tail Sampling:

Collector keeps 100% of error spans and slow outlier requests, while sampling out 99% of normal 200 OK calls.

Sending spans directly from every pod to the backend couples your app's uptime to the tracing backend's uptime and makes it hard to change sampling policy without redeploying services. The Collector decouples this: it buffers, retries, redacts sensitive attributes, and can fan out to multiple backends (e.g., Jaeger for exploration + a span-metrics connector for RED metrics) without touching application code.

Setup & Implementation (Spring Boot 3 + Micrometer Tracing)

In Spring Boot 3, Micrometer Tracing (which integrates with OpenTelemetry) handles context propagation automatically for Servlet/WebFlux HTTP calls, RestTemplate, WebClient, and JDBC (via datasource-micrometer).

1. Add Dependencies

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

2. Configure Propagation and Exporting

# application.yml
management:
  tracing:
    sampling:
      probability: 1.0  # Sample 100% of requests for debugging (use ~0.05-0.1 in high-traffic production)
    propagation:
      type: w3c   # explicit: avoid falling back to B3 by default
  otlp:
    tracing:
      endpoint: http://otel-collector.monitoring:4318/v1/traces # Send spans to the Collector, not directly to Jaeger

3. Java Code Example: Manual Span Instrumentation

While Spring Boot automatically traces incoming and outgoing HTTP REST calls, you can define custom spans for critical business logic:

@Service
@Slf4j
public class OrderProcessingService {

    private final Tracer tracer;

    public OrderProcessingService(Tracer tracer) {
        this.tracer = tracer;
    }

    public void processComplexOrder(Order order) {
        // Create and start a custom child span
        Span customSpan = tracer.nextSpan().name("complex-validation").start();

        try (Tracer.SpanInScope ws = tracer.withSpan(customSpan)) {
            // Tag the span with useful context
            customSpan.tag("order.id", String.valueOf(order.getId()));
            customSpan.tag("customer.tier", order.getCustomerTier());

            // Run business logic
            validateOrderConstraints(order);

            log.info("Validating order rules within custom span context");
        } catch (Exception e) {
            customSpan.error(e);
            throw e;
        } finally {
            customSpan.end(); // Make sure to close the span
        }
    }
}

4. Propagating Context Across Async Boundaries

The single biggest source of broken traces in real Spring Boot systems is manual thread hand-off. Micrometer Tracing only auto-propagates context through instrumented infrastructure (Servlet filters, WebClient, @Async when configured correctly) — a raw ExecutorService, CompletableFuture.supplyAsync with the common pool, or a manually spawned Thread will silently drop context.

@Configuration
public class TracingExecutorConfig {

    // Wrap a plain ExecutorService so submitted tasks inherit the caller's trace context
    @Bean
    public ExecutorService tracedExecutorService(ContextPropagator contextPropagator) {
        ExecutorService delegate = Executors.newFixedThreadPool(16);
        return ContextExecutorService.wrap(delegate, ContextSnapshot::captureAll);
    }
}

@Service
public class NotificationService {

    private final ExecutorService tracedExecutorService;

    public NotificationService(ExecutorService tracedExecutorService) {
        this.tracedExecutorService = tracedExecutorService;
    }

    public void sendAsync(Order order) {
        // Context snapshot is captured at submission time and restored inside the task
        tracedExecutorService.submit(() -> {
            log.info("Sending notification, still inside traceId={}", order.getId());
        });
    }
}

For @Async methods, register a TaskDecorator that snapshots and restores context instead of relying on the default SimpleAsyncTaskExecutor:

@Configuration
@EnableAsync
public class AsyncConfig implements AsyncConfigurer {

    private final ContextSnapshotFactory contextSnapshotFactory;

    public AsyncConfig(ContextSnapshotFactory contextSnapshotFactory) {
        this.contextSnapshotFactory = contextSnapshotFactory;
    }

    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(8);
        executor.setTaskDecorator(runnable -> {
            ContextSnapshot snapshot = contextSnapshotFactory.captureAll();
            return () -> {
                try (ContextSnapshot.Scope scope = snapshot.setThreadLocals()) {
                    runnable.run();
                }
            };
        });
        executor.initialize();
        return executor;
    }
}

5. Propagating Context Through Kafka

Kafka producers/consumers are not HTTP calls, so trace context has to be carried in message headers rather than request headers. Spring Kafka's KafkaTemplate auto-instruments this when micrometer-tracing is on the classpath and ObservationRegistry is wired in, but understanding the header mechanics matters for cross-team debugging:

@Configuration
public class KafkaObservationConfig {

    // Enables trace context injection into Kafka record headers (traceparent) on send,
    // and extraction on consume, without manual header manipulation.
    @Bean
    public ProducerFactory<String, OrderEvent> producerFactory(ObservationRegistry registry) {
        DefaultKafkaProducerFactory<String, OrderEvent> factory =
                new DefaultKafkaProducerFactory<>(producerConfig());
        factory.addPostProcessor(kt -> kt.setObservationEnabled(true));
        return factory;
    }

    @Bean
    public ConsumerFactory<String, OrderEvent> consumerFactory() {
        DefaultKafkaConsumerFactory<String, OrderEvent> factory =
                new DefaultKafkaConsumerFactory<>(consumerConfig());
        return factory;
    }
}

With this enabled, a consumed record's processing span becomes a child of the producer's span, even though the two ran in different JVMs minutes apart — Kafka's durability breaks the usual "parent is still active" assumption, so tracing backends must support long-lived / asynchronous span links rather than only strict parent-child timing.

Sampling Strategies

Sampling is the difference between an affordable tracing system and a storage bill nobody signed off on. There are three broad strategies, and production systems typically combine them:

Strategy	How it decides	Pros	Cons
Head-based (probabilistic)	Decision made at the root span, before the request executes (e.g., 10% of requests)	Cheap, simple, low latency overhead	Can miss the exact slow/error request you care about — it's random
Rate-limiting	Cap traces per second regardless of volume	Predictable cost under traffic spikes	Under bursty error conditions, may cap out before capturing the interesting traces
Tail-based	Collector buffers the entire trace, then decides after seeing all spans (e.g., "keep if any span errored or p99 latency exceeded")	Guarantees you keep error and slow traces	Requires buffering the full trace in the Collector (memory cost), and all spans of a trace must route to the same Collector instance

A common production setup: head-based sampling at 5–10% for baseline visibility, combined with tail-based sampling rules in the OTel Collector that always keep traces containing an error span or exceeding a latency threshold — so you get statistical coverage plus guaranteed capture of the traces that actually matter for debugging.

:::tip Deep Dive Guide For a comprehensive architectural breakdown of Tail-based Sampling with OpenTelemetry Collector, Two-Tier Routing with loadbalancing exporter, RAM/OOM protection, and real-world failure case studies, see the dedicated guide: OpenTelemetry Sampling Strategies (Tail-based vs Head-based). :::

Correlating Traces with Metrics and Logs

Traces alone tell you about one request; they don't tell you "is this happening a lot." The three pillars become genuinely useful when cross-linked:

Logs → Traces: covered below via traceId/spanId in the log pattern — lets you jump from a log line to the full request waterfall.
Metrics → Traces (Exemplars): Micrometer supports attaching a sampled traceId as an exemplar to a Prometheus histogram bucket. This lets you click on a latency spike in a Grafana p99 chart and jump directly to an example trace that fell into that bucket, rather than guessing which recent trace was representative.

# application.yml — enable exemplar attachment to histograms
management:
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true
  observations:
    key-values:
      # ensures trace context is available when the histogram sample is recorded

Without exemplars, an engineer sees "p99 spiked to 2s at 14:03" and has to manually search tracing UI by time range and guess which trace explains it. With exemplars, the dashboard links straight to a specific traceId.

Log Correlation

To tie trace metrics to text output, configure your logging layout to print the active Trace ID and Span ID on every line:

<!-- logback-spring.xml -->
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level [traceId=%X{traceId}, spanId=%X{spanId}] %logger{36} - %msg%n</pattern>

This generates output matching the pattern:

2026-07-03 10:15:30.123 [http-nio-8080-exec-1] INFO  [traceId=d22f03f7e53a2ba1, spanId=b9a8cf5c66e2c342] c.e.o.OrderService - Saving order database entry

Security and PII Considerations

Spans are easy to over-instrument, and unlike metrics they routinely carry high-cardinality, human-readable data — which makes them a common source of accidental PII leaks into a tracing backend that may have looser access controls than your primary datastore.

Never tag spans with raw PII (email, full name, card numbers, auth tokens). Use hashed or truncated identifiers (customer.id, not customer.email).
Redact at the Collector, not just the app. Application-level discipline drifts over time across dozens of services; a redaction processor in the OTel Collector pipeline (regex-based attribute stripping) is a single enforcement point.
Treat baggage as public. Because baggage propagates to every downstream service and often into logs, anything placed in baggage should be treated as effectively broadcast within the trust boundary of the trace.
Restrict tracing UI access (Jaeger/Tempo/Grafana) with the same access control rigor as production logs — span attributes often reveal internal hostnames, query shapes, and business logic that shouldn't be broadly visible.

Testing Tracing Instrumentation

Tracing code is easy to get "silently wrong" — spans still show up, just with a broken parent-child relationship, and nothing throws an exception. Verify propagation explicitly in integration tests using an in-memory exporter:

@SpringBootTest
class OrderProcessingTracingTest {

    static InMemorySpanExporter spanExporter = InMemorySpanExporter.create();

    @Autowired
    private OrderProcessingService orderProcessingService;

    @Test
    void childSpanShouldBeLinkedToParentTrace() {
        spanExporter.reset();

        orderProcessingService.processComplexOrder(new Order(1L, "GOLD"));

        List<SpanData> spans = spanExporter.getFinishedSpanItems();
        SpanData validationSpan = spans.stream()
                .filter(s -> s.getName().equals("complex-validation"))
                .findFirst()
                .orElseThrow();

        assertThat(validationSpan.getAttributes().get(AttributeKey.stringKey("order.id")))
                .isEqualTo("1");
        // Verifies the span was actually created within an active trace, not orphaned
        assertThat(validationSpan.getParentSpanContext().isValid()).isTrue();
    }
}

Tool Comparison

Backend	Storage model	Best fit	Trade-off
Jaeger	Pluggable (Cassandra, Elasticsearch, or in-memory Badger)	Teams wanting a mature, self-hosted, Kubernetes-native UI	Operating the storage backend at scale is nontrivial
Grafana Tempo	Object storage (S3/GCS) — no indexing engine required	Teams already on Grafana/Prometheus/Loki wanting a unified stack	Trace search relies on exemplars/TraceQL rather than a full-text index by default
Zipkin	Simpler data model, Cassandra/Elasticsearch/MySQL	Legacy systems already standardized on B3 propagation	Smaller ecosystem momentum than OpenTelemetry-native backends
AWS X-Ray	Managed, AWS-native	Teams fully on AWS wanting zero operational overhead	Vendor lock-in; W3C interop requires the X-Ray OTel exporter
Vendor SaaS (Honeycomb, Datadog APM, etc.)	Managed, proprietary	Teams prioritizing time-to-value and advanced querying (BubbleUp, etc.)	Cost scales with span volume; another vendor dependency

Pros vs. Cons

Pros	Cons
Rapid Diagnostics: pinpoints precisely which service in a call graph is throwing errors or adding latency.	Storage Overhead: Tracing generates massive volumes of data; storing every trace is expensive.
Dependency Graphing: Automatically builds service-to-service dependency topologies for architectural auditing.	Context Propagation Fragility: If any service fails to forward the `traceparent` headers, the trace is broken.
Performance Bottleneck Detection: Highlights slow database queries or blocking client HTTP calls within a request flow.	Code Intrusion: Instrumenting legacy systems or custom protocols requires complex manual setup.
Metric/Log correlation via exemplars: turns "something is slow" into "here is the exact request" in one click.	PII surface area: spans are easy to over-tag with sensitive data compared to metrics.

Common Gotchas & Anti-Patterns

Broken Trace Chains on Manual Threads: Forgetting to pass trace headers when spawning asynchronous threads manually in Java (e.g., using Runnable, CompletableFuture.supplyAsync on the common pool, or a custom ExecutorService). The child threads will execute under a brand new traceId or without context.
- Solution: Wrap thread pools with ContextExecutorService, or use a TaskDecorator for @Async executors as shown above.
Sampling Errors: Sampling 100% of traffic in a high-scale production system. This generates massive network traffic and storage bills. Use adaptive/tail-based sampling instead of a flat probability in production.
Mismatched Propagation Headers: API Gateway sends W3C traceparent headers (00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01), but downstream services are configured to look for Zipkin B3 headers (X-B3-TraceId). This causes traces to split. Standardize on W3C and set management.tracing.propagation.type explicitly rather than relying on defaults.
Kafka Traces Appearing as Orphaned Roots: If setObservationEnabled(true) isn't configured on the producer/consumer factories, consumed messages start brand-new traces instead of linking back to the producing request — the trace silently "ends" at the point of publish.
Tail-based Sampling Split Across Collector Instances: In a horizontally scaled OTel Collector deployment, tail-based sampling requires all spans of a given trace to land on the same Collector instance (usually via consistent hashing on traceId at a load-balancing layer). Naive round-robin load balancing in front of Collectors breaks tail sampling because decisions are made on incomplete traces.
Unbounded Span Cardinality: Tagging spans with unbounded values (raw user input, full URLs with query strings, UUIDs as span names rather than attributes) blows up cardinality in downstream systems that index span names/attributes, degrading tracing UI query performance.