4: Distributed Tracing: Following a Request Across Services

Distributed tracing tracks a single user request as it travels across multiple microservices by propagating a shared Trace ID via HTTP headers (the W3C traceparent standard). OpenTelemetry handles this automatically: instrument your services, and every cross-service call stitches into one unified trace visible in Jaeger or Grafana Tempo.

In Chapter 3, we instrumented a single FastAPI service with OpenTelemetry and visualized its traces in Jaeger. We saw spans for incoming HTTP requests, database queries, and even our own custom business logic, all within one service.

But production systems are rarely one service. Your API gateway calls an order service, which calls a payment service, which calls a notification service. When something breaks, the question becomes: “Which service in this chain is the culprit?”

This is where Distributed Tracing comes in. It carries the same Trace ID across the network, stitching together spans from multiple services into a single, unified trace.

Why can’t single-service tracing show cross-service failures?

In Chapter 3, we had a single service producing a trace like this:

    gantt
    title Single Service Trace
    dateFormat  X
    axisFormat %s
    section API Gateway
    GET /checkout       :a1, 0, 10
    Validate Cart       :a2, 1, 3
    Invoke Order Service :a3, 4, 9

That last span, Call Order Service, is a black box. We see that it took 5 seconds, but we have no idea why. Was it the Order Service’s database? Was it a downstream call the Order Service made? The trace ends at your service boundary.

What we want is this:

    gantt
    title Distributed Trace (Two Services)
    dateFormat  X
    axisFormat %s
    section API Gateway
    GET /checkout         :a1, 0, 10
    Validate Cart         :a2, 1, 3
    HTTP POST /orders     :a3, 4, 9
    section Order Service
    POST /orders          :b1, 4.5, 9
    Check Inventory       :b2, 5, 6
    Insert DB Record      :b3, 6.5, 8.5

Now you can see: the API Gateway spent 5 seconds waiting, and most of that was the Order Service’s Insert DB Record span (2 seconds). Root cause identified.

How It Works: The traceparent Header

The magic behind distributed tracing is surprisingly simple. When Service A calls Service B over HTTP, it sends a special header called traceparent.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
              |             |                       |             |
           version       trace_id                span_id        flags

This is the W3C Trace Context standard. The key rules:

• trace_id stays the same across all services for one request.
• span_id is unique per span, each service creates its own.
• The receiving service reads this header, creates a child span linked to the parent, and continues the trace.

The Setup: Two FastAPI Services

We are going to extend our Chapter 3 setup. Instead of one service, we now have two:

• API Gateway (port 8000): Receives the user’s request, calls the Order Service.
• Order Service (port 8001): Processes the order, inserts a record.

The Order Service (New)

This is a minimal FastAPI app that simulates processing an order.

# order_service.py
import asyncio
from fastapi import FastAPI

app = FastAPI()

@app.post("/orders")
async def create_order(order: dict):
    # Simulate inventory check
    await asyncio.sleep(0.3)
    
    # Simulate a slow DB insert
    await asyncio.sleep(1.5)
    
    return {"order_id": "ord-789", "status": "created"}

Nothing special here, no OpenTelemetry code at all. We will let the auto-instrumentation from Chapter 3 handle everything.

The API Gateway (Extended from Chapter 3)

The only change from Chapter 3 is that our gateway now makes an outbound HTTP call to the Order Service.

# api_gateway.py
from contextlib import asynccontextmanager

import httpx
from fastapi import FastAPI

ORDER_SERVICE_URL = "http://localhost:8001"

@asynccontextmanager
async def lifespan(application: FastAPI):
    # Create a single client for the app's lifetime (reuses connection pools)
    application.state.http_client = httpx.AsyncClient()
    yield
    await application.state.http_client.aclose()

app = FastAPI(lifespan=lifespan)

@app.get("/checkout")
async def checkout():
    client = app.state.http_client
    response = await client.post(
        f"{ORDER_SERVICE_URL}/orders",
        json={"item": "widget", "qty": 2}
    )
    return {"checkout": "complete", "order": response.json()}

Again, zero OpenTelemetry code. The key difference from Chapter 3 is just the httpx call.

Running It: Zero-Code Distributed Tracing

Make sure Jaeger is running (same as Chapter 3):

# If you already have the jaeger container from Chapter 3:
docker start jaeger

# Or, if starting fresh:
docker run -d --rm --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Install all the dependencies for both services. Here is the full list:

# requirements.txt
fastapi
uvicorn[standard]
httpx
opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-httpx
loguru

pip install -r requirements.txt

Start both services in separate terminals:

# Terminal 1: Order Service
opentelemetry-instrument \
    --traces_exporter otlp \
    --exporter_otlp_endpoint http://localhost:4317 \
    --service_name order-service \
    uvicorn order_service:app --port 8001

# Terminal 2: API Gateway
opentelemetry-instrument \
    --traces_exporter otlp \
    --exporter_otlp_endpoint http://localhost:4317 \
    --service_name api-gateway \
    uvicorn api_gateway:app --port 8000

Fire a request:

curl http://localhost:8000/checkout

Open http://localhost:16686, select api-gateway and click Find Traces. You will see a single trace with spans from both services.

In the Jaeger UI, the trace view will show a waterfall timeline similar to our Mermaid diagram above: the api-gateway root span at the top, the outgoing HTTP POST span nested below it, and the order-service POST /orders span as a child, all sharing the same trace_id. You can expand each span to see its attributes, duration, and service name.

What Just Happened?

Let’s trace the flow step by step:

1. curl hits api-gateway → OTel auto-instrumentation creates a root span with a new trace_id.
2. api-gateway calls order-service via httpx → the httpx instrumentation automatically injects the traceparent header into the outgoing request.
3. order-service receives the request → the FastAPI instrumentation reads the traceparent header and creates a child span linked to the same trace_id.
4. Both services export their spans to Jaeger → Jaeger stitches them together into one trace.

You wrote zero tracing code. The auto-instrumentation libraries for fastapi and httpx handled the propagation for you.

Under the Hood: Manual Context Propagation

Step 1: Caller Side (Manual Span + Manual Inject)

In the API Gateway, start a span for the incoming request and inject the active context into outbound headers.

# api_gateway.py (fully manual propagation)
from opentelemetry import propagate, trace

tracer = trace.get_tracer(__name__)

@app.get("/checkout")
async def checkout():
    client = app.state.http_client

    # 1. Start a span for this unit of work
    with tracer.start_as_current_span("GET /checkout (manual)"):
        # 2. Use a plain dict as the carrier
        headers = {}

        # 3. Inject current context into outbound headers
        #    (default propagator typically writes traceparent and baggage)
        propagate.inject(headers)

        response = await client.post(
            f"{ORDER_SERVICE_URL}/orders",
            json={"item": "widget", "qty": 2},
            headers=headers,
        )
        return {"checkout": "complete", "order": response.json()}

Step 2: Receiver Side (Manual Extract + Child Span)

In the Order Service, read the incoming headers, extract context and start a child span with that extracted context.

# order_service.py (fully manual propagation)
import asyncio
from fastapi import FastAPI, Request
from opentelemetry import propagate, trace

app = FastAPI()
tracer = trace.get_tracer(__name__)

@app.post("/orders")
async def create_order(order: dict, request: Request):
    # 1. Build a carrier from incoming HTTP headers
    carrier = dict(request.headers)

    # 2. Extract remote parent context from trace headers
    extracted_context = propagate.extract(carrier)

    # 3. Start a span that becomes a child of the remote parent
    with tracer.start_as_current_span("POST /orders (manual)", context=extracted_context):
        await asyncio.sleep(0.3)
        await asyncio.sleep(1.5)
        return {"order_id": "ord-789", "status": "created"}

This is the complete manual loop: inject on the sender, extract on the receiver. Both spans share the same trace_id, so Jaeger renders one connected distributed trace.

What OTel Recommends

OpenTelemetry recommends this manual pattern when you are instrumenting custom or unsupported flows. If a mature instrumentation library exists for your framework or client, use it in production to avoid missed propagation on individual calls.

Adding Custom Spans (Building on Chapter 3)

In Chapter 3, we learned how to create custom spans for business logic. That same technique works here across services. Let’s add a custom span inside the Order Service:

# order_service.py, adding a custom span
import asyncio
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/orders")
async def create_order(order: dict):
    with tracer.start_as_current_span("check_inventory") as span:
        span.set_attribute("item", order.get("item", ""))
        await asyncio.sleep(0.3)

    with tracer.start_as_current_span("insert_order_record") as span:
        span.set_attribute("order.item", order.get("item", ""))
        span.set_attribute("order.qty", order.get("qty", 0))
        await asyncio.sleep(1.5)

    return {"order_id": "ord-789", "status": "created"}

Now in Jaeger, instead of one opaque POST /orders span, you see the breakdown: check_inventory (300ms) and insert_order_record (1.5s). The slow DB insert is immediately obvious.

Connecting Logs to Distributed Traces

Remember our Loguru + OTel patcher from Chapter 3? It injected trace_id and span_id into every log line. That setup works across services without any changes.

Since both services share the same trace_id, you can correlate logs across service boundaries. If your services output structured JSON logs (as we set up in Chapter 1), you can search by trace_id using simple command-line tools:

# Search across both service logs for a specific trace
cat api_gateway.log order_service.log | \
  jq -s 'sort_by(.timestamp)' | \
  jq '.[] | select(.trace_id == "0af7651916cd43dd8448eb211c80319c")'

This returns log lines from both the API Gateway and the Order Service, in chronological order, all correlated to the same user request.

If you later adopt a log aggregation tool like Grafana Loki, you can run a similar query in LogQL:

{service_name=~"api-gateway|order-service"} | json | trace_id = "0af7651916cd43dd8448eb211c80319c"

This is the payoff of the foundation we built in Chapters 1–3.

Conclusion

Distributed tracing connects the dots across service boundaries. With the auto-instrumentation foundation from Chapter 3 and a few additional dependencies, we went from tracing one service to tracing an entire request chain, without writing any propagation code.

What we built across the series so far:

Next Steps:

1. Add a second service to your existing Chapter 3 setup.
2. Install the corresponding HTTP client instrumentation package.
3. Make a request and find the unified trace in Jaeger.

Frequently Asked Questions

What is distributed tracing?

Distributed tracing follows a single request across multiple services by propagating a unique Trace ID through HTTP headers. Each service adds its own spans to the trace, creating a unified timeline that shows exactly where time is spent and where failures occur across your entire system.

How does OpenTelemetry propagate trace context between services?

OTel uses the W3C Trace Context standard, injecting a traceparent header into outgoing HTTP requests. The receiving service extracts this header and continues the trace. This happens automatically when you install the appropriate instrumentation libraries.

Do I need to modify my HTTP client code for distributed tracing?

No. OpenTelemetry’s auto-instrumentation patches popular HTTP libraries (requests, httpx, aiohttp) to automatically inject trace headers. You just install the instrumentation package and the propagation happens transparently.

Can I trace across services written in different languages?

Yes. OpenTelemetry supports Python, Java, Go, Node.js, .NET, Rust and more. Since all implementations use the same W3C Trace Context standard, traces propagate seamlessly across language boundaries.

What is the difference between the traceparent and tracestate headers?

traceparent carries the trace ID, parent span ID and sampling flag (required). tracestate carries vendor-specific metadata like tenant ID or deployment ring (optional). Both are part of the W3C Trace Context specification.

How do I trace asynchronous message queues like Kafka or RabbitMQ?

OpenTelemetry provides instrumentation libraries for Kafka, RabbitMQ, Celery and other message brokers. The trace context is serialized into message headers, so the consuming service can continue the trace even though the communication is asynchronous.

Resources

• W3C Trace Context Specification
• OTel Python: Instrumentation Libraries
• OTel Propagation API
• OpenTelemetry Collector
• Chapter 4: Code