6: Correlating Telemetry Signals , The Pivot Workflow

Correlating telemetry signals means linking your logs, traces and metrics together via a shared trace_id so you can start from a metric spike in Grafana, click through to the exact trace in Jaeger, and land on the specific log line that caused the issue. This “Pivot Workflow” is what turns three separate data streams into a unified debugging experience.

In Chapter 5, we added Metrics alongside our Logs (Chapter 1) and Traces (Chapters 3–4). We can now count requests, measure latencies, and track business events.

But here’s the dirty secret: having all three telemetry signals means nothing if they aren’t connected.

Right now, your metrics dashboard shows a latency spike. Your traces show individual requests. Your logs show error messages. But when an incident happens, you’re still context-switching between three different tools, manually copying IDs from one into the other.

The real power of observability comes from the Pivot Workflow: the ability to start from a high-level signal and drill down to the exact cause in seconds.

The Pivot Workflow

This is the mental model that every on-call engineer needs:

    graph TD
    A[📊 Metrics Dashboard
'P99 latency spike at 14:30'] -->|Click anomaly| B[🔍 Traces
'Show me the slowest traces
from 14:25-14:35']
    B -->|Expand slowest trace| C[📝 Logs
'Show me all logs for
trace_id = abc123']
    C -->|Root cause found| D[🐛 Fix
'DB connection pool exhausted']
    style A fill:#cfc,stroke:#333
    style B fill:#ccf,stroke:#333
    style C fill:#f9f,stroke:#333
    style D fill:#ffc,stroke:#333

• Metrics → tell you something is wrong (alert, dashboard spike).
• Traces → tell you where it’s wrong (which service, which span).
• Logs → tell you why it’s wrong (the exact error message, stack trace).

Without correlation, each step requires manual searching. With correlation, each step is a single click.

The Glue: trace_id

The secret ingredient that connects all three signals is the Trace ID. If every log line and every metric data point carries a reference to the trace that produced it, you can navigate freely between signal types.

We’ve already done half the work:

Step 1: Log Correlation (Recap + Enhancement)

In Chapter 3, we created an OTel patcher for Loguru that injects trace_id and span_id. Let’s enhance it to also include the service name and span name, making cross-service log searches even more powerful.

# logging_setup.py
import sys
from opentelemetry import trace
from loguru import logger


def otel_patcher(record):
    """Inject OTel context into every log record."""
    span = trace.get_current_span()
    if span.is_recording():
        ctx = span.get_span_context()
        record["extra"]["trace_id"] = format(ctx.trace_id, "032x")
        record["extra"]["span_id"] = format(ctx.span_id, "016x")
        record["extra"]["span_name"] = span.name
    else:
        record["extra"]["trace_id"] = "00000000000000000000000000000000"
        record["extra"]["span_id"] = "0000000000000000"
        record["extra"]["span_name"] = ""


def setup_logging(service_name: str):
    """Configure Loguru for production with OTel correlation."""
    logger.remove()
    logger.configure(patcher=otel_patcher)

    # JSON format for production
    log_format = (
        '{{"timestamp": "{time:YYYY-MM-DDTHH:mm:ss.SSSZ}", '
        '"level": "{level.name}", '
        '"service": "' + service_name + '", '
        '"message": "{message}", '
        '"trace_id": "{extra[trace_id]}", '
        '"span_id": "{extra[span_id]}", '
        '"span_name": "{extra[span_name]}"'
        "{extra_json}}}"
    )

    logger.add(sys.stdout, format=log_format, level="INFO")
    return logger

Now both services import this shared module:

# api_gateway.py (top of file)
from logging_setup import setup_logging

logger = setup_logging("api-gateway")

# order_service.py (top of file)
from logging_setup import setup_logging

logger = setup_logging("order-service")

Every log line now carries the trace_id, so you can search across all services:

# Find all logs for a specific trace across both services
cat api_gateway.log order_service.log | \
  jq -s 'sort_by(.timestamp)' | \
  jq '.[] | select(.trace_id == "0af7651916cd43dd8448eb211c80319c")'

Step 2: Metric-to-Trace Correlation with Exemplars

This is the missing link. Right now, when you see a latency spike in Prometheus, you have to guess which trace to look at. Exemplars solve this by attaching a sample trace_id to each metric data point.

    graph LR
    A[Histogram Bucket
P99 = 2100ms] -->|exemplar| B[trace_id: abc123]
    B -->|click| C[Jaeger Trace View
Shows full request timeline]
    style A fill:#cfc,stroke:#333
    style B fill:#ffc,stroke:#333
    style C fill:#ccf,stroke:#333

What Is an Exemplar?

An exemplar is a sample reference attached to a metric measurement. When you record a histogram value of 2100ms, the exemplar says: “Here’s one specific trace that contributed to this bucket: trace_id=abc123.”

Think of it as a bookmark: the metric tells you “latency is high”, and the exemplar gives you a direct link to one specific slow request to investigate.

Implementation

We modify our metrics middleware from Chapter 5 to attach the current trace context as an exemplar:

# api_gateway.py , updated metrics middleware
import time

from fastapi import Request
from opentelemetry import context, metrics, trace


@app.middleware("http")
async def metrics_middleware(request: Request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    duration_ms = (time.perf_counter() - start) * 1000

    attributes = {
        "http.method": request.method,
        "http.route": request.url.path,
        "http.status_code": response.status_code,
    }
    request_counter.add(1, attributes)

    # Attach the current trace context as an exemplar
    ctx = context.get_current()
    request_duration.record(duration_ms, attributes, ctx)

    return response

The key change is passing ctx (the current OTel context) to request_duration.record(). The Prometheus exporter will extract the trace_id and span_id from this context and attach them as exemplars.

Querying Exemplars in Prometheus

In the Prometheus UI, enable the “Show Exemplars” toggle when running a histogram query:

histogram_quantile(0.99, rate(gateway_request_duration_bucket[5m]))

You will see small diamond markers on the graph. Hovering over one shows traceID=abc123. If you have Grafana connected (we’ll set this up in Chapter 10), clicking that diamond takes you straight to Jaeger.

Step 3: The Complete Pivot , A Worked Example

Let’s walk through a realistic debugging scenario using our two-service setup from Chapter 4, now enhanced with correlation.

The Symptom

Your Prometheus dashboard shows: P99 latency for GET /checkout jumped from 1.8s to 4.5s at 14:30.

Step 1: Metrics → Trace

You query the histogram and see exemplars on the high-latency data points. One exemplar shows traceID=3bf30133d2eeba199.....

You open Jaeger, paste the trace ID, and see the waterfall:

api-gateway: GET /checkout                 [4.3s]
  └── api-gateway: HTTP POST /orders       [4.1s]
       └── order-service: POST /orders     [4.0s]
            ├── check_inventory            [0.3s]
            └── insert_order_record        [3.5s] ← HERE

The insert_order_record span normally takes 1.5s but is now taking 3.5s. The Order Service’s database is the bottleneck.

Step 2: Trace → Logs

You click on the insert_order_record span and copy its trace_id. You search your logs:

jq 'select(.trace_id == "e4b2a9f3...") | select(.service == "order-service")' \
  order_service.log

Output:

{
  "timestamp": "2026-02-09T14:30:12.456Z",
  "level": "WARNING",
  "service": "order-service",
  "message": "Database connection pool exhausted, waiting for available connection",
  "trace_id": "e4b2a9f3...",
  "span_id": "b7c1d2e3...",
  "span_name": "insert_order_record"
}

Root cause found. The database connection pool is too small for the current traffic. The fix: increase max_connections or add connection pooling with PgBouncer.

The Full Loop

Metric spike (Prometheus)
    → Exemplar click (trace_id)
        → Trace waterfall (Jaeger)
            → Slow span identified (insert_order_record)
                → Log search by trace_id
                    → Root cause: "connection pool exhausted"
                        → Fix: increase pool size

This entire debugging workflow takes under 2 minutes. Without correlation, you would be grepping through logs for 30 minutes trying to match timestamps.

Structured Logging Best Practices for Correlation

Now that all three signals are connected, here are the rules for making correlation work reliably:

1. Always Log at Span Boundaries

with tracer.start_as_current_span("check_inventory") as span:
    logger.info("Starting inventory check", item=item)
    # ... work ...
    logger.info("Inventory check complete", available=True)

This ensures every span has at least entry/exit logs, making trace-to-log correlation useful.

2. Log Business Context, Not Just Technical Context

# Technical-only (limited debugging value)
logger.info("Request processed")

# Business context (immediately tells you what matters)
logger.info(
    "Order created",
    order_id="ord-789",
    item="widget",
    qty=2,
    processing_time_ms=1834,
)

3. Use Consistent Field Names Across Services

If the API Gateway logs request_id and the Order Service logs req_id, you can’t correlate across services. Define a shared schema:

# Shared field names (use across all services)
FIELD_NAMES = {
    "trace_id": "trace_id",       # From OTel context
    "user_id": "user_id",         # From auth middleware
    "order_id": "order_id",       # From business logic
    "request_id": "request_id",   # From X-Request-ID header
}

Looking Ahead: From Correlation to Wide Events

The Pivot Workflow we just built works well, but notice the friction: you’re still hopping between tools (Prometheus → Jaeger → log search). Each hop requires you to copy an ID, switch context, and search again.

In Chapter 5, we discussed the Observability 2.0 perspective where all telemetry lives in one unified source of truth. In that model, the Pivot Workflow isn’t hop-hop-hop between tools, it’s zoom in, zoom out within a single interface, because all your data is stored together.

The good news: everything we’ve built in this chapter is compatible with that direction. The trace_id correlation, the rich structured logs, the span attributes, these are the same building blocks that power wide, structured events. The difference is where the data is stored and how you query it, not how you instrument your code.

In the RAG capstone (Chapters 12–13), we’ll demonstrate building wide, context-rich events that carry 50+ attributes per request, the practical bridge between what we’re doing now and the O11y 2.0 approach.

Conclusion

What we built across the series so far:

With all three signals connected via trace_id, you now have a complete observability stack. In the next chapter, we address a critical production gap: what happens when your code catches exceptions gracefully but your dashboard shows everything is healthy, even though users are experiencing errors.

Frequently Asked Questions

What is the Pivot Workflow in observability?

The Pivot Workflow is a debugging pattern where you start from a metric alert (e.g., latency spike in Grafana), use exemplars to jump to the specific trace in Jaeger, then use the trace_id to find the exact log lines that explain the root cause. It connects all three observability pillars into one investigation flow.

How do you link metrics to traces in Grafana?

Use exemplars: when recording a metric, attach the current trace_id as an exemplar. Grafana renders these as clickable dots on your metric graphs. Clicking an exemplar opens the corresponding trace in Jaeger or Tempo.

What is trace_id and why is it important for correlation?

trace_id is a unique 128-bit identifier generated at the start of each request by OpenTelemetry. By injecting this ID into your logs (via a Loguru patcher) and your metrics (via exemplars), all three telemetry signals become queryable by the same key.

What is Observability 2.0 and how does it relate to correlation?

Observability 2.0 (coined by Charity Majors) proposes storing all telemetry in one unified data store as “wide events” with 50+ attributes per request. Instead of hopping between Prometheus, Jaeger and log search, you zoom in and out within a single interface. The correlation techniques in this chapter are the building blocks for that approach.

Do I need all three pillars or can I start with just logs?

You can start with just structured logs (Chapter 1), but you’ll miss latency visibility (traces) and aggregate health monitoring (metrics). The real power comes from connecting all three. Start with logs + traces, then add metrics when you need dashboards and alerting.

Resources

• OpenTelemetry Exemplars Specification
• Grafana Exemplars Documentation
• Loguru Documentation
• There Is Only One Key Difference Between O11y 1.0 and 2.0 (Charity Majors)
• A Practitioner’s Guide to Wide Events (Jeremy Morrell)
• Chapter 6: Code