7: Error Handling & Semantic Instrumentation

Proper error recording with OpenTelemetry means calling span.record_exception(), setting span.set_status(StatusCode.ERROR) and incrementing an error counter inside every except block. Without these three steps, your try/except blocks silently swallow failures: your dashboards show 100% success while users experience errors. This chapter shows the pattern that makes every failure visible.

In Chapter 6, we connected all three observability pillars and demonstrated the Pivot Workflow: starting from a metric spike, drilling into a trace, and landing on the exact log line. The system works beautifully, when errors are visible.

But there’s a silent problem hiding in most Python codebases: graceful error handling that makes failures invisible to your observability stack.

The Silent Failure Problem

Consider this perfectly reasonable Python code:

@app.post("/orders")
async def create_order(order: dict):
    try:
        result = await db.insert(order)
        return {"order_id": result.id, "status": "created"}
    except DatabaseError as e:
        logger.error(f"Database insert failed: {e}")
        return JSONResponse(
            status_code=500,
            content={"error": "Internal server error"},
        )

This looks correct. The exception is caught, logged, and a 500 response is returned. But from OpenTelemetry’s perspective:

• The span status is UNSET (not ERROR), because no unhandled exception propagated.
• The span has no record of the exception type, message, or stack trace.
• The trace waterfall in Jaeger shows a green (successful) span.
• Your error rate metric may not reflect this failure if it only counts unhandled exceptions.

The user got a 500 error, but your dashboard says everything is fine. This is the observability gap that graceful error handling creates.

The Fix: Semantic Error Recording

OpenTelemetry provides two methods to make errors visible on spans:

1. span.record_exception(e), Record What Happened

This adds an Event to the span containing the exception type, message, and full stack trace. The span appears with an exception marker in Jaeger.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("insert_order") as span:
    try:
        result = await db.insert(order)
    except DatabaseError as e:
        span.record_exception(e)  # Records the exception details on the span
        logger.error("Database insert failed", error=str(e))
        return JSONResponse(status_code=500, content={"error": "Internal error"})

After record_exception(), the span in Jaeger will show:

Events:
  exception
    exception.type: "DatabaseError"
    exception.message: "connection refused"
    exception.stacktrace: "Traceback (most recent call last):\n  File..."

2. span.set_status(ERROR) , Mark the Span as Failed

Recording the exception captures details, but the span is still technically UNSET. To mark it as failed in the UI (red instead of green), you explicitly set the status:

from opentelemetry.trace import StatusCode

with tracer.start_as_current_span("insert_order") as span:
    try:
        result = await db.insert(order)
    except DatabaseError as e:
        span.record_exception(e)
        span.set_status(StatusCode.ERROR, str(e))  # Marks span as ERROR
        logger.error("Database insert failed", error=str(e))
        return JSONResponse(status_code=500, content={"error": "Internal error"})

3. span.set_attribute("error.type", ...) , The Standard Error Classifier

The official OTel semantic conventions require a third piece: the standard error.type attribute on the span itself. This is what backends use to group, filter and alert by error category:

span.set_attribute("error.type", type(e).__name__)  # e.g. "DatabaseError"

The exception.escaped Parameter

record_exception() takes an optional escaped boolean. Set it to True when the exception is re-raised after recording, False (the default) when you catch and return a response:

span.record_exception(e, escaped=False)  # caught, returning a response
span.record_exception(e, escaped=True)   # re-raised after recording

The complete trio for any failed operation:

span.record_exception(e, escaped=False)         # or escaped=True if re-raising
span.set_attribute("error.type", type(e).__name__)
span.set_status(StatusCode.ERROR, str(e))

The Difference in Jaeger

The Three Error Patterns

In production Python code, errors fall into three patterns. Each requires a different instrumentation approach.

Pattern 1: Let It Propagate (Unhandled)

When you don’t catch the exception, OTel auto-instrumentation handles it automatically. The FastAPI instrumentor catches the unhandled exception, records it on the span, and sets the status to ERROR.

@app.post("/orders")
async def create_order(order: dict):
    # If this throws, OTel auto-instrumentation catches it
    result = await db.insert(order)
    return {"order_id": result.id}

When to use: For truly unexpected errors where you want the request to fail loudly. No additional OTel code needed.

Pattern 2: Catch and Record (Handled but Failed)

When you catch the exception for graceful degradation but the operation did fail. This is the most common pattern and the one most developers get wrong.

@app.post("/orders")
async def create_order(order: dict):
    with tracer.start_as_current_span("insert_order") as span:
        try:
            result = await db.insert(order)
            return {"order_id": result.id}
        except DatabaseError as e:
            # 1. Record the exception on the span
            span.record_exception(e, escaped=False)
            # 2. Set the standard error classifier
            span.set_attribute("error.type", type(e).__name__)
            # 3. Mark the span as failed
            span.set_status(StatusCode.ERROR, str(e))
            # 4. Log for correlation
            logger.error("Order insert failed", error=str(e))
            # 5. Return graceful response
            return JSONResponse(status_code=500, content={"error": "Try again later"})

When to use: For expected errors (DB timeouts, API rate limits, validation failures) where you return a meaningful response but the operation failed.

One choice worth being explicit about: this pattern returns a JSONResponse instead of raiseing. If you raise after set_status(ERROR), the parent span (the FastAPI route span) also sees the propagated error. If you return a structured response, only the child span is marked red. Both are valid. Choose raise when you want the error visible at every level of the trace hierarchy.

Pattern 3: Catch and Continue (Handled and Recovered)

When you catch the exception and successfully recover, for example, falling back to a cache or a default value. The span should NOT be marked as ERROR.

@app.get("/product/{product_id}")
async def get_product(product_id: str):
    with tracer.start_as_current_span("fetch_product") as span:
        try:
            product = await db.get_product(product_id)
        except DatabaseError as e:
            # Record the exception as an Event (for visibility)
            span.record_exception(e)
            # But DON'T set status to ERROR ,  we recovered
            span.add_event("cache_fallback", {"reason": str(e)})
            logger.warning("DB failed, falling back to cache", error=str(e))
            product = await cache.get_product(product_id)

        span.set_attribute("product.source", "db" if product else "cache")
        return product

When to use: For errors where a fallback succeeds. The span remains green (OK) but carries a record of the exception and the fallback event, so you know it happened.

Adding Business Context with Span Attributes

Error recording tells you what broke. But to debug efficiently, you also need to know what was being processed when it broke. This is where semantic attributes shine.

The Bare Span (Hard to Debug)

Span: insert_order [ERROR]
  exception.type: DatabaseError
  exception.message: "connection refused"

You know the database call failed. But which order? Which user? How big was the payload?

The Rich Span (Easy to Debug)

with tracer.start_as_current_span("insert_order") as span:
    # Set business context BEFORE the operation
    span.set_attribute("order.item", order.get("item"))
    span.set_attribute("order.qty", order.get("qty"))
    span.set_attribute("order.total_usd", calculate_total(order))
    span.set_attribute("user.tier", get_user_tier(request))

    try:
        result = await db.insert(order)
        span.set_attribute("order.id", result.id)
    except DatabaseError as e:
        span.record_exception(e, escaped=True)  # re-raised, so escaped=True
        span.set_attribute("error.type", type(e).__name__)
        span.set_status(StatusCode.ERROR, str(e))
        raise

Now the span in Jaeger shows:

Span: insert_order [ERROR]
  order.item: "widget"
  order.qty: 2
  order.total_usd: 49.98
  user.tier: "premium"
  exception.type: DatabaseError
  exception.message: "connection refused"

You immediately know: a premium user’s $49.98 order for 2 widgets failed because the DB connection was refused. No log searching required.

The Wide Event Mindset

This “make spans wider” approach is exactly the principle behind wide structured events that we mentioned in Chapters 5 and 6. The more context you pack into a single span, business fields, timing data, error details, user context, the more powerful your debugging becomes.

In mature production systems, a single span might carry 50–200 attributes. Every attribute you add is essentially free (it’s just a few more bytes on the same event), unlike metrics where every new dimension multiplies your storage cost. Think of it this way: don’t ask “is this worth adding?”, ask “might this ever be useful for debugging?” If yes, add it.

Instrumenting External Service Calls

In production, your service calls external APIs: payment gateways, notification services, third-party data providers. These calls are common failure points and benefit greatly from semantic instrumentation.

Here’s a pattern for wrapping any external call:

async def call_external_api(
    tracer: trace.Tracer,
    service_name: str,
    operation: str,
    call_func,
    *args,
    **kwargs,
):
    """Wrap an external API call with full observability."""
    with tracer.start_as_current_span(
        f"external.{service_name}.{operation}",
        kind=trace.SpanKind.CLIENT,
    ) as span:
        span.set_attribute("external.service", service_name)
        span.set_attribute("external.operation", operation)

        start = time.perf_counter()
        try:
            result = await call_func(*args, **kwargs)
            span.set_attribute("external.status", "success")
            return result
        except Exception as e:
            span.record_exception(e, escaped=True)
            span.set_attribute("error.type", type(e).__name__)  # standard semconv attribute
            span.set_status(StatusCode.ERROR, str(e))
            span.set_attribute("external.status", "error")
            raise
        finally:
            duration_ms = (time.perf_counter() - start) * 1000
            span.set_attribute("external.duration_ms", duration_ms)

Usage:

# Calling a payment API
result = await call_external_api(
    tracer,
    service_name="stripe",
    operation="charge",
    call_func=stripe_client.charge,
    amount=4998,
    currency="usd",
)

This produces a span like:

Span: external.stripe.charge [OK or ERROR]
  external.service: "stripe"
  external.operation: "charge"
  external.status: "success"
  external.duration_ms: 234.5

Combining Error Recording with Metrics

In Chapter 5, we created error counters. Now we can make them more precise by recording errors at the span level AND incrementing the metric:

# From Chapter 5
error_counter = meter.create_counter(
    name="orders.errors.total",
    description="Total order processing errors",
    unit="1",
)

with tracer.start_as_current_span("insert_order") as span:
    span.set_attribute("order.item", item)
    try:
        result = await db.insert(order)
    except DatabaseError as e:
        # Trace: record the exception
        span.record_exception(e, escaped=True)
        span.set_attribute("error.type", type(e).__name__)  # span attribute (standard semconv)
        span.set_status(StatusCode.ERROR, str(e))

        # Metric: use the same error.type value for consistency
        error_counter.add(1, {
            "error.type": type(e).__name__,
            "operation": "insert_order",
        })

        # Log: structured error message
        logger.error(
            "Order insert failed",
            error_type=type(e).__name__,
            error_message=str(e),
            item=item,
        )
        raise

This gives you the complete picture:

• Metric: error count goes up (visible on dashboard, triggers alerts).
• Trace: span is marked red with full stack trace (visible in Jaeger).
• Log: structured error with context (searchable by trace_id).

All three signals fire from the same except block, all carrying the same trace_id.

Testing Your Error Recording

The patterns above mean nothing if the spans aren’t actually being marked correctly. OTel’s InMemorySpanExporter lets you write unit tests without Jaeger running:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import StatusCode

def test_db_error_marks_span_failed():
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    tracer = provider.get_tracer("test")

    with tracer.start_as_current_span("insert_order") as span:
        try:
            raise DatabaseError("connection refused")
        except DatabaseError as e:
            span.record_exception(e, escaped=False)
            span.set_attribute("error.type", type(e).__name__)
            span.set_status(StatusCode.ERROR, str(e))

    spans = exporter.get_finished_spans()
    assert spans[0].status.status_code == StatusCode.ERROR
    assert spans[0].status.description == "connection refused"
    assert spans[0].attributes["error.type"] == "DatabaseError"
    assert any(ev.name == "exception" for ev in spans[0].events)

You can validate every pattern in this chapter using this approach, without a running backend.

Error Handling Checklist

Use this checklist when instrumenting any new operation:

Conclusion

What we built across the series so far:

We’ve now completed the entire application-layer instrumentation. Your code produces rich, correlated telemetry across all three pillars with proper error recording and business context.

In the next chapter, we move to the Infrastructure Layer: setting up the OpenTelemetry Collector to manage, process, and route your telemetry data at scale, a critical step for production deployments.

Frequently Asked Questions

Why does my dashboard show 100% success when users report errors?

Your try/except blocks are catching exceptions and returning fallback responses without recording the error on the OpenTelemetry span. The span completes with status OK, so your metrics and traces show no failures. Always call span.record_exception() and span.set_status(StatusCode.ERROR) inside except blocks.

What is the difference between span.record_exception() and span.set_status()?

record_exception() adds the full stack trace as a span event (visible in Jaeger’s event list). set_status(StatusCode.ERROR) marks the span red in the trace UI and makes it filterable. You need both: one for diagnostics and one for alerting.

Should I record business errors like “insufficient funds” as span errors?

Yes, if the error affects user experience. Use span attributes to distinguish error categories (e.g., error.type = "business" vs error.type = "infrastructure"). This lets you create separate alert rules for business vs technical failures.

How do I avoid duplicate error recording across middleware and business logic?

Use a consistent pattern: business logic records the exception and sets span status, middleware only records errors that weren’t already caught. Check span.status.status_code before setting it again to avoid overwriting a more specific error message.

Does recording exceptions on spans increase trace storage significantly?

Stack traces add roughly 1-5 KB per exception event. Since errors are typically a small percentage of total spans, the storage impact is minimal. If storage is a concern, use the tail sampling processor (Chapter 9) to keep 100% of error traces while sampling successes.

Resources

• OpenTelemetry Span Status Specification
• OpenTelemetry Recording Exceptions
• Recording Errors: OTel Semantic Conventions
• Exceptions on Spans: Semantic Conventions
• Span Events vs. Span Status
• OTel Python: Manual Instrumentation
• opentelemetry-semantic-conventions package
• Chapter 7: Code