dd-trace auto-instrumentation not capturing error and stack trace details for HTTP Exception errors in FastAPI

[sats268842]

dd-trace auto-instrumentation not capturing error and stack trace details for HTTP Exception errors in FastAPI

[quiqueg]

@sats268842 Do you have any middleware installed that are catching exceptions and not re-raising them? I was running into the same issue, and it was due to this middleware: https://gist.github.com/nymous/f138c7f06062b7c43c060bf03759c29e, which does something like:

response = Response(status_code=500)
try:
    response = await call_next(request)
except Exception:
    # do something
    raise
finally:
    # do something else
    return response

The problem was specifically with the return response inside of the finally block—this swallows the exception, so ddtrace's TracingMiddleware never receives the exception and is therefore never able to add it to the trace.

I fixed it by moving the return response to outside of the finally block, and being OK with not returning my own Response(status_code=500) from this middleware. I'm handling it in an exception handler instead:

@app.exception_handler(Exception)
async def handle_exceptions(exc: Exception):
    return Response(status_code=500)

[emmettbutler]

Thanks for the report @sats268842. We'll see if we can put together a demo app that reproduces this behavior. If you have one you can share, that would be helpful.

[sats268842]

@emmettbutler Additionally, I've discovered that if you install both the auto instrumentation library injection and the ddtrace Python package, the tracing and metrics won't work. Also I will look into demo app, if time permits.

[sats268842]

@emmettbutler Also got workaround for this issue using below code

from ddtrace.opentelemetry import TracerProvider
from opentelemetry.trace import set_tracer_provider
set_tracer_provider(TracerProvider())


@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc):
    import traceback
    current_span = trace.get_current_span()
    if (current_span is not None) and (current_span.is_recording()):
        print(exc.__class__.__name__)
        exc_type, exc_value, exc_traceback = sys.exc_info()
        filename, line_number, function_name, code = traceback.extract_tb(exc_traceback)[-1]
        attributes = {
            "error.message": str(exc.detail),
            "error.kind": exc.__class__.__name__,
            "error.type": exc.__class__.__name__,
            "error.stack": traceback.format_exc(),
            "error.file": filename,
            "error.line": line_number,
            "error.function": function_name,
            "http.status_code": exc.status_code,
        }
        current_span.set_status(StatusCode.ERROR, str(exc.detail))
        current_span.set_attributes(attributes)

    return ORJSONResponse(({"detail": str(exc.detail)}), status_code=exc.status_code)