uWSGI Segmentation Fault

[SabriArslan]

uWSGI Segmentation Fault

[SabriArslan]

thank you @gnufede ,

I'll try the fix with 2.7.1 and let you know if the issue still persists.

[sstoops]

@gnufede It appears unrelated to IAST, thus, I've opened a new issue #8648

[gnufede]

Hi @SabriArslan

Can you help me reproduce this? Do you have any DD_ prefixed environment variables (API key aside)?

For example, are you setting DD_IAST_ENABLED?

[SabriArslan]

hi @gnufede ,

Thank you for looking into this, unfortunately i have no steps to reproduce, it happens sporadically and the stack trace is the last thing we see in the logs, and the last log entry before that for the pid is about an access log which took 1948 msecs which has resulted with HTTP 200.

the only configured environment variables for DD are DD_ENV, DD_AGENT_HOST and DD_SERVICE. DD_IAST_ENABLED is not configured and yet we see IAST in the logs.

[sstoops]

I've no idea if it could be related, but we recently also began seeing consistent seg faults under gunicorn after upgrading google-api-core to 2.17.0. I've narrowed our issue down to this specific diff: googleapis/python-api-core@v2.16.2...v2.17.0

We're running ddtrace==2.6.0.

---

I've set DD_IAST_ENABLED=False to no effect.
I've set DD_APPSEC_ENABLED=False to no effect.

---

I've confirmed settings DD_PATCH_MODULES=grpc:false eliminates the issue.

---

I think I've narrowed it down to this specific commit.

[sstoops]

@SabriArslan Are you able to confirm if setting DD_PATCH_MODULES=grpc:false resolves your error?

Also, since google-api-python-client pulls in google-api-core, could you also try pinning google-api-core<2.17 to see if that resolves your error as well?

I'm curious, too, if you could post the output of pip freeze in your original issue. The package list you provided appears to be incomplete.

[SabriArslan]

Hi @sstoops ,

I didn't change any setting yet, and did not try DD_PATCH_MODULES=grpc:false since the error happening with uWSGI and I don't use gRPC directly, and no google-api is used in the last HTTP request before seg fault.

Although I still couldn't verify the intentions of the caller functions of the addresses mentioned in stack trace.

I've updated the pip freeze result, and the version of google-api-core is < 2.17

google-api-core==1.34.1
google-api-python-client==1.11.0

---

one more update, i have dd-trace-py enabled on two other similar apps with uWSGI with almost same packages and versions, one with v2.5.1 and other with same v2.4.0, and this behavior doesn't exists with both of them.

The one uses dd-trace v2.5.1 uses google-api-core v2.15.0 and the other uses google-api-core v2.14.0.

So given these I couldn't figure out why same seg fault is not happening with them, and don't know if I need to explicitly disable IAST and APPSEC.

[gnufede]

@SabriArslan @sstoops I haven't been able to reproduce it, but did a couple PRs to:

1. address a potential issue with the atexit handler, and
2. avoid importing the IAST native code at all costs if it's disabled, even raising an ImportError to be sure we catch it.

With this, hopefully the issue is solved, but don't hesitate to reopen if the problem persists.

[sstoops]

@gnufede Thanks for your swift work! Unfortunately, I'm still seeing segfaults.

• I've fully upgraded to ``ddtrace==2.7.1`
• I've tried each DD_IAST_ENABLED=False and DD_APPSEC_ENABLED=False and then both together to False.
• I've confirmed again that disabling ddtrace's grpc module resolves the issue.

This may not be useful to you, but in my previous debugging, I'd gotten as far as wrapping this await call.code() line in log statements and found it to be where the segfault occurs. I didn't dig much beyond this into the cpython module.

Since this is not my ticket, would you like me to open a new one with my details, or continue piggybacking this one?

[gnufede]

Since this is not my ticket, would you like me to open a new one with my details, or continue piggybacking this one?

@sstoops If you don't see anything related to IAST in your stacktrace, the segfault may have a different cause, hence worth opening a different issue. I mean a line like this:

/app/.venv/lib/python3.7/site-packages/ddtrace/appsec/_iast/_taint_tracking/_native.cpython-37m-x86_64-linux-gnu.so(+0x6973d) [0x7f7e3e17573d]

If that kind of line does appear, then we can use this ticket.

Thank you!