snok/django-guid

Inject an ID into every log message from a Django request. ASGI compatible, integrates with Sentry, and works with Celery

Django GUID

Now with ASGI support!

---

Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words, all logs connected to a request now has a unique ID attached to it, making debugging simple.

Which version of Django GUID you should use depends on your Django version and whether you run ASGI or WSGI servers. To determine which Django-GUID version you should use, please see the table below.

Django version	Django-GUID version
3.1.1 or above	3.x.x - ASGI and WSGI
3.0.0 - 3.1.0	2.x.x - Only WSGI
2.2.x	2.x.x - Only WSGI

Django GUID >= 3.0.0 uses ContextVar to store and access the GUID. Previous versions stored the GUID to an object, making it accessible by using the ID of the current thread. (Version 2 of Django GUID is supported until Django2.2 LTS is over.)

---

Resources:

• Free software: MIT License
• Documentation: https://django-guid.readthedocs.io
• Homepage: https://github.com/snok/django-guid

---

Examples

Log output with a GUID:

INFO ... [773fa6885e03493498077a273d1b7f2d] project.views This is a DRF view log, and should have a GUID.
WARNING ... [773fa6885e03493498077a273d1b7f2d] project.services.file Some warning in a function
INFO ... [0d1c3919e46e4cd2b2f4ac9a187a8ea1] project.views This is a DRF view log, and should have a GUID.
INFO ... [99d44111e9174c5a9494275aa7f28858] project.views This is a DRF view log, and should have a GUID.
WARNING ... [0d1c3919e46e4cd2b2f4ac9a187a8ea1] project.services.file Some warning in a function
WARNING ... [99d44111e9174c5a9494275aa7f28858] project.services.file Some warning in a function

Log output without a GUID:

INFO ... project.views This is a DRF view log, and should have a GUID.
WARNING ... project.services.file Some warning in a function
INFO ... project.views This is a DRF view log, and should have a GUID.
INFO ... project.views This is a DRF view log, and should have a GUID.
WARNING ... project.services.file Some warning in a function
WARNING ... project.services.file Some warning in a function

See the documentation for more examples.

Installation

Install using pip:

pip install django-guid

Settings

Package settings are added in your settings.py:

DJANGO_GUID = {
    'GUID_HEADER_NAME': 'Correlation-ID',
    'VALIDATE_GUID': True,
    'RETURN_HEADER': True,
    'EXPOSE_HEADER': True,
    'INTEGRATIONS': [],
    'IGNORE_URLS': [],
    'UUID_LENGTH': 32,
}

Optional Parameters

• GUID_HEADER_NAME

  The name of the GUID to look for in a header in an incoming request. Remember that it's case insensitive.

  Default: Correlation-ID

• VALIDATE_GUID

  Whether the GUID_HEADER_NAME should be validated or not. If the GUID sent to through the header is not a valid GUID (uuid.uuid4).

  Default: True

• RETURN_HEADER

  Whether to return the GUID (Correlation-ID) as a header in the response or not. It will have the same name as the GUID_HEADER_NAME setting.

  Default: True

• EXPOSE_HEADER

  Whether to return Access-Control-Expose-Headers for the GUID header if RETURN_HEADER is True, has no effect if RETURN_HEADER is False. This is allows the JavaScript Fetch API to access the header when CORS is enabled.

  Default: True

• INTEGRATIONS

  Whether to enable any custom or available integrations with django_guid. As an example, using SentryIntegration() as an integration would set Sentry's transaction_id to match the GUID used by the middleware.

  Default: []

• IGNORE_URLS

  URL endpoints where the middleware will be disabled. You can put your health check endpoints here.

  Default: []

• UUID_LENGTH

  Lets you optionally trim the length of the package generated UUIDs.

  Default: 32

Configuration

Once settings have set up, add the following to your projects' settings.py:

1. Installed Apps

Add django_guid to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'django_guid',
]

2. Middleware

Add the django_guid.middleware.guid_middleware to your MIDDLEWARE:

MIDDLEWARE = [
    'django_guid.middleware.guid_middleware',
    ...
 ]

It is recommended that you add the middleware at the top, so that the remaining middleware loggers include the requests GUID.

3. Logging Configuration

Add django_guid.log_filters.CorrelationId as a filter in your LOGGING configuration:

LOGGING = {
    ...
    'filters': {
        'correlation_id': {
            '()': 'django_guid.log_filters.CorrelationId'
        }
    }
}

Put that filter in your handler:

LOGGING = {
    ...
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'medium',
            'filters': ['correlation_id'],
        }
    }
}

And make sure to add the new correlation_id filter to one or all of your formatters:

LOGGING = {
    ...
    'formatters': {
        'medium': {
            'format': '%(levelname)s %(asctime)s [%(correlation_id)s] %(name)s %(message)s'
        }
    }
}

If these settings were confusing, please have a look in the demo projects' settings.py file for a complete example.

4. Django GUID Logger (Optional)

If you wish to see the Django GUID middleware outputs, you may configure a logger for the module. Simply add django_guid to your loggers in the project, like in the example below:

LOGGING = {
    ...
    'loggers': {
        'django_guid': {
            'handlers': ['console', 'logstash'],
            'level': 'WARNING',
            'propagate': False,
        }
    }
}

This is especially useful when implementing the package, if you plan to pass existing GUIDs to the middleware, as misconfigured GUIDs will not raise exceptions, but will generate warning logs.

django-guid's Issues

Port celery support for django-guid 2.2

Flask support

I haven't touched Flask in a while, but I suspect django-guid is compatible with Flask, either out of the box, or with some simple modifications. I suggest we investigate what it would take to integrate with Flask, and add Flask-setup-docs.

BSD-3-Clause, not BSD-4-Clause

Hi,

LICENSE contains a standard BSD license, but in pyproject.toml it says BSD-4-Clause, which confuses our automatic license checker.

If I submit a PR to fix it will it be merged?

Thanks,
Aur

Full UUID as Text

Why we don't have the option of using the full UUID 776336d4-545e-43e1-bd3b-017794af48e9 instead of the hex output? 776336d4545e43e1bd3b017794af48e9

Pass the GUID of a request to the Celery workers

Celery tasks triggered by a request will not have the GUID (Correlation-ID) attached to it. Implement a solution to inject the GUID into the Celery logs.

For Celery tasks that are not triggered by a request, use the task-id instead as a GUID.

Provide error page views that add the guid to the HTML output

I would like an easy way to override the Django error page views so that the request guid is included as part of the HTML. This is useful so that customers can easily send the guid to a support team to get faster help.

Pipeline for test and codecov

Related to #2

Read the docs documentation

Need read the docs documentation

Elastic APM Integration

The APM-docs describe how to add a transaction id to APM events.

I think this would be a good candidate for a new integration.

IGNORE_URLS regex support

It would be great to add regex support for the IGNORE_URLS config.
For example, I would like to ignore UUID generation for requests for /static/* artifacts.

Add the option to send the GUID/Correlation-ID as an outgoing header

Describe the solution you'd like
Add the option to add the GUID/Correlation-ID to the response header.
Should be a setting.

Additional context
Allowing users or sites that are using the Django app to use the Correlation-ID to report issues.
Example:

• User sends a request
• django-guid adds a Correlation-ID to that request
• Somewhere in the request something unexpected goes wrong, the user gets a !2xx status code back.
• The front end prints the Correlation-ID to the user.
• The user can use that specific ID when contacting the developers.

v4 planning

• Make ID generator and validator generic, implement transformer (mirror asgi-correlation-id)
• Split celery functionality into a separate package
• Split sentry functionality into a separate package
• Create extras for both
• Update license
• Update test project to handle Django deprecation warnings
• Update to Poetry v1.2
• Update classifiers
• Rework docs (drop sphinx and just have readmes per module?)
• Remove get_guid and set_guid
• Create example projects for each feature

Essentially we want to de-bloat the core a little, making it possible to install the core middleware without anything else. It's possible to install the middleware today, without adding celery as a dependency, but the celery code is still contained in the wheel.

There's also some Sentry + Celery logic to consider @JonasKs. Do we put that in its own third package? Something to think about 🤔

Write tests

Test everything

Example project

Add example project

Best way to pass on the correlation-id to child threads?

I am running a django application using gunicorn server, and there are some child threads that will be created by the parent thread. I am getting the guid (contextvar) using get_guid method in the parent thread, passing it as an argument to the child threads, and setting the guid contextvar in the child threads using set_guid method (both of these methods are present in django_guid.api module). Only then the guid is visible in the log messages.

Is this the best way to pass on the correlation-ids to child threads? Or is there an alternate better way?

Implement black and flake8

Implement black and flake8.

Support Django 5

I tried bumping to Django 5 and am being told that django-guid does not support it.

django-guid (3.3.1) which depends on django (>=3.1.1,<5.0)

ASGI support for later Django versions (works with Django 3.0.x)

With ASGI being stable and more async support to Django is coming up, this package eventually have to change it's design to be ASGI compliant.

• ASGI by protocol can handle multiple requests on one thread. Every time await is used, the control of the loop will be handed back to the event loop, queuing the resumption of the function. The next function will run until it calls on an await, and then the entire thing repeats. This means that a thread will by design be able to handle multiple requests.

• Most of Django do not support awaitat the moment, so django-guid will work on ASGI today. The way it works right now is that when it gets an ASGI request on the event loop, it dispatches it to a thread pool where each thread again handles one request at the time.

Essentially this means that in the future, in order to keep this package living we have to find another ID to use. Luckily the Django people has been kind to us and implemented a replacement for us.

I will look into implementing this ASAP.

Thanks to knbk on #freenode.

Docs for Celery Integration and release

• Celery Integration docs
• Release new version to PyPi

IGNORE_URL setting to opt-out for some URLs

I have a few requests that I don't want to see in the logs

eg. a load balancer sends a "ping" once in a while to see the server is up
the logs are spammed with generated new GUID messages:

django_guid - Header `Correlation-ID` was not found in the incoming request. Generated new GUID: cf306e6e90bc42238f025d8d3fddc8ac

for this request I would prefer to opt out of the middleware - a decorator would work nicely

@no_guid
def ping(request):
    return HttpResponse("Ok\n")

Edit:
It turns out that to use a decorator for opt-out would require to use the process_view hook, which runs after all the middelwares __call__ methods have been run. If the GUID would only be added at that point, it wouldn't be useful for debugging of other middleware, which is not a good idea.
For that reason, instead of a decorator, the plan is to use a "black list" of URLs to ignore. I edited the Issue title to reflect this change.

[BUG] Can't add/override HTTP headers in other middlewares

Describe the bug
Putting the middleware on top of middleware list — as suggested by doc — doesn't allow next middlewares to potentially add/override HTTP headers through:

request.META['HTTP_CUSTOM_HEADER'] = "CUSTOM VALUE"

This is due to using django_guid.utils.get_id_from_header() in django_guid.middleware.process_incoming_request(), which triggers an earlier immutable HttpHeader object build, so any add/override headers intent has no impact.
Would be better to use request.META to get/set headers.

To Reproduce

1. Create a middleware

   def foo_middleware(get_response):
       def middleware(request):
           request.META["HTTP_X_FOO"] = "foo"
           return get_response(request)
   
       return middleware

2. Add middleware after django_guid

   MIDDLEWARE = [
       "django_guid.middleware.guid_middleware",
       "foo.foo_middleware",
   ]

3. Try to access the header in a view

   @api_view(["GET"])
   def test_view(request):
       assert request.headers.get("X-Foo") == "foo"

4. Now retry by removing "django_guid.middleware.guid_middleware" from MIDDLEWARE.

Change from f'strings' to %strings

Thanks to @BBT#8008 at the Django Discord for reporting this.

Three issues of f'strings' in logging:

1. If interpolation fails, code will break because it's not handled by the logging module
2. Tools like Sentry aggregate logs based on the log template string, interpolating before that will make them show up as different errors/messages, while they are actually the same
3. Interpolation is done even if the message is not to be logged

So, changing from

logger.info(f'{given_guid} is not a valid GUID. New GUID is {new_guid}')

to

logger.info("%s is not a valid GUID. New GUID is %s", given_guid, new_guid)

Will ensure the opposites of the points above:

1. If the interpolation fails, code will not break because it's handled by the logging module
2. Tools like Sentry will behave correctly
3. Interpolation is only done if the message is to be logged

[BUG] Overriding of "Access-Control-Expose-Headers" when

Describe the bug
It is sometimes necessary to modify the Access-Control-Expose-Headers within the request cycle. As an example, a developer might need to set the Content-Disposition and Content-Type when returning a file from an API so that it has a name, for e.g.:

Content-Disposition: attachment; filename="myfile.csv"
Content-Type:  text/csv; charset=utf-8
Correlation-ID: b038c7a8662f4d21962c80ef894d0946
Access-Control-Expose-Headers: Content-Type Content-Disposition Correlation-ID

We came across a bug in production with the way Django-GUID implements the EXPOSE_HEADER setting, in that on the outgoing request processing, it overrides any setting set by the user in the request flow, because it replaces the already set Access-Control-Expose-Headers rather than being additive.

To Reproduce

• Create a view in Django
• Set "Access-Control-Expose-Headers" to some value within the view
• Set Django GUID setting "EXPOSE_HEADER" to "True"
• Make an API request to the view, look at the request, and see that the developer set value has been overwritten.

Full stack trace
N/A

Make correlation-header a django setting

At the moment the Correlation-ID header is hardcoded. Should be a django setting.

Integration with Dramatiq

Would it be possible to add integration with Dramatiq message queue?

If not, do you guys have any pointers how I can send a correlation ID into a dramatiq task so I can track how each task spawned by a given request is handled?

[BUG] - Missing CHANGELOG.rst from packaged tar.gz

Describe the bug
setup.py includes a dependency on CHANGELOG.rst which is not included in the .tar.gz packages

To Reproduce
Retrieve django-guid-.tar.gz and try to run setup.py

I've tested this on 1.1.1, 2.0.0 2.1.0 & 2.20

Full stack trace
[root@dev-rhel7-acme-python-django2-guid django-guid-2.2.0]# /opt/acme/bin/python3 setup.py install
Traceback (most recent call last):
File "setup.py", line 7, in
with open('CHANGELOG.rst') as changelog_file:
FileNotFoundError: [Errno 2] No such file or directory: 'CHANGELOG.rst'