django/deps

Django Enhancement Proposals

Django

Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. Thanks for checking it out.

All documentation is in the "docs" directory and online at https://docs.djangoproject.com/en/stable/. If you're just getting started, here's how we recommend you read the docs:

• First, read docs/intro/install.txt for instructions on installing Django.
• Next, work through the tutorials in order (docs/intro/tutorial01.txt, docs/intro/tutorial02.txt, etc.).
• If you want to set up an actual deployment server, read docs/howto/deployment/index.txt for instructions.
• You'll probably want to read through the topical guides (in docs/topics) next; from there you can jump to the HOWTOs (in docs/howto) for specific problems, and check out the reference (docs/ref) for gory details.
• See docs/README for instructions on building an HTML version of the docs.

Docs are updated rigorously. If you find any problems in the docs, or think they should be clarified in any way, please take 30 seconds to fill out a ticket here: https://code.djangoproject.com/newticket

To get more help:

• Join the #django channel on irc.libera.chat. Lots of helpful people hang out there. Webchat is available.
• Join the django-users mailing list, or read the archives, at https://groups.google.com/group/django-users.
• Join the Django Discord community.
• Join the community on the Django Forum.

To contribute to Django:

• Check out https://docs.djangoproject.com/en/dev/internals/contributing/ for information about getting involved.

To run Django's test suite:

• Follow the instructions in the "Unit tests" section of docs/internals/contributing/writing-code/unit-tests.txt, published online at https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-unit-tests

Supporting the Development of Django

Django's development depends on your contributions.

If you depend on Django, remember to support the Django Software Foundation: https://www.djangoproject.com/fundraising/

deps's Issues

Feature request: allow writing a DEP in Markdown as well as rST

Reflecting on #28 and my first draft DEP in #84 – I’d like the ability to write a DEP in Markdown, as Jacob suggested on the forum: Updating our DEP process / DEP 1. There are a few reasons for this:

1. It’s how I write all of my content on the Django Forum, which is where I’ll usually flesh out technical discussions before going for a DEP.
2. It’s how I write content in GitHub comments on issues and pull requests, which is also how I flesh out technical discussions before going for a DEP.

I believe it would greatly simplify drafting DEPs if people were able to write them in a language that they’re more familiar with than reStructuredText. It also creates a significant amount of friction if we have to always convert syntax between a forum post and a DEP. We don’t have to rewrite existing DEPs, they’re fine as-is in rST. We would need to:

1. Update DEP 1 to allow writing in Markdown
2. Agree on how to do the DEP’s metadata fields in Markdown – either a table (identical output to reStructuredText), or YAML frontmatter
3. Add a template for Markdown DEPs (same content, different formatting)

DEP 1 on format

Here is what DEP 1 has to say about format for reference:

DEPs must be written in `reStructuredText <http://docutils.sourceforge.net/rst.html>`_ 
(the same format as Django's documentation). 

Each DEP should have the following parts:

[…]

#. A preamble -- a rST `field list <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#field-lists>`_ 
   containing metadata about the DEP, including the DEP number, the names of the
   various members of the `DEP team <#forming- the-team>`_, and so forth. See
   `DEP Metadata`_ below for specific details.

---

I’d also recommend dropping the requirement to hard-wrap lines at 80 characters for prose, for future Markdown and reStructuredText DEPs. It makes it harder to read content in the DEP’s source format, and creates silly diffs. If we want to retain specific line break guidelines, I would recommend Semantic Line Breaks.

Draft Proposal: Django Mergers with Terms

Hi All,

I want to at least outline an alternative proposal based on feedback from James’ PR ( #47 ). And I’d like to propose an actual process for selecting Mergers.

Jacob summarized the biggest problem I see with the current core team: “The problem (as I see it) is that we're currently also recognizing people for (long-)past contributions, giving them power and status that may not be appropriate any more.”

Roughly, I think it should have these features:

• There should still be recognition / “perceived status” associated with the mergers (not just bureaucratic), but only active contributors actually actually get that status.
• There should be a way for people to try out being a merger for a period of time (to encourage more people to get involved)
• You need to opt-in regularly to continue being a merger, though it’s really easy to do.
• There should be a natural time to stop being an active merger, but it should be really easy to become an active merger again. (Stepping down isn’t irreversible.) (Could help with burnout, too)
• There should be an activity requirement for mergers, but not an actual limit to the number of mergers as long as they’re active enough.
• Tech board is elected by django-developers

Proposal:

Mergers

• Mergers regularly review tickets, author patches, and/or review and merge patches
• They are expected to be contributing regularly on a weekly basis
• They only commit to being a Merger for one release cycle at a time (a term). Maybe the term ends about a month after a release of Django, just after the x.x.1 release.
• Mergers don’t automatically get push access to Django, but can ask for access at any time. The request gets voted on by the tech board. (so not exactly “merger means push access”, but similar to current/former core team) (also, other roles could have push access, like security team, releasers, etc, so push access isn’t exclusive to mergers, though that’s out of scope for this proposal)
• If you stop being a merger, you no longer get push access (or other permissions) to Django (again, unless you have some other role like security team, or board members can retain push access)

There are two types of mergers: Core Mergers and Guest Mergers.

Guest Mergers

• Anyone can apply/request at any time to be a Guest Merger by filling out a google form, though they should have some recent contribution history before applying.
• There should be a call for Guest Mergers at the beginning of a new term. (and anyone should also encourage anyone to apply)
• Applicants are reviewed (and possibly interviewed) by the Core Mergers and they make a recommendation to the tech board to approve or disapprove. If tech board approves, this gets announced on django-developers.
• It’s expected that guest mergers need oversight and help from the core mergers.
• Any core merger can revoke Guest Merger push access (or other access) at any time.
• At the end of their term, if a Guest Merger wants to continue being a Merger they need to re-apply.
• (There’s nothing wrong with just serving one term as Guest Merger and then stepping down. That’s welcomed. You can always apply be a Guest Merger again later.)
• You can’t serve more than two full consecutive terms as a Guest Merger (either Core Mergers need to invite you to be a Core Merger, or you need to take a break)

Core Mergers

• Core Mergers (as a group) can invite anyone who has served a full term as Guest Merger to be a Core Merger. This gets voted on by the tech board and if approved, announced to django-developers.
• Any former Core Merger, or former core team member can become a Core Merger by publicly requesting this on django-developers at any time. (There’s an expectation that they’ll then be fairly active until the end of that term.) This gets voted on by the tech board and if approved, announced on django-developers.
• At the end of their term, Core Mergers simply need to say something like “i’m interested in continuing to be a merger” publicly on django-developers. This also gets voted on by the tech board (tech board should confirm that they’ve been regularly reviewing tickets, patches, etc over their term.)
• There’s no limit to the number of consecutive terms a Core Merger can serve.

Though normally not needed, both DSF board and Tech board have the right make exceptions or to add or remove anyone from any of these roles at any time.

(I like the term “Merger” because it’s the only thing Mergers can do that other people can’t do. “Guest Reviewer” for example makes it sound like other people aren’t allowed to review, and “core team” doesn’t explain what they do as well. You can also be an “active” contributor without being active enough to qualify for the “merger” role. I think “Committer” is also an ok term.)

Also, just like currently, anyone (besides the patch writer) can mark a ticket/patch as “ready for check-in”, and mergers shouldn’t be committing their own stuff without review. I don’t see the need for any changes there. I don’t see any need to change anything about the release process either.

I also think we need better transparency as to who actually has permissions to what, security team and ops teams might also need terms, we somehow need to recognize “moderator” roles on django-users and irc channels, etc, but that’s all out of scope for this proposal.

DEP 9: Please do not ignore the async-only usecase

• For how long you have considered making django async-only?
• (Andrew Godwin) For about 2 minutes.

@andrewgodwin yes, I decided to write this after watching the presentation you gave at PyConAu 2019

summary

Please consider the usecase where a developer doesn't want to mix sync and async code, but rather has a sync application and a (separate) async application and wants them to have common parts (for example, definitions of the models).

the accepted approach (DEP 9)

The stated goal was to enable async in django, while preserving backwards compatibility. The accepted approach is to make some parts of django async. Partly because it allows you to have synchronous parts in your async application (that are easier to write), partly because django is big and its migration to async is better to be gradual (thus having a lot of sync parts at start).

To provide that, it is proposed to use something like asgiref.sync_to_async and asgiref.async_to_sync adapters. The important detail is that it pins all sync handlers for a given request to a single thread.

downsides

• sync parts exist = we still have the slowdown
• the implementation is complicated. We are going from sync to async and back, possibly multiple times. We are required to make sync parts to be executed in the same thread, that means we cannot just use the first available thread, hence more slowdown
• long-polling (one of the examples that will benefit from django 3.0) is quite exotic

async-only?

Well, it doesn't mean async-only, but it does mean that you cannot mix sync and async code. You can have gunicorn serving the sync handlers and twisted serving the async handlers, both using the same database, config, etc. Actually, you can use that approach today. Just the amount of code you can reuse between sync and async applications is close to 0. That can be improved: the async part of django is unwritten yet, and we have some degree of freedom.

orm

The models are mostly declarative, so conceptually there shouldn't be problems here. The same declarations of models can be used for sync and async code. Let migrations be synchronous only. Let django-admin be synchronous only.

middleware

Middleware will have to be implemented differently for sync and async cases. It would be nice if we could still reference same things in the configuration, like this:

MIDDLEWARE = [
    'common.SessionMiddleware',
    'common.AuthenticationMiddleware',
]

Here common.SessionMiddleware can be an empty class, an actual implementation being provided separately for sync and async cases.

DEP 9?

While I am personally for async-only django, that is configuration-compatible with traditional django, rather then for DEP 9, they do not exclude each other. I guess that a lot of implementation code of DEP 9 will be useful in async-only usecase anyway. I am just asking to add it as a legit usecase to avoid bad API design.

Also, if you know a proper place for this writing, please tell me.

New DEP's related to composite primary key and other related improvement of django based on previous works

compare and re assess https://gist.github.com/koniiiik/5408673 too

DEP 7, incomplete sentence/thought line 28

@jacobian

In DEP 7 at line 28:

28: ...community and core team that the chosen dependency...

The sentence just dangles off into space there...

Unfortunately, there's no way to make a line-comment without editing the document and I'm not sure how verbose/specific you want to be here or whether that was just a repeat of the prior sentence that got left hanging accidentally.

DEP 5: Needs more discussion of the effects of changing semantics

The new approach to middleware in DEP 5 looks great, making it simpler and more elegant.

One thing that wasn't clear to me was exactly why the old approach had the semantics it did. For example, was there a good reason for making a short-circuited response pass through all the layers (not just the ones above)? Are there any known middleware classes in the Django ecosystem that depend on that functionality? If so, how will this change impact them?

The DEP points out that the new design "allows more powerful idioms that aren't currently possible", but doesn't say whether it makes impossible some idioms that are currently possible. Perhaps it's the case that these foreclosed possibilities (for example, as above, modifying all responses even if the request was short-circuited by a higher layer) aren't useful and don't exist, but either way it would be great to see that explicitly acknowledged in the DEP.

"Getting to a draft DEP" process should be simplified

I would like to see the barrier to merging a draft DEP be lowered. In particular:

• A reference implementation should not be expected, but we could mention for example some public API designs would be advisable. I understand this increases the likelihood that the DEP may not be completely resolvable with the patterns discussed, but in practice they will change anyway. A decent discussion of the problem and the proposed architecture of the solution should be sufficient to get a general consensus from the technical board as to whether the idea is worth pursuing.
• A shepherd should be recommended but the implementation team should not be necessary. This allows us to potentially source an implementation team using DSF money or direct sponsorship of a DEP, but that it has already reached a general consensus of "we want to see this area explored and believe in the rough plan".
• We should require all DEPs to be peer reviewed to reach draft status, not give the core team a bye.
• We should ensure that there is a clean process for evolving a draft DEP after it has been merged as people want to work on it.

Comments welcome! I can probably look at drafting some changes to DEP0001. In particular the sections around the DEP submission workflow and submitting the draft need some work.

DEP201, exception raised when type conversion fails

Failure to perform a type conversion against a captured string should result in an Http404 exception being raised.

Reading the actual implementation django/django#7482 I’m wondering whether this should be Resolver404 instead. Type conversion failure seems symmetric to a regex non-match in the current style, which raises Resolver404. I understand Resolver404 is implemented as an Http404 subclass, but that seems like implementation detail (or maybe I’m wrong).

DEP 5: less streaming

I was confused by the use of the term upstream in this DEP:

That is, when a middleware short-circuits the upstream middleware and view by returning a response, that response will only pass through previous middleware in the list.

and

Similarly, a middleware that modifies the request on the way in and does pass it on upstream can be guaranteed that it will always see the response on the way back out.

If I'm reading this right, I would have used the term downstream. You process a request and pass it downstream to the next level, no?

Upon further reflection, a stream probably isn't a good metaphor for the bi-directional nature of middleware. Higher and lower might be clearer terms, given the presentation of middleware as a series of layers (e.g. the image in the documentation and the ordered list in settings).

Document the current DEP process

Can we have documentation for the DEP process as it stands at this point in time, combining what appears to be three different documents?

---

The README states:

See DEP 1 for details.

Said DEP refers to the Technical Board and to Core Developers, two groups which no longer exist as far as I know.

We have the same issue in Django’s contributor documentation:

Changes to this document require the use of the DEP process, with modifications described in DEP 0010.

DEP 10 is also no longer up-to-date with how Django operates. So – in order to write a DEP, I have to read at least DEP 1, 10, and 12.

Postgres 10 support

Postgres10 RC1 has been released recently with some exciting feature such as native partitioning. I have created this DEP to discuss about adding pg10 support to the django.

DEP 0210, some annotations

Instead of a type string I would use slug

The terminology slug is widely used in Django and everybody knows that it is a string without whitespace, slashes and other special characters, while string has a much wider scope of application.

Do not offer a type float

First, I don't see any use-case for floats, but more important, numbers of type float sometimes are localized using a comma as a decimal separator instead of the dot. This would allow two different URLs for the same Django view and this therefore can be very confusing.

Adopt and Adapt "Tango With Django" as an official tutorial

DEP n: Adopt and Adapt "Tango With Django" as an official tutorial.

Created: 2014-04-14
Author: James Rubino
Status: Draft

• Overview.
  Adopt and Adapt "Tango With Django" as an official tutorial.
• Rationale.
  Web Application programming is vastly more involved today then it has ever been to the web application programming style in the official "Poll" tutorial. What was once considered the domain of software on the local machine has moved onto the web. No longer is are web applications used only as interactive adhoc features. Instead, web applications have multiple parts that need to be integrated into the whole application. New developers find the current tutorial too narrow in scope to complete non trivial work.
• Implementation.
  The Tango with Django tutorial is complete and well written. It needs only approval from the Django Project.

Ref:
https://github.com/leifos/tango_with_django
https://github.com/leifos/tango_with_django/blob/master/LICENSE.txt
http://www.tangowithdjango.com/book/chapters/acknowledgements.html

Copyright

This document has been placed in the public domain.