Choosing content-Length vs chunked encoding for request bodies

[python-trio]

Choosing content-Length vs chunked encoding for request bodies

[pquentin]

Thanks for the analysis! I was not aware of this.

requests currently documents a simple rule for using chunked transfer encoding: is the body an iterator? https://requests.kennethreitz.org/en/master/user/advanced/#chunk-encoded-requests

Of course the reality is much more complicated. The prepare_body function decides on the headers, based on a number of factors (you've actually mentioned some of them), see https://github.com/psf/requests/blob/428f7a275914f60a8f1e76a7d69516d617433d30/requests/models.py#L453-L520

This makes sense for a higher-level API but we're not at this level yet, so for now I think we can look at the headers and if they're not there just perform a simple auto-detection to see if we have an iterator.

See urllib3/urllib3@3e586ef, we already look at existing headers, and already check if the body exists, so we only need to check if the body is an iterator to avoid chunking when it's not. requests uses this code:

is_stream = (
    hasattr(data, '__iter__')
    and not isinstance(data, (basestring, list, tuple, Mapping)))

And I guess that's enough for now? I don't think we want to provide the chunked parameter, because advanced users can already specify this differently, and well, urllib3 has too many parameters!

We need to make sure that this plays well with retries: if the first request uses chunked encoding, then we should continue to do so with other requests. This is the bug in urllib3 that prompted urllib3#1715 and urllib#1734.

[sethmlarson]

I'd say the way Requests handles this is the proper technique and has worked well for them. I'll note here that I'd like to change how multipart forms are encoded so a typical use-case for them can rely on Content-Length instead of chunking.