Overall plans for the project

[python-trio]

Overall plans for the project

[njsmith]

@tomchristie Yeah, we definitely need to figure out more about how hip/httpx relate. For me the biggest difference is the visions; as you wrote as you wrote in encode/httpx#522:

The big design goals of HTTPX have been to meet two features lacking in requests...

• HTTP/2 (and eventually HTTP/3) support.
• Async support.

[...]
Given that requests already provides a battle tested HTTP/1.1 client for the threaded concurrency masses, my inclination is that rather than trying to meet all possible use cases we should focus on httpx being a "the right tool for the right job" rather than "one size fits all".

OTOH, for hip, like I wrote above, the goal is explicitly to "meet all possible use cases" and be "one size fits all" :-). So that's a pretty clear difference in focus, and I think it's been somewhat in the background of a lot of conversations we've had, and it's great that we've all managed to articulate it clearly now.

We could talk about the motivation for our approach; for me it's closely related to what @pquentin wrote here:

can explain why I am working on the urllib3 fork since January 2018: I don't want to waste the huge amount of real world experience urllib3 has. It seems wasteful to have to relearn this by waiting for a slow trickle of bug reports.

For me, the number one challenge for building a production quality HTTP library is figuring out how handle all these edge cases that we don't even know about. This motivates us building on top of urllib3 (versus httpx's start-from-scratch strategy), but it also motivates the "one http client for everyone" philosophy, because that way we only have to address each weirdo edge case once, instead of every project having to rediscover them independently.

Of course the downside is that we inherit all of urllib3's technical debt, but by the time httpx is as mature as urllib3 I'm sure it will have plenty of technical debt too, because HTTP is just intrinsically messy :-) And technical debt is something we can tackle incrementally on our own schedule, instead of being stuck in reactive mode waiting for reports to come in.

I think the different visions also lead to different short-term priorities: we've certainly thought some about HTTP/2, but our primary focus for now is 100% on HTTP/1.1, because that's what we need for an MVP to replace the old libraries. OTOH, httpx has focused more on the shiny new features that older libraries lack. Both approaches make sense given the respective visions, but they're different.

So I have absolutely no animosity towards httpx; it just seems focused on solving a different problem than the one I'm interested in. If we can find ways to collaborate though, then that would be excellent; there's no point in reinventing the wheel. So let's think a bit about what that might look like.

Like you say, the high-level interface is probably where we have the most to learn from httpx.

One my concerns before was about httpx's layered design, with redirects/cookies/etc. handled as separate concerns through the "middleware"/dispatcher mechanism. As a design, it's super elegant! But HTTP is not very elegant :-). And in particular, when I sat down to study requests, my conclusion was that the core "high-level" features are redirects, cookies, retries, and auth handling – and that these are all tightly coupled in a way that doesn't fit well with a layered architecture like this. (In fact every CVE in requests's history has been because of trying to de-couple redirects/auth.) So my feeling has been that the "high level" concerns are best handled as a single piece of tightly-integrated code. But it sounds like you're going in the same direction, so I guess that's no longer an issue?

I don't think I agree about separating the high-level and low-level stuff into separate packages; but that doesn't actually affect too much. Regardless of whether hip starts growing higher-level features, we'll still have the lower-level networking stuff that httpx can use if that makes sense. And if hip does start growing its own higher-level features, we'll obviously look at httpx when doing that. So either way, I don't think any work will be wasted.

I do think that simply adopting httpx might be difficult, because I think hip's approach to handling sync+async has a pretty fundamental effect on API design, so we'll need to think those issues through while adopting httpx's ideas. (And fwiw delaying unasync for later seems risky to me, but I guess you know what you're doing...)

Some finer details: I think you'd also want a timeout argument on that interface, to support independently-set-per-request timeouts, but other stuff like requests/httpx's equivelents of verify/cert ought be configured on a per-pool instantiation basis, rather than per-request, since you don't know/care if you'll actually be connecting or reacquiring an existing connection

I think urllib3's idea for handling this is pretty elegant: it makes stuff like verify/cert part of the connection cache key, so you can have a single connection pool, but you won't reuse a connection unless these settings are compatible. (The exact details of urllib3's code have some unnecessary awkwardness here. But I like the core idea.) One consequence is that you can have a global connection pool so top-level requests like hip.get/httpx.get can re-use connections, but can also accept verify=... arguments. And then you can reduce a Session to just being a bundle of preconfigured kwargs, rather than a substantive object in its own right.

[njsmith]

• new name acquired
• sent request to github support to detach us from the urllib3/urllib3 "fork network"

[tomchristie]

Hey folks,

So, we've poured a huge amount of work into getting a "high level" API squared away with httpx. From my point of view we've put in the legwork there, and I'd love to see us working together, rather than replicating that work. 😌🤞

At least from my POV, what would make a huge amount of sense would be for hip to focus squarely on providing a rock solid low-level API, as @njsmith mentions...

class AbstractAsyncHTTPTransport(ABC):
    async def request(self, verb: bytes, url: bytes, headers: List[Tuple[bytes, bytes]], body: AsyncIterator[bytes]):
        ...
        return (status_code, status_message, headers, response_body)

No redirect handling. No cookie cleverness. No fancy HTTP retries. No multipart handling. No authentication or .netrc lookups. No response decoding. No "are we able to rewind the request body on redirection?" etc.etc. Just the raw basic "here's a connection pool, make a request with it" or "here's a proxy, make a request via it".

If hip provided that, we can plug httpx into it directly / potentially switch to it, since it's dispatch interface is exactly the same kind of abstraction layer.

Personally I think it'd probably be a good thing to have the low-level networking and the high-level model abstractions strictly seperated into independent nicely-scoped packages, since it helps keep different concerns strictly and clearly isolated.

Some finer details: I think you'd also want a timeout argument on that interface, to support independently-set-per-request timeouts, but other stuff like requests/httpx's equivelents of verify/cert ought be configured on a per-pool instantiation basis, rather than per-request, since you don't know/care if you'll actually be connecting or reacquiring an existing connection. There's probably also some thinking to do wrt. either making sure both levels use the same URL parsing.

We do still have an open question on httpx wrt. the best way to support both sync and async variants. Version 0.8 just down-scoped to only supporting async, since our bridging approach lead to a profileration of interface types, and sync vs. async classes both subclassing from base classes was leading to less comprehensible code. I think we're likely to consider an unasync approach instead, at a later date.

[tomchristie]

From my side I probably don't think that we've got any fundamental differences in goals, and the any failure to collaborate between our two sets of work would probably be down to a failure to communicate, rather than due to any technical blockers.

My opening salvo in #522 isn't actually where I ended up, and was triggered by us having an underlying approach to async+sync that wasn't working for the project, which we've now removed with 0.8. There's a bunch of places where I think we're on either the same, or similar pages:

• For sync support, we'll be switching over to the same code-gen style as hip. (We'll end up with exactly the same issue to resolve as your #168 wrt. how we name/scope our Sync vs. Async interfaces)
• We'll likely switch our concurrency backends to the same interface & implementation as hip.
• Having taken a look at the sync backend that hip uses, I think we're actually pretty likely to be providing a sync interface again sooner, rather than later.
• I think our HTTP/2 support ought to be optional, and off by default.

I don't think I agree about separating the high-level and low-level stuff into separate packages; but that doesn't actually affect too much.

Sure, so let me frame that differently...

We could trivally drop everything in httpx's dispatch implementation, and plug it into hip's PoolManager & ProxyManager classes instead. Where I think there's some really productive scope for collaboration would be if the hip team are able to clearly specify what interface they're aiming for at that layer, then I think we've got a huge slice of great work that we can contribute to providing all the model layers that sit above that.

it makes stuff like verify/cert part of the connection cache key, so you can have a single connection pool, but you won't reuse a connection unless these settings are compatible.

That's smart. Worth noting that it is actually functionally identical behavior to "verify/cert should be per-connection-pool settings, not per-request settings". If they're not sharing the same cache keys, then they are essentially just different connection pools.

[njsmith]

Oh, I guess we also need to rename the repo, which is a bit annoying. I think we'll need to break the github metadata that says that this is a "fork" of urllib3 upstream, because we're splitting off to become an independent project and if we don't do that then github will do things like assume that PRs intended for us should instead be directed back to urllib3/urllib3. And github doesn't actually offer any way to do that except to create a new repo. Transferring over the git history to a new repo is trivial, fortunately; the bigger problem is transferring over the issues. Maybe we should make a new hip repo, and then just go through the ~30 issues on here manually, and for each one click "Transfer issue" and send it to the new repo? I think that might be the best option that github offers.

Regarding orgs: it just occurred to me that the urllib3/ org might also make sense as a possible home? Dunno what y'all think.

[njsmith]

Oh, look what @pquentin found:

To detach the fork and turn it into a standalone repository on GitHub, contact GitHub Support or GitHub Premium Support. If the fork has forks of its own, let support know if the forks should move with your repository into a new network or remain in the current network.

https://help.github.com/en/github/setting-up-and-managing-your-github-profile/why-are-my-contributions-not-showing-up-on-my-profile#commit-was-made-in-a-fork

So I guess we'll do that, maybe after we get confirmation on ownership of our new name?

[pquentin]

Thank you for doing this. I wholeheartedly agree with 1/ the vision 2/ the governance with you as BDFL 3/ the next steps. But my agreement is also loosely held, in the sense that if someone feels strongly about some specific aspect I'd be happy to discuss to get consensus.

[sethmlarson]

I'd want you as BDFL and for the project to live under python-trio for the foreseeable future. I agree with everything else here in terms of direction. :)

[njsmith]

Hey Nathaniel,
Thank you for writing in to GitHub Support!
We've detached python-trio/hip and made it a standalone repository.

I think this means we all need to make new forks.