support raw UTF8 paths

[seanmonstar]

support raw UTF8 paths

[seanmonstar]

I feel like the bytes that httparse initially allowed were from an RFC, but I no longer remember which one it was (7230? URIs? WHATWG Fetch?)

[lovasoa]

But if actual user agents, including safari, do send requests with utf8 paths, shouldn't we support them in server-side libraries, even if there is an RFC saying they are invalid ?

[seanmonstar]

The general answer to that question is not always easy. There's sometimes good reasons to prevent them.

In this case, it could be fine to make the parser more relaxed. I was trying to remember what said to make it strict originally, so I could read if they included any comments as to why.

[JakeChampion]

I believe this is coming from rfc3986 section 3.3 and the definition for a path segment not allowing non-us-ascii characters

In particular it's the pchar definition, where characters outside the allowed set need to be percent encoded.

   path          = path-abempty    ; begins with "/" or is empty
                 / path-absolute   ; begins with "/" but not "//"
                 / path-noscheme   ; begins with a non-colon segment
                 / path-rootless   ; begins with a segment
                 / path-empty      ; zero characters

   path-abempty  = *( "/" segment )
   path-absolute = "/" [ segment-nz *( "/" segment ) ]
   path-noscheme = segment-nz-nc *( "/" segment )
   path-rootless = segment-nz *( "/" segment )
   path-empty    = 0<pchar>

   segment       = *pchar
   segment-nz    = 1*pchar
   segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
                 ; non-zero-length segment without any colon ":"

   pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"


   pct-encoded   = "%" HEXDIG HEXDIG

   unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
   sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"
                 / "*" / "+" / "," / ";" / "="

The whole abnf can be view at either https://abnf-uri.edgecompute.app/ or https://www.rfc-editor.org/rfc/rfc3986.html#appendix-A

right now applications which use httparse can be sure that it returns a valid uri according to rfc3986

perhaps a feature flag could be added to allow utf8 (rfc3986-non-compliant) path support, so that applications can choose whether they want httparse to be strict or relaxed?

[lovasoa]

But if there is a flag, is it realistic to think people will care enough to activate it ? I only learned about this after getting bitten by it, and httparse is probably a sub-sub-dependency for most people who just want to handle http requests made by web browsers. I think it would be great if httparse were able to parse web requests made by Safari by default, and if the update just silently fixed this issue that is currently present by transitivity almost everywhere in the rust server framework ecosystem.