Utf-8 getting decoded 2nd time

[jangernert]

Utf-8 getting decoded 2nd time

[markpritchard]

Great find and analysis @jangernert ! Very similar to issues I've run into in the past (e.g. Java servlet containers + URL parsing libraries both decoding parameters etc).

My initial thought is similar to yours - retrieving feeds over HTTP/S is just one storage/transport mechanism (albeit the most popular by far). Other sources might be a local database or file system, or some other mechanism that is "dumb" and does not provide transparent decoding like reqwest does.

So my preference would be to clearly state we expect the content to be in its original, encoded form (i.e. raw bytes) and feed-rs handles the decoding:

• it keeps the decoding approach consistent across all sources of content
• its convenient for users of the library (just provide raw bytes)

If it proves problematic in the future, feed-rs could be extended to not decode as required.

I might push a change that makes it clear in the documentation + update testurls to use .bytes() too as a basic reference implementation. Sound reasonable?

[jangernert]

Yes, I think a clear description in the docs of parse and fixing testurls would be an acceptable solution.