Unhandled incorrect header/body check

[synzen]

Unhandled incorrect header/body check

[danmactough]

@synzen This doesn't appear to be related to feedparser.

Make the following change to your example script:

- res.body.pipe(feedparser)
+ res.body.pipe(process.stdout)

You'll still see the pathological behavior -- no output, no errors. 🤷‍♂

[rdbcci]

Not sure what your asking, but if fetch returns encoded (gzip) you need to decode it before piping to feedparser. When I do this using a mikeal request, this feed parses just fine. Note, this feed is not encoded.

[synzen]

node-fetch doesn't auto-decode the body unless prompted by the user (which is why it's lightweight and faster from my own tests - like the browser's fetch API) - instead, I let feedparser take on the body. node-fetch works over 99.999% of the time (it works reliably, and is lighter than other libs!) until I encountered this issue, where feedparser failed to detect an invalid feed for a feed like this (had me scratching me head for days as to what was happening).

You can see the link above has invalid content-encoding headers when you open it in chrome and you see ERR_CONTENT_DECODING_FAILED

and the explanation for it: https://stackoverflow.com/questions/14039804/error-330-neterr-content-decoding-failed. TLDR; response says its gzipped, and it's actually not - browser is confused. I assume feedparser is confused here as well, but instead of giving an error, feedparser just hangs indefinitely

[rdbcci]

Not a node-fetch user, but parses fine as a mikeal request (feed header in this case indicates that is not encoded).

[synzen]

I get that request works, as you've said before, but I'm saying the problem is that feedparser isn't emitting any errors here, as I mentioned in my last sentence

[rdbcci]

Yea, just catching up to subtlety of your problem.

[rdbcci]

I just set myself up to use node-fetch. When I run it with your feed and {compress:false} option, it works.

[synzen]

Yes it will work in that case because then their server will send an uncompressed body, with no gzip in content-encoding (the main problem with their feed)

but the point here is that feedparser is not emitting an error here when I really think it should (if it's possible) - the issue is basically being swallowed up by the parser. I'm not trying to find a way to work with the feed, just trying to suggest that feedparser should emit an error so

1. Someone else doesn't spend hours scratching their heads why a feed may never resolve through feedparser
2. To handle it appropriately in error handling

[rdbcci]

I think Feedparser is just waiting for input. I think node-fetch maybe hung, not handling the decoding correctly in your offending feeds case. If that's the case Feedparser may not be the place for the fix.