feed-rs fails to parse a feed

[ayrat555]

feed-rs fails to parse a feed

[markpritchard]

Hi,

Quick update on this - I have the library compiling after the switch to quick-xml and most of the test cases fixed. Just tracking down the last couple of test failures.

[jangernert]

Used in production, but 0 open issues and 0 open pull requests. THE DREAM!

[markpritchard]

Hi @ayrat555 , could this be another example of a redirection? The feed works fine for me e.g. using the testurls program included with feed-rs:

$ echo 'https://www.tjrs.jus.br/site_php/noticias/news_rss.php' | cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `/home/mpritcha/devel/projects/external/rust/feed-rs/target/debug/testurls`
https://www.tjrs.jus.br/site_php/noticias/news_rss.php  ... ok

[ayrat555]

@markpritchard I doesn't seem like it.
It seems like an encoding issue. I'm passing raw bytes to feed-rss which it can not handle

    let client = match HttpClient::builder()
        .timeout(Duration::from_secs(5))
        .default_header("User-Agent", "el_monitorro/0.1.0")
        .redirect_policy(RedirectPolicy::Limit(10))
        .build()
    {
        Ok(cl) => cl,
        Err(er) => {
            let msg = format!("{:?}", er);

            return Err(FeedReaderError { msg });
        }
    };

    match client.get(url) {
        Ok(mut response) => {
            println!("{:?}", response);

            let mut writer: Vec<u8> = vec![];

            if let Err(err) = io::copy(response.body_mut(), &mut writer) {
                let msg = format!("{:?}", err);

                return Err(FeedReaderError { msg });
            }
            println!("{:?}", writer);
            Ok(writer)
        }
        Err(error) => {
            let msg = format!("{:?}", error);

            Err(FeedReaderError { msg })
        }
    }

the same issue is for https://pikabu.ru/xmlfeeds.php?cmd=popular .

[ayrat555]

Converting to utf-8 and passing to feed-rss works. I will use this workaround for now. But still, it's not the right way.

So maybe this behaviour should be documented. Also, I think it would be nice to make it configurable

[jangernert]

Smells a bit like #10

[ayrat555]

rss and atom-syndication crates can handle such cases. you can check out how they solved these cases. I think they use encoding tag from xml

[markpritchard]

@ayrat555 this was actually solved nicely by @jangernert with a patch to xml-rs. Unfortunately that crate doesn't seem to be maintained any more so I forked it and merged Jan's patch into my version. I'll write up some test cases and release a new version of feed-rs with this fix in it soon (just been busy with other things ... I live in Victoria, Australia and we are back into lockdown now... urgh).

[markpritchard]

I tested with the xml-rs patch, but decoding the Characters event after it is created fails because its expecting to decode the accumulated buffer as a valid UTF8 string.

I'll try switching to quick-xml (thanks for the tip @jangernert )

[ayrat555]

rust-syndication/rss#87 (comment) - rss also uses quick-xml

[jangernert]

@markpritchard before you start working on this. I have a WIP branch here switching to quick-xml. I'll try to get it into a working state and propose a MR for discussion.

[jangernert]

WIP is here: https://github.com/jangernert/feed-rs/tree/quick-xml
It's still messy. I'm trying to get all the tests to pass before opening the MR.

[markpritchard]

That is cool, thanks @jangernert ... I got a fair way on my changes too, but will take a look at your WIP as well.

[jangernert]

Oh, how far along are you? I have it compiling again and am at 8/28 tests passing. But I did probably do more copying than necessary to get out of lifetime hell with elements also possessing a reference to the element source.
If you are at a similar stage I don't think it makes sense for both of us to write more or less the same code.

[markpritchard]

Hi Jan, I’m taking the opportunity to redesign the interface with the underlying XML reader since I’ll probably switch it out for something else later (eg the owner of the XML crate is keen on doing a standards compliant parser too).

…

-- Mark Pritchard

On 15 Jul 2020, at 14:08, Jan Lukas Gernert ***@***.***> wrote:  Oh, how far along are you? I have it compiling again and am at 8/28 tests passing. But I did probably do more copying than necessary to get out of lifetime hell with elements also possessing a reference to the element source. If you are at a similar stage I don't think it makes sense for both of us to write more or less the same code. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[jangernert]

Okay then I'll stop working on it and let you do your thing.

[kkszysiu]

Hey guys,

How it is going with this issue?

[markpritchard]

Hey @kkszysiu ,

Apologies for the delay ... I've been pretty distracted with COVID stuff lately (live in Melbourne, AU ... we are now in stage 4 lockdown).

Hoping to get back on this later this week.

[kkszysiu]

@markpritchard thank you, no rush. I'm just trying to understand if you're planning to still maintain this library :)

I've prepared a feedparser library based on this lib for Python, here: https://github.com/kkszysiu/ultrafeedparser
As for now it is the fastest library for Python to parse feeds: https://github.com/kkszysiu/ultrafeedparser/tree/master/bench#statistics

Thanks!

[markpritchard]

@markpritchard thank you, no rush. I'm just trying to understand if you're planning to still maintain this library :)

I've prepared a feedparser library based on this lib for Python, here: https://github.com/kkszysiu/ultrafeedparser
As for now it is the fastest library for Python to parse feeds: https://github.com/kkszysiu/ultrafeedparser/tree/master/bench#statistics

Thanks!

Out of curiosity @kkszysiu, - do you have a Python project that needs to parse various feeds? Or was it just interest that led you to prepare the benchmark?

I'm interested to see how things change with the switch to quick-xml as the parser backend - the crate documentation mentions its 50x faster, and some more stats here: https://github.com/RazrFalcon/choose-your-xml-rs

[kkszysiu]

@markpritchard thank you, no rush. I'm just trying to understand if you're planning to still maintain this library :)
I've prepared a feedparser library based on this lib for Python, here: https://github.com/kkszysiu/ultrafeedparser
As for now it is the fastest library for Python to parse feeds: https://github.com/kkszysiu/ultrafeedparser/tree/master/bench#statistics
Thanks!

Out of curiosity @kkszysiu, - do you have a Python project that needs to parse various feeds? Or was it just interest that led you to prepare the benchmark?

I'm interested to see how things change with the switch to quick-xml as the parser backend - the crate documentation mentions its 50x faster, and some more stats here: https://github.com/RazrFalcon/choose-your-xml-rs

I'm planning to start using it on production to parse various feeds, yeah. I'm working in a project, where we have few hundreds of them to parse as a periodic tasks and the faster we can do that, the better of course.
But here the performance is just a value added. Not a requirement.

Yet I will definitely make new benchmarks, when new version of feed-rs that uses quick-xml would be released :)

[markpritchard]

Cool, thanks for the background @kkszysiu :)

I'll fix up #60 and #61 too, then cut another release (likely 0.4.0 given the internal library change)