orottier/rust-warc

A high performance and easy to use Web Archive (WARC) file reader

rust-warc's Issues

Bug: reads only the archive info record of common crawl record(s)

I don't know if this is maintained, but I came here because warc took 34 seconds just to iterate through an archive with no processing and I wanted to see if this crate was faster.

I tried to read the Commoncrawl archive file CC-MAIN-20221210042021-20221210072021-00866.warc.gz (link, 1.1 GB compressed file) through a GzDecoder with a BufReader on top but I only get the first record:

pub fn analyse_warc(path: &str) -> Result<(), std::io::Error> {
    let f = File::open(path)?;
    let decoder = GzDecoder::new(f);
    let reader = BufReader::new(decoder);

    let warc = WarcReader::new(reader);
    for item in warc {
        println!("WARC: {:#?}", item.unwrap().header);
    }
}

Prints:

WARC: {
    CaseString {
        inner: "content-length",
    }: "500",
    CaseString {
        inner: "warc-date",
    }: "2022-12-10T04:20:21Z",
    CaseString {
        inner: "warc-type",
    }: "warcinfo",
    CaseString {
        inner: "content-type",
    }: "application/warc-fields",
    CaseString {
        inner: "warc-filename",
    }: "CC-MAIN-20221210042021-20221210072021-00866.warc.gz",
    CaseString {
        inner: "warc-record-id",
    }: "<urn:uuid:e609d058-9d7c-45ee-9292-eba237b4de5c>",
}

My guess would be that this would apply to all of the commoncrawl records.