extract entire warc file?

[catharsis71]

extract entire warc file?

[wumpus]

Are you trying to run code on each webpage in the file?

Or do you want to leave all of the webpages on disk somehow?

The first way is what you probably should do. That's just a loop like the one in the documentation:

https://github.com/webrecorder/warcio#warc-and-arc-streaming

The second thing, leaving all of the webpages on disk as separate files, is difficult to do, because how will you name the files? And what happens if the WARC file has two webpages with exactly the same URL?

[catharsis71]

Yes I'm trying to unpack the entire contents to disk similar to warcat's "extract" option, preserving all the directory structure and filenames

Unfortunately warcat apparently has a bug where if it encounters an HTTP header in the WARC that it doesn't like it bombs out with no option to ignore/continue

So I'm exploring other options

but it seems like there's not really anything that does the job

[wumpus]

I suggest that you answer the questions that I asked, then I will implement it for you. If the WARC is from wget for a website that has no CGI arguments weirdnesses, then it's not so bad, but I would like you to be clear if this is sufficient.

[wumpus]

@catharsis71 ?