Comments (4)
Hi @payingattention, your feedback and experience is important is remedying the issue you are experiencing. I have yet to isolate the part of the WARC that WARCreate is creating that is causing Webrecorder to choke. It is the responsibility of WARCreate to generate valid WARCs and not the expectation of other tools to account for any violations of the spec.
While I hope Webrecorder will improve to make the processing of invalid WARCs (ping @ikreymer) still usable in their tool suite (as there might be other tools with a similar affliction as WARCreate), I will need to isolate exactly what sort of record or formatting issue is causing the issue of tools meant to make the files WARCreate generates usable.
I do not have the time to do this at the moment (WARCreate is currently a volunteer-run, unfunded project), but am committed to get it back to a working state with time. Further independent contributions or studies onto the reasons for the effects you described (and of course, PRs) are welcomed.
from warcreate.
Also, @payingattention, if you would be willing to provide the resultant WARC, this can be used as the basis for reproducibility.
from warcreate.
Thank you, the greater spec/format is drastic improvement over just trusting archive.org/webcitation.org/archive.today for user-centric evidence gathering and giving users proof, we are fortunate to be the first to know the toolability exists, so troubleshooting WARCs is actually important for me:
- 20200704064631467.warc.zip is exact same archive from screenshot given.
- 20200708172743178.warc.zip is the same site but tried again fresh today, sadly same error.
While I have limited forensics as a user, I gather by my Ubuntu Linux system's U.I. behavior, that it's not showing the official WARC mime-type thumbnail (other .warc's show a package thumb/icon) because the generated archive copy itself is not saving completely/fully, if I'm forced to guess.
I appreciate the .warc still shows basic metadata for reference in a human-readable format. Thank you for the free open source extension, I hope my WARC zips help reproduce the error, I value your time.
from warcreate.
$ wget https://github.com/machawk1/warcreate/files/4892191/20200704064631467.warc.zip
$ unzip 20200704064631467.warc.zip
$ pip install warcio
$ python3
Python 3.8.2 (default, Apr 23 2020, 14:22:33)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from warcio.archiveiterator import ArchiveIterator
>>> with open('20200704064631467.warc', 'rb') as stream:
... for record in ArchiveIterator(stream):
... if record.rec_type == 'response':
... print(record.rec_headers.get_header('WARC-Target-URI'))
...
https://twitter.com/prosodyContext
WARNING: Record not followed by newline, perhaps Content-Length is invalid
Offset: 616195
Remainder: b'C/1.0\r\n'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 224, in _detect_type_load_headers
rec_headers = self.warc_parser.parse(stream, statusline)
File "/usr/local/lib/python3.8/site-packages/warcio/statusandheaders.py", line 270, in parse
raise StatusAndHeadersParserException(msg, full_statusline)
warcio.statusandheaders.StatusAndHeadersParserException: Expected Status Line starting with ['WARC/1.1', 'WARC/1.0', 'WARC/0.17', 'WARC/0.18'] - Found: WARC-Type: request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/local/lib/python3.8/site-packages/warcio/archiveiterator.py", line 110, in _iterate_records
self.record = self._next_record(self.next_line)
File "/usr/local/lib/python3.8/site-packages/warcio/archiveiterator.py", line 257, in _next_record
record = self.loader.parse_record_stream(self.reader,
File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 85, in parse_record_stream
(the_format, rec_headers) = (self.
File "/usr/local/lib/python3.8/site-packages/warcio/recordloader.py", line 229, in _detect_type_load_headers
raise ArchiveLoadFailed(msg + str(se.statusline))
warcio.exceptions.ArchiveLoadFailed: Invalid WARC record, first line: WARC-Type: request
>>>
from warcreate.
Related Issues (20)
- URIs with invalid characters are not escaped HOT 1
- Consider recording Memento Traces to WARCs HOT 1
- Store screenshot of page in WARC, too HOT 2
- WARCs of PDF include browser's wrapper
- Working status, how does it work? HOT 9
- [discussion/thought] Would a custom browser solution work better in terms of capabilities/UI than most current tools/proxies? HOT 14
- Generate WARC from offline MHTML HOT 2
- Adapting code to use manifest v3 HOT 6
- Use WABAC to replay WARCs
- Irrelevant Web Worker requests included HOT 1
- Add BibTeX reference to README
- Embedded fonts are not included in WARCs HOT 3
- What are the barriers in adapting WARCreate be used in the TOR browser? HOT 1
- Extension not working on most websites HOT 8
- Would it be possible to port this to Firefox? HOT 4
- Decouple from TravisCI for linting/testing
- WARC file names should follow the format recommended in Annex C
- Rádio HOT 1
- Be more transparent with the status of the WARC creation process
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from warcreate.