Comments (3)
I coded this up by hacking LimitReader to be configurable by recordloader.py to test digests. If I turn it on for all of the existing tests, the only ones that fail are the truncated warc (expected) and the block_digest revisit record in example.warc (which might be your bug, not mine... I'll figure it out)
Because of your enthusiasm for streaming, the digest check will only fire if the user reads to the end of the record. So be it.
I have a few style questions:
-
Should I raise ArchiveLoadFailed if the digest check fails? ArchiveDigestFailed? ValueError?
-
Should the digest check default off? I didn't check perf, but it ought to be a lot cheaper than decompressing.
from warcio.
New try: #54
from warcio.
Done by "warcio check"
from warcio.
Related Issues (20)
- Add version tags to the repository HOT 2
- Invalid WARCs are silently accepted instead of raising an error HOT 5
- warcio mangles non-ASCII HTTP headers HOT 9
- warcio does not preserve HTTP header whitespace HOT 3
- quoted-string WARC header values are not parsed correctly
- Not compatible with WARC-files/records writtin by ArchiveSpark HOT 1
- get_test_file missing from the PyPI release HOT 4
- Offline tests HOT 2
- extract entire warc file? HOT 4
- warcio check does not raise error when GZip records are truncated HOT 5
- `capture_http` fails in tests, but works otherwise HOT 5
- Record not followed by newline (conversion error) HOT 1
- Warcio does not support replay of sites hosted on NCSA 1.5 HOT 3
- Issues with encoding of http-answers HOT 2
- Documentation: Clarify that capture_http writer with filename has no get_stream methood HOT 3
- warcio.exceptions.ArchiveLoadFailed: Unknown archive format HOT 3
- Empty WARC files when deploying warcio on Airflow HOT 5
- Trying to write to closed file when using `requests.Session`
- Patching WARCs using warcio
- warcio cannot write wet files
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from warcio.