maxcountryman/warc-parquet

🗄️ A simple CLI for converting WARC to Parquet.

warc-parquet

🗄️ A utility for converting WARC to Parquet.

📦 Install

The binary may be installed via cargo:

$ cargo install warc-parquet

To use the crate in your project, add the following to your Cargo.toml file:

[dependencies]
warc-parquet = "0.6.1"

🤸 Usage

The Binary

Once installed, the warc-parquet utility can be used to transform WARC into Parquet:

$ wget --warc-file example 'https://example.com'
$ cat example.warc.gz | warc-parquet --gzipped > example.zstd.parquet

warc-parquet is meant to fit organically into the UNIX ecosystem. As such processing multiple WARCs at once is straightforward:

$ wget --warc-file github 'https://github.com'
$ cat example.warc.gz github.warc.gz | warc-parquet --gzipped > combined.zstd.parquet

It's also simple to preprocess via standard UNIX piping:

$ cat example.warc.gz | gzip -d | warc-parquet > example.zstd.parquet

Various compression options, including the option to forego compression altogether, are also available:

$ cat example.warc.gz | warc-parquet --gzipped --compression gzip > example.gz.parquet

💡 warc-parquet --help displays complete options and usage information.

The Crate

Refer to the docs for more details about how to use the Reader within your own programs.

DuckDB

There are any number of ways to consume Parquet once you have it. However a natural fit might be DuckDB:

$ duckdb
v0.3.3 fe9ba8003
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

D select type, id from 'example.zstd.parquet';
┌──────────┬─────────────────────────────────────────────────┐
│   type   │                       id                        │
├──────────┼─────────────────────────────────────────────────┤
│ warcinfo │ <urn:uuid:A8063499-7675-4D8D-A736-A1D7DAE84C84> │
│ request  │ <urn:uuid:3EB20966-D74F-4949-AACB-23DB3A0733A7> │
│ response │ <urn:uuid:8B92CADC-F770-45BE-8B72-E13A61CD6D1C> │
│ metadata │ <urn:uuid:4C0E9E17-E21B-49E0-859A-D1016FBDE636> │
│ resource │ <urn:uuid:14F502A5-3BDE-4D0B-8A43-95F4BB8398C6> │
│ resource │ <urn:uuid:6B6D6ADD-52FF-4760-AA00-FB9E739CABBE> │
└──────────┴─────────────────────────────────────────────────┘

D describe select * from 'example.zstd.parquet';
┌─────────────────────────┬─────────────┬──────┬─────┬─────────┬───────┐
│       column_name       │ column_type │ null │ key │ default │ extra │
├─────────────────────────┼─────────────┼──────┼─────┼─────────┼───────┤
│ id                      │ VARCHAR     │ YES  │     │         │       │
│ content_length          │ UINTEGER    │ YES  │     │         │       │
│ date                    │ TIMESTAMP   │ YES  │     │         │       │
│ type                    │ VARCHAR     │ YES  │     │         │       │
│ content_type            │ VARCHAR     │ YES  │     │         │       │
│ concurrent_to           │ VARCHAR     │ YES  │     │         │       │
│ block_digest            │ VARCHAR     │ YES  │     │         │       │
│ payload_digest          │ VARCHAR     │ YES  │     │         │       │
│ ip_address              │ VARCHAR     │ YES  │     │         │       │
│ refers_to               │ VARCHAR     │ YES  │     │         │       │
│ target_uri              │ VARCHAR     │ YES  │     │         │       │
│ truncated               │ VARCHAR     │ YES  │     │         │       │
│ warc_info_id            │ VARCHAR     │ YES  │     │         │       │
│ filename                │ VARCHAR     │ YES  │     │         │       │
│ profile                 │ VARCHAR     │ YES  │     │         │       │
│ identified_payload_type │ VARCHAR     │ YES  │     │         │       │
│ segment_number          │ UINTEGER    │ YES  │     │         │       │
│ segment_origin_id       │ VARCHAR     │ YES  │     │         │       │
│ segment_total_length    │ UINTEGER    │ YES  │     │         │       │
│ body                    │ BLOB        │ YES  │     │         │       │
└─────────────────────────┴─────────────┴──────┴─────┴─────────┴───────┘

🦺 Safety

This crate uses #![forbid(unsafe_code)] to ensure everything is implemented in 100% safe Rust.

👯 Contributing

We appreciate all kinds of contributions, thank you!

warc-parquet's Issues

thread 'main' panicked at 'range end out of bounds'

Issue Title: thread 'main' panicked at 'range end out of bound

---

🐛 Description

• What did you do?
  I attempted to convert a Common Crawl WARC file to a Parquet file for analysis in DuckDB. I downloaded the file, attempted to process it with warc-parquet, but it wasn't able to correctly/fully process the warc.gz file.

export RUST_BACKTRACE=full; cat CC-NEWS-20230803071746-00285.warc.gz | gzip -d | warc-parquet > CC-NEWS-20230803071746-00285.warc.gz.parquet

• What did you expect to see?
  A new parquet file with a similar size to the original compressed warc.gz file, around 1 GiB.

• What did you see instead?
  A new parquet file with only 6,476,466 bytes.

---

📎 Steps to Reproduce

wget https://data.commoncrawl.org/crawl-data/CC-NEWS/2023/08/CC-NEWS-20230803071746-00285.warc.gz
# Length: 1072778533 (1023M) [binary/octet-stream]
# Saving to: ‘CC-NEWS-20230803071746-00285.warc.gz’
export RUST_BACKTRACE=full; cat CC-NEWS-20230803071746-00285.warc.gz | gzip -d | warc-parquet > CC-NEWS-20230803071746-00285.warc.gz.parquet
# thread 'main' panicked at 'range end out of bounds: 18446744071562881016 <= 6014243295', /home/niro/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bytes-1.4.0/src/bytes.rs:261:9

---

🖥 System Information

• Rust Version: rustc 1.72.0 (5680fa18f 2023-08-23)
• Operating System and Version: Ubuntu 22.04.3 LTS
• warc-parquet version: warc-parquet 0.4.0

---

📝 Additional Context

• Logs, error messages, or panic messages: See attached backtrace.txt
  backtrace.txt