Fastest way to read a csv file?

[burntsushi]

Fastest way to read a csv file?

[phiresky]

Just in case someone else finds this, here's an overview of things to make it faster (tested on a file with four columns and one billion lines):

1. compile with --release (huge difference, >10x perf)

2. wrap your input in a BufReader::with_capacity(1_000_000). No difference for me, probably depends on where the data comes from

3. use .byte_records instead of .records if you don't need string parsing (only minor difference for me)

4. enable this: (opt-level 3 is not much faster than level 2, lto=fat improves perf by 15%!)

    [profile.release]
    opt-level = 3
    debug = true
    lto = "fat"
   

5. compile with RUSTFLAGS="-C target-cpu=native" (only minor difference)

6. Instead of for result in reader.into_byte_records(), use:

   let mut record = csv::ByteRecord::new();
   while reader.read_byte_record(&mut record)? {

   this doubles the performance!! This is also what xsv does: https://github.com/BurntSushi/xsv/blob/3de6c04269a7d315f7e9864b9013451cd9580a08/src/cmd/select.rs#L77

With the release mode and read_byte_record, perf of using the library is the same as xsv select for me.

[Eh2406]

Did you compile in release? Don't know the real answer, just a knee jerk response to "Rust beginner" and "expecting to run faster" :-)

[BurntSushi]

@patricebellan Have you read the section in the docs about iterating over records?

I would somewhat expect xsv to run quite a bit faster than 25 seconds on a mere 7 million rows. I share @Eh2406's concerns. Instead of cargo build you might try cargo build --release.

xsv select barely does anything either. It should run within spitting distance of a simple count loop.

[BurntSushi]

Thanks for adding your tip here! There is more explanation here on the technique: https://docs.rs/csv/1.1.3/csv/tutorial/index.html#amortizing-allocations

[patricebellan]

Nice try @Eh2406, I did compile in release ;)

I'm running it on a VM, so overall performance may not be the best.
But I was mostly concerned about comparing both, not pure performance per se.

[hakunin]

Just ran into this myself - debug mode reads mere 2M rows in 18seconds, release build does it in 630ms.

[phiresky]

Thanks for the link. Funny, I didn't actually see that, I just searched for "performance" within the docs index page but it didn't have results, and I assumed "tutorial" was more about handling different types of files etc, not about improving performance. And the docs search (obviously i guess) didn't yield it either :)