b3sum incredibly slow

[tERyceNzAchE]

b3sum incredibly slow

[oconnor663]

You know, I wonder if the performance problem here is audible :)

[tERyceNzAchE]

I wonder if the performance problem here is audible :)

It does seem to be.

[tERyceNzAchE]

That took it from minutes to seconds!

[tERyceNzAchE]

Yes, this is from a spinning drive.

[xzfc]

This isn't documented in b3sum itself, but you can disable multi-threading by setting RAYON_NUM_THREADS=1 environment variable.

[tERyceNzAchE]

This isn't documented in b3sum itself, but you can disable multi-threading by setting RAYON_NUM_THREADS=1 environment variable.

This didn't make a significant difference (like b3sum < big_file did).

[oconnor663]

Out of curiosity, what happens if you hash via stdin like this: b3sum < big_file? (Does that work on Windows?) I ask because that'll have the side effect of disabling memory mapping and multi-threading, and there's a chance that having a bunch of threads thrashing your disk could trigger some pathological performance issue.

[oconnor663]

Cool. "Thrashing the disk" is my current theory for your slow numbers. The next thing I'd be curious to try is a file small enough to fit in memory. That should work better with all the threads, because you won't actually be hitting the disk do much. Also, do you happen to have a spinning drive?

[oconnor663]

"Don't bother with multi-threading when you're on a spinning drive and the file is too small large to fit in cache" would be a decent performance heuristic, but I don't know of any simple or reliable way to detect either of those two conditions...

[tERyceNzAchE]

"Don't bother with multi-threading when you're on a spinning drive and the file is too small to fit in cache" would be a decent performance heuristic, but I don't know of any simple or reliable way to detect either of those two conditions...

This isn't a small file. Perhaps a command line flag to disable multi-threading?

[oconnor663]

Oops, meant to say "too large" above. Some flag like that might be useful for benchmarking and testing (situations like this), and we could totally add it. That said, I don't expect that most users facing this sort of issue will be able to discover that the --single-threaded flag (or whatever we call it) fixes things.

[oconnor663]

Interesting. I wonder if memory mapping itself is leading to thrashing, even if the access pattern of the mapping is serial? I don't know enough about the inner working of memory mapping to say, and especially not on Windows.

[oconnor663]

I'm going to close this issue for now, but we can re-open if it starts to look like there's something b3sum itself should be doing differently in the default case.