Terminal crashes when pushing to sonic

[philiure]

Terminal crashes when pushing to sonic

How are you pushing the dataset? Memory is not necessarily used by Sonic, your ingest program can also use memory. I didn't have any problems when pushing 25M documents (~2 GB) using node-sonic-channel.

[philiure]

Found the problem! It was the logs from the server in the terminal that crashed it. Running the server with sonic > /dev/null fixed it and improved my push speed by 50%!

[philiure]

The push seemed to have slowed down significantly, at the start it was 2.5 ms/doc, and now it seems 50:ms/doc how much time did it take you to push 25M documents @Sly-Little-Fox ?

The push seemed to have slowed down significantly, at the start it was 2.5 ms/doc, and now it seems 50:ms/doc how much time did it take you to push 25M documents @Sly-Little-Fox ?

~4 hours with default config (with loglevel "error" though, "debug" can actually slow things down). I used tmpfs for storage and one core (Ampere A1, Oracle Cloud). I actually observed that using a lot of threads for ingesting sometimes makes Sonic slow down significantly even though it doesn't use even a half of my threads (8 threads, 4 cores). I don't know why it gets stuck (it's not I/O, because tmpfs).

[valeriansaliou]

Note that write operations are not lock-free, so if something is I/O bound, eg. the SSD I/O, the CPU core responsible for the RocksDB threads (try increasing parallelism?), then things will start slowing down as Sonic Channel and other threads rely on this main DB threads during write operations.

[philiure]

Alright, I'll see if I can improve it. Thanks a lot!