Anything I can do to speedup keyspace dump ?

[gianlucaborello]

Anything I can do to speedup keyspace dump ?

[gianlucaborello]

That is mostly because the read path is not concurrent, as a simple toy tool it was never meant to be used for such big use cases :)

Some workarounds I can give you, from "easy" to "hard":

1. Increase the value of FETCH_SIZE in the code, it might help making the requests more efficient when reading a lot of data, you should be able to notice an improvement.

2. Ditch this tool altogether, and adopt a more sound strategy with tools that scale well (e.g. nodetool snapshot + sstableloader), which make a lot of sense when dealing with dozens of GB like your case.

3. Run multiple instances of the tool at the same time, each one focusing on a different column family or a different subset of filters and then merge the files manually, so you should be able to concurrently get a lot of stuff done at the same time.

4. Implement proper concurrency inside the tool: in other words, instead of always doing a SELECT * FROM foo, you should split the range of primary keys and start doing a whole lot of sub queries and then execute them concurrently with cassandra.concurrent.execute_concurrent() (which I'm sort of using during the import phase, since it's easier there). Cassandra will happily scale to thousands of concurrent read requests per second, whereas now I'm doing just one.

If I were to do this professionally, I would definitely go for option 2, or if I really needed a plain text dump I'd implement 4. Unfortunately, I don't have time at the moment to embark in this big feature.

Thanks