appliedinfrastructure/cassandra-snapshot-tools

Create, move, and restore keyspace snapshots across Apache/Datastax Cassandra clusters

Cassandra Snapshot Tools

A handy couple of scripts that make creating, moving, and restoring SSTable snapshots between clusters easy. Some common use cases include:

• Copying keyspaces between clusters (QA environments, for example)
• Staging an upgrade (application, Cassandra, both) using a copy of live data
• Seeding data in a test environment
• Snapshotting a keyspace for offline analytics
• Backup and restore
• Generally moving a keyspace from one Cassandra cluster to another

Cassandra Snapshot Tools currently includes two BASH shell scripts, getSnapshot and putSnapshot, which leverage standard Cassandra utilities like nodetool, cqlsh, and sstableloader to simplify the process of creating snapshots and moving them between hosts. A convenient compressed tar archive is generated by getSnapshot, and includes all of the SSTable snapshot files, metadata, and schema information necessary to restore the keyspace into another Cassandra cluster (or the same cluster). The putSnapshot tool is then used to alter various attributes of the original snapshot, and restore into the destination Cassandra cluster.

Features

• Automatically generates a keyspace snapshot and packages into an easy-to-move archive
• Copies snapshots out of the Cassandra data directory, allowing space to be reclaimed using nodetool clearsnapshot
• Archive a previously created snapshot (e.g. created manually using nodetool snapshot)
• Schedulable using Cron (e.g. for scheduled backups)
• Change keyspace name, datacenter name and replication factor on restore
• Restore to local or remote clusters, either privately hosted or through hosted services like Datascale.io
• Easy to use with sane defaults
• Most Cassandra versions supported (tested against Cassandra 2.0, 2.1, 2.2, 3.0, and 3.7)

Usage

getSnapshot

Usage: ./getSnapshot -h
       ./getSnapshot -k <keyspace name> [-s <snapshot name>] [-y <cassandra.yaml file>] [--no-timestamp]
    -h,--help                          Print usage and exit
    -v,--version                       Print version information and exit
    -k,--keyspace <keyspace name>      REQUIRED: The name of the keyspace to snapshot
    -s,--snapshot <snapshot name>      The name of an existing snapshot to package
    -y,--yaml <cassandra.yaml file>    Alternate cassandra.yaml file
    --no-timestamp                     Don't include a timestamp in the resulting filename

putSnapshot

Usage: ./putSnapshot -h
       ./putSnapshot -f <snapshot file> [-n <node address>] [-k <new ks name>] [-d <new dc name>] [-r <new rf>] [-y <cassandra.yaml file>]
    -h,--help                          Print usage and exit
    -v,--version                       Print version information and exit
    -f,--file <snapshot file>          REQUIRED: The snapshot file name (created using the
                                       getSnapshot utility
    -n,--node <node address>           Destination Cassandra node IP (defaults to the local
                                       Cassandra IP if run on a Cassandra node, otherwise
                                       required in order to connect to Cassandra.  Will take
                                       precedence if provided and run on a Cassandra node
    -k,--keyspace <new ks name>        Override the destination keyspace name (defaults to
                                       the source keyspace name)
    -d,--datacenter <new dc name>      Override the destination datacenter name (defaults
                                       to the sourcen datacenter name)
    -r,--replication <new rf>          Override the destination replication factor (defaults
                                       to source replication factor)
    -y,--yaml <cassandra.yaml file>    Alternate cassandra.yaml file

Basic Examples

1. Copy a keyspace to the same Cassandra cluster using a different keyspace name:

   $ getSnapshot -k <keyspace name>
   $ putSnapshot -f <snapshot package file> -k <new keyspace name>

2. Copy a keyspace to a remote Cassandra cluster using the same keyspace name:

   $ getSnapshot -k <keyspace name>
   $ putSnapshot -f <snapshot package file> -n <destination node IP>

3. Copy a keyspace from a Cassandra cluster to a remote cluster using a different keyspace name and replication factor:

   $ getSnapshot -k <keyspace name>
   $ putSnapshot -f <snapshot package file> -n <destination node IP> -k <new keyspace name> -r 1

4. Copy a snapshot previously created using nodetool snapshot to a new keyspace:

   $ nodetool snapshot <keyspace name> -t <custom snapshot name>
   $ getSnapshot -k <keyspace name> -s <custom snapshot name>
   $ putSnapshot -f <snapshot package file> -k <new keyspace name>

Caveats

• Currently supports snapshots/restores between clusters running similar versions of Cassandra.
• Partitioner configuration (e.g. RandomPartitioner, Murmur3Partitioner, etc.) must be the same between source and destination Cassandra clusters.
• Local access to the source Cassandra node is required (to create/collect snapshot files). Fairly open network access is required to the destination Cassandra node (to create the schema and load SSTables).

Contributing

All contributions welcome! Please see How to Contribute.

cassandra-snapshot-tools's Issues

Key space names are handled incorrectly when the new name includes the old one

Do you have the plan to support the new version of Cassandra? When I run it on Cassandra 3.11.2, putSnapshot gives lots of errors.

Diagram

May be a simple diagram will make it easy to understand as how it works.

Implement client authentication support

Need to support authentication for clusters that use it.

Tar command is incorrect on getSnapshot

I believe there is a slight error on the tar command:

tar --directory "$DUMPDIR" \
            -zcvf $FILENAME \
                  $KEYSPACE \
                  $SCHEMA \
                  $KEYSPFILE \
                  $SNAPSFILE \
                  $HOSTSFILE \
                  $DATESFILE >/dev/null 2>&1

$KEYSPACE should not be there, I think.

Support SimpleStrategy replica placement

Currently the NetworkTopologyStrategy and its options are assumed (because that's what we currently use) - need to support (and test for) SimpleStrategy as well.

GetSnapshot doesn't handle multiple data directories

On a node where more than one data_file_directory is defined in cassandra.yaml, the getSnapshot script searches for the appropriate keyspace in all data_file_directory entries, but uses only the first directory containing the keyspace and snapshot. This will likely not include all of the necessary SSTables required to successfully restore the keyspace.

Need to implement a loop that searches in all data file directories for the keyspace and snapshot, and includes them in the snapshot dump.

Error finding required executables: cqlsh

When I run the script Im seeing the following error. I have placed the script in /Users/ksachdeva/dse/bin directory. Any idea ?

Kunals-Mac:bin ksachdeva$ ./getSnapshot -k admin -s admin.backup -y ../resources/cassandra/conf/cassandra.yaml
Error finding required executables: cqlsh

SSL Support

It will be good to add SSL , keystore and truststore support for the sstableloader options. Or you can pick up from the Cassandra.yaml as well

typo in get snapshot

diff getSnapshot.orig getSnapshot
169c169
< for directory in ${data_file_directories_[@]}; do

        for directory in ${data_file_directories[@]}; do

ie. seems to be a trailing underscore. Works for me with it removed.

Snapshot info tool to view snapshot details

Need a simple way to view details about a snapshot file. For example, it would be nice to be able to view the datacenter/replication factor configuration in a given snapshot.

getSnapshot fails to parse cassandra.yaml if indented with less than 3 spaces

We hit a confusing bug recently where getSnapshot was printing No data directories, or no cassandra.yaml file found on a new Cassandra installation, even though all our paths were the same as on prior working installations, and our cassandra.yaml looked the same. What had happened was that the YAML had been autoformatted, setting the indentation to our usual in-house value of 2 spaces. It turns out that the awk-based YAML quasi-parser in getSnapshot didn't correctly read the data directories line when it was only indented by two. 3 or 4 worked fine, 2 or 1 didn't. I'd try to offer a fix but don't have enough awk skills to feel safe modifying this.

Failing snippet of cassandra.yaml:

data_file_directories:
  - /var/lib/cassandra/data

$ ./getSnapshot -k the_keyspace -s the_snapshot
No data directories, or no cassandra.yaml file found

Working snippet of cassandra.yaml:

data_file_directories:
    - /var/lib/cassandra/data

$ ./getSnapshot -k the_keyspace -s the_snapshot
Using provided snapshot name the_snapshot
...

Implement client encryption support

Need a friendly way to use SSL in clusters that enforce SSL client connections