o19s/trireme

Migration tool providing support for Apache Cassandra, DataStax Enterprise Cassandra, & DataStax Enterprise Solr.

Trireme

Trireme is a tool providing migration support for Apache Cassandra, DataStax Enterprise Cassandra & Solr. Commands are run using the Python Invoke CLI tool.

Update: March 2019 - We are no longer maintaining this Gem. Please contact [email protected] if you would like to take over this project.

System Dependencies

• cqlsh must be on the PATH. Some tasks utilize the cqlsh tool for running scripts and dumping schemas to disk.

Integration

To use this tool within the scope of your project follow these steps.

1. Install trireme with pip install trireme

2. Create a tasks.py file with the following content:

   from invoke import Collection
   from trireme import trireme
   
   namespace = Collection(trireme)

3. Create a trireme_config.py file with your Cassandra and Solr information.

   # Cassandra Configuration
   
   # Contact points for your cluster, currently only the first is used
   contact_points = ['127.0.0.1']
   
   # Keyspace to work with, this doesn't have to exist yet.
   keyspace = 'foo'
   # Replication options. Defined as a map just as you would in CQL.
   replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }
   # replication = {'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2}
   
   # Authentication Information
   username = None
   password = None
   
   # Flag indicating whether this host is the migration master. Migrations are only run on the migration master
   migration_master = True
   
   
   # Solr Configuration
   solr_url = 'http://127.0.0.1:8983/solr'

4. Run the trireme setup task to create the basic directories

   inv trireme.setup

Usage

Migrators contain logic to run migrations. This project contains a migrator for simple CQL scripts and Solr core configuration files. Not every migrator supports all actions. For example Solr included in DSE 4.6 does not include the ability to delete cores. In this case we do not have a task named drop.

List all commands:

inv -l

To list optional parameters for a command:

inv --help command.name.

Example: inv --help cassandra.add_migration

Cassandra

Note: This feature works with both Apache Cassandra and DataStax Enterprise

Actions supported:

• cassandra.create - Creates the keyspace along with a table to track migrations. Note: The default replication strategy is SimpleStrategy with a Replication Factor of 3.
• cassandra.drop - Drops the keyspace.
• cassandra.migrate - Runs all missing migrations against the keyspace.
• cassandra.add_migration --name migration_name - Generates a migration file under db/migrations with the current timestamp prepended to the provided --name value.
• cassandra.dump_schema - Dumps the current schmea to ```db/schema.cql``. Be careful when using this feature when Solr is enabled.
• cassandra.load_schema - Loads the schema from db/schema.cql. This may be faster than running all migrations in a project.

Examples:

• inv cassandra.create cassandra.migrate - Creates the keyspaces and runs all migrations
• inv cassandra.load_schema - Creates the keyspace and loads the schema from db/schema.cql

Solr

Note: This feature only works with DataStax Enterprise

Actions supported:

• solr.create [--core foo.bar] - Uploads the core configuration files and calls the CREATE API endpoint
• solr.migrate [--core foo.bar] - Uploads the core configuration files and calls the RELOAD API endpoint
• solr.add_core --name foo.bar - Creates a core configuration directory and files. Use the format keyspace.table_name when naming your cores.

Example: inv solr.create - Uploads all core configuration files and calls the create core API endpoint.

solr.create and solr.migrate support the --core core.name flag. This will run the task against only one core instead of all cores. Remember the core name in DSE Solr is keyspace.table_name.

Directory Layout

db/migrations

CQL migration files generated by the cassandra.add_migration command will be placed in this directory with a timestamp prepended.

Example directory layout:

db/
  migrations/
    201501301409_create_users_table.cql
    201501301623_create_tweets_table.cql
    ...

db/solr

Folder containing all the Solr core configuration files. With DataStax Enterprise the core name is comprised of the keyspace and table name in the format keyspace.table_name. Within this directory we house sub-directories for each core. These directories in turn have the schema.xml and solrconfig.xml files needed for configuring the core.

Example directory layout:

db/
  solr/
    example_keyspace.a_table/
      schema.xml
      solrconfig.xml
    example_keyspace.b_table/
      schema.xml
      solrconfig.xml

Trireme Project Layout

trireme/
  migrators/
    cassandra.py
    solr.py
  trireme.py

migrators

The code that powers the migration engine. Each migrator receives its own file and provides invoke tasks.

trireme.py

Collects all of the Invoke tasks into a common namespace along with a simple setup task

Python Dependencies

All required items have been specified in requirements.txt and setup.py. Select items are outlined below.

• cassandra-driver - DataStax driver for connecting with Cassandra, used when creating and dropping keyspaces
• requests - HTTP Client, used when communicating with the Solr APIs
• invoke - Task execution tool & library. This is used to run the exposed migration tasks

Extending Trireme

Adding a new migrator involves placing the code to invoke annotations in a file within the migrators directory. Next add your migrator to the Collection entry in trireme.py. If you create a new migrator and would like to share it with the community, please fork the repo, add your migrator, and then open a pull request.

trireme's Issues

Store date a migration has been run or applied

This can help the migrations table to act as a sort of audit trail

Make communicating with devs easier...

https://gitter.im/spark-jobserver/spark-jobserver?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge

Add tests

These are important, get on this issue.

Recommended packages:

• Responses - Mocking library for requests
• Nose - Testing framework
• Coverage - Code coverage tool
• Freezegun - Freezes datetime interactions (for use when testing)

Make location of CQL files configurable

Add support for authenticating against a cluster.

This tool should work against a secured cluster.

Do not include the solr_query column in schema dumps

Support specifying replication class and options from cassandra.create call

Package into something that may be installed with pip

Currently the process to install is a little involved. Make this easier by providing the package via a tool like pip.

Rollbacks (for Cassandra)

I like how light this tool is, and how clean the source is, but I do miss having rollbacks in a migration tool.

I think it would require 2 changes to get them though:

• the migrate step would need to filter out .down.cql files.
• A new task would need to get added that takes a migration name as a prameter. Something like rollback --target 20160101-new-table, which would run all the .down.cql files after the provided migration. Basically undo all the migrations after the 20160101-new-table.

How does this sound? I could write up a PR if it sounds good.
Thanks,
Charles

Don't store Solr secondary indexes in schema dump

These should be omitted. They will be added when the Solr core is brought up.

Support pushing all files under a solr core to DSE solr

For example, I may want a stopwords.txt file uploaded along with my core. Currently only the schema.xml and solrconfig.xml file are pushed.

Replace call() command in ```cassandra.migrate``` with Invoke's ```run()``` method

Populate default Solr Core configuration files with some content

There should be something we can put in place to bootstrap a fresh Solr Core.

Add a hash of the migration that has been run to the migrations table

Use proper quotes throughout .py files

This is driving me crazy

Any plans to continue maintaining this repo?

I would like to fork and develop on this, but am wondering if this repository is being maintained?

This is important, as I would like to be able to publish changes made into Pip, whether this be done by you or myself.

I ask this as the last change is a while ago.

Writing migration CQL-scripts

Can you give me a hint how you write scripts for a migration from one schema to another, please? It's offtopic a little bit but I just don't understand how to use this solution without such knowledge. For example,

• You need to add new table in the next migration. And you should fill it with existing data.
• You should change the type or name of an existsing column

Thanks!

Failed to launch tasks : TypeError("Tasks must have an initial Context argument!")

When tried to list all the task supported by trireme using inv -l, the command failed with the following stacktrace:

Traceback (most recent call last):
  File "/usr/local/bin/inv", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 286, in run
    self._parse(argv)
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 352, in _parse
    self.load_collection()
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 500, in load_collection
    coll = loader.load(coll_name) if coll_name else loader.load()
  File "/usr/local/lib/python2.7/dist-packages/invoke/loader.py", line 53, in load
    module = imp.load_module(name, fd, path, desc)
  File "/home/ajmal/test/migrate/tasks.py", line 2, in <module>
    from trireme import trireme
  File "/usr/local/lib/python2.7/dist-packages/trireme/trireme.py", line 4, in <module>
    from trireme.migrators import cassandra, solr
  File "/usr/local/lib/python2.7/dist-packages/trireme/migrators/cassandra.py", line 53, in <module>
    @task
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 275, in task
    return Task(args[0], **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 61, in __init__
    self.positional = self.fill_implicit_positionals(positional)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 152, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 147, in argspec
    raise TypeError("Tasks must have an initial Context argument!")

I think the error is related to a recent change in the invoke library, and can be worked around by adding a context parameter to every task function.

On Solr create / update compare files on disk with assets in C*

We should only run migrations when there are changes. Currently the Solr implementation is basic at best.

Flesh out solr.migrate to support querying the collection status.

If a collection is running warn the user and do NOT perform the RELOAD call.

TODO Determine if we want to push the updated files if a RELOAD will not be performed.

Generate documentation to be published along with the package.

Add migration type to the migrations table

CQL or Solr

Dump the schema to a file once migrations have been run

Also add a task to load the schema from the dumped export into a clean keyspace.This would save time as migrations wouldn't need to be run when initially setting up the project. Optionally inspect the migrations table to see if the keyspace is in a state to accept the schema load.

Code Path 1: New developer spinning up on project

inv cassandra.create cassandra.load_schema

The keyspace is created and db/schema.cql will be loaded.

Code Path 2: Developer adds a migration

inv cassandra.add_migration --name add_foo_table

developer fills out the new migration file

inv cassandra.migrate

The migration file is run (along with any other migrations that have not run). The db/schema.cql file is replaced with an updated version that contains all CREATE statements that include the new migration.