o19s/trireme

Migration tool providing support for Apache Cassandra, DataStax Enterprise Cassandra, & DataStax Enterprise Solr.

trireme's Issues

Populate default Solr Core configuration files with some content

There should be something we can put in place to bootstrap a fresh Solr Core.

Add support for authenticating against a cluster.

This tool should work against a secured cluster.

Any plans to continue maintaining this repo?

I would like to fork and develop on this, but am wondering if this repository is being maintained?

This is important, as I would like to be able to publish changes made into Pip, whether this be done by you or myself.

I ask this as the last change is a while ago.

On Solr create / update compare files on disk with assets in C*

We should only run migrations when there are changes. Currently the Solr implementation is basic at best.

Support pushing all files under a solr core to DSE solr

For example, I may want a stopwords.txt file uploaded along with my core. Currently only the schema.xml and solrconfig.xml file are pushed.

Add tests

These are important, get on this issue.

Recommended packages:

• Responses - Mocking library for requests
• Nose - Testing framework
• Coverage - Code coverage tool
• Freezegun - Freezes datetime interactions (for use when testing)

Writing migration CQL-scripts

Can you give me a hint how you write scripts for a migration from one schema to another, please? It's offtopic a little bit but I just don't understand how to use this solution without such knowledge. For example,

• You need to add new table in the next migration. And you should fill it with existing data.
• You should change the type or name of an existsing column

Thanks!

Don't store Solr secondary indexes in schema dump

These should be omitted. They will be added when the Solr core is brought up.

Failed to launch tasks : TypeError("Tasks must have an initial Context argument!")

When tried to list all the task supported by trireme using inv -l, the command failed with the following stacktrace:

Traceback (most recent call last):
  File "/usr/local/bin/inv", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 286, in run
    self._parse(argv)
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 352, in _parse
    self.load_collection()
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 500, in load_collection
    coll = loader.load(coll_name) if coll_name else loader.load()
  File "/usr/local/lib/python2.7/dist-packages/invoke/loader.py", line 53, in load
    module = imp.load_module(name, fd, path, desc)
  File "/home/ajmal/test/migrate/tasks.py", line 2, in <module>
    from trireme import trireme
  File "/usr/local/lib/python2.7/dist-packages/trireme/trireme.py", line 4, in <module>
    from trireme.migrators import cassandra, solr
  File "/usr/local/lib/python2.7/dist-packages/trireme/migrators/cassandra.py", line 53, in <module>
    @task
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 275, in task
    return Task(args[0], **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 61, in __init__
    self.positional = self.fill_implicit_positionals(positional)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 152, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 147, in argspec
    raise TypeError("Tasks must have an initial Context argument!")

I think the error is related to a recent change in the invoke library, and can be worked around by adding a context parameter to every task function.

Rollbacks (for Cassandra)

I like how light this tool is, and how clean the source is, but I do miss having rollbacks in a migration tool.

I think it would require 2 changes to get them though:

• the migrate step would need to filter out .down.cql files.
• A new task would need to get added that takes a migration name as a prameter. Something like rollback --target 20160101-new-table, which would run all the .down.cql files after the provided migration. Basically undo all the migrations after the 20160101-new-table.

How does this sound? I could write up a PR if it sounds good.
Thanks,
Charles

Use proper quotes throughout .py files

This is driving me crazy

Replace call() command in ```cassandra.migrate``` with Invoke's ```run()``` method

Support specifying replication class and options from cassandra.create call

Do not include the solr_query column in schema dumps

Add a hash of the migration that has been run to the migrations table

Make location of CQL files configurable

Store date a migration has been run or applied

This can help the migrations table to act as a sort of audit trail

Flesh out solr.migrate to support querying the collection status.

If a collection is running warn the user and do NOT perform the RELOAD call.

TODO Determine if we want to push the updated files if a RELOAD will not be performed.

Add migration type to the migrations table

CQL or Solr

Dump the schema to a file once migrations have been run

Also add a task to load the schema from the dumped export into a clean keyspace.This would save time as migrations wouldn't need to be run when initially setting up the project. Optionally inspect the migrations table to see if the keyspace is in a state to accept the schema load.

Code Path 1: New developer spinning up on project

inv cassandra.create cassandra.load_schema

The keyspace is created and db/schema.cql will be loaded.

Code Path 2: Developer adds a migration

inv cassandra.add_migration --name add_foo_table

developer fills out the new migration file

inv cassandra.migrate

The migration file is run (along with any other migrations that have not run). The db/schema.cql file is replaced with an updated version that contains all CREATE statements that include the new migration.

Package into something that may be installed with pip

Currently the process to install is a little involved. Make this easier by providing the package via a tool like pip.

Generate documentation to be published along with the package.

Make communicating with devs easier...

https://gitter.im/spark-jobserver/spark-jobserver?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge