stxnext/pymongo-migrate

Mongodb data migrations using Python

pymongo-migrate

Mongodb migrations using Python.

Since mongodb is schema-less most of the time you can do without data migrations. Sometimes tho you want to create some new entities, or migrate old data instead adding another IF statement to your code. This is where pymongo-migrate comes in.

Usage

Usage: pymongo-migrate [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  downgrade  apply necessary downgrades to reach target migration
  graph
  migrate    automagically apply necessary upgrades or downgrades to reach
             target migration
  show       show migrations and their status
  upgrade    apply necessary upgrades to reach target migration

The most used command will be migrate, which works can be called like this:

$ pymongo-migrate migrate -u 'mongodb://localhost/test_db' -m tests/migrations

Use pymongo-migrate command --help to learn more about particular command.

Development & contributing to the project

Contributions and raising Issues is welcome; standard netiquette rules apply.

Use make init to setup the project for development. To format your code & test your changes run:

make check

Alternatives

ATM there seem only two active python projects like this:

• https://github.com/DoubleCiti/mongodb-migrations
• https://github.com/skynyrd/cikilop

So if something already existed, why then another project?

Goals of this project, where at least one of them were not fulfilled by above:

• tests and CI pipeline for ensuring that tool indeed works
• keeping it usable both as standalone tool and as python dependency
• use of modern Python version, which allows use of type annotations, dataclasses, f-strings and other goodies

Inspiration and design

Other than Alternatives mentioned above, both alembic and django were used as references when designing this tool.

For now only linear revision history is supported. The support for squash migrations is planned.

pymongo-migrate's Issues

Fresh install | don't run the scripts, but mark in the db that should start from the correct script.

In fresh install, I don't want the scripts to run (since there is no data that need to be transferred).

use case:

1. version 1,2,3 has released.
2. there are two migration script: a,b:
   1->a->2
   2->b->c
3. If I'm installing for a new customer, I will give my newest version: 3, I don't want to run the scripts since they are relevant for version 1->2->3...

Is there a way to mark that It's a fresh install, which means we need to mark that the script does not need to run, without really running them.

Documentation level is low

Hey,
can you update the readme file with the possible commands (for example, "generate" is missing)?

More importantly, the explanation for how to use the project is Incomplete.
Can you explain how the project should be used, and what/how its done?

Add additional pymongo client args

Could you add the ability to pass additional arguments through to the pymongo MongoClient? Specifically I need to be able to set some of the ssl parameters, but I imagine it could be implemented in a way that allows any of the MongoClient kwargs to be specified. Maybe an argument something like --pymongo_kwargs="key=value,key=value,..." or whatever you think would be an elegant solution.

Make the MongoMigrate class more flexible

We are trying to use pymongo-migrate in a project using mongo-engine, unfortunately your MongoMigrate class doesn't really play nice with that.

After taking a look at the source, I noticed MongoMigrate only really uses a database. The client passed for initialisation is only used to get the default database.

In my case, the client I can get from mongoengine with get_connection() doesn't have a default database and the code raises an exception. However I can get the database I want with get_db().

So I thought MongoMigrate should rather take the db as a constructor argument and maybe provide a classmethod to init from a pymongo.MongoClient.

I can work on a PR if you're okay with that.

Logging in upgrade

Make the migrator call the upgrade/downgrade method passing the logger, this way you can use it manually whenever you want.

Detection of missing migration files

If migrations was already applied it would be nice for pymongo-migrate to notice if a corresponding migration file goes missing (i.e. migration is marked as applied in db, but there is no mention of in migration graph). This probably should be a hard fail (error out). Such situation of course shouldn't occur, but is quite common when switching around between code branches in git.

Squash migrations support

To speed up migration process we want to support squash migrations.

Django can be used as inspiration:
https://docs.djangoproject.com/en/2.1/topics/migrations/#migration-squashing

Remove a script, is it possible? if so, how?

After setting few scripts, if I want to remove one of the scripts, how it is done? (is it possible?)

The full use case:

1. I wrote 4 scripts:

❯ ~/.local/bin/pymongo-migrate graph         
// Migrations
digraph {
	'dummy_migration' -> 'dummy_migration_2'
	'dummy_migration_2' -> 'dummy_migration_0'
	'dummy_migration_0' -> 'dummy_migration_3'
}

2. now I want to remove script dummy_migration_0 which will leave us with: (the expected graph):

digraph {
	'dummy_migration' -> 'dummy_migration_2'
	'dummy_migration_2' -> 'dummy_migration_3'
}

But now the commands give error:

❯ ~/.local/bin/pymongo-migrate graph                                                 
Traceback (most recent call last):
  File "/home/vagrant/.local/bin/pymongo-migrate", line 11, in <module>
    sys.exit(cli())
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/vagrant/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/vagrant/.local/lib/python3.6/site-packages/pymongo_migrate/cli.py", line 68, in wrap_with_client
    logger=get_logger(verbose),
  File "<string>", line 6, in __init__
  File "/home/vagrant/.local/lib/python3.6/site-packages/pymongo_migrate/mongo_migrate.py", line 44, in __post_init__
    for migration in load_module_migrations(self.migrations_path):
  File "/home/vagrant/.local/lib/python3.6/site-packages/pymongo_migrate/loader.py", line 22, in load_module_migrations
    name=migration_name, module=cast(MigrationModuleType, migration_module)
  File "/home/vagrant/.local/lib/python3.6/site-packages/pymongo_migrate/migrations.py", line 45, in __init__
    assert self.name == getattr(self.module, "name", name)
AssertionError

Migration should be optional in `downgrade` CLI

Hi,

first of all, thanks a lot for this nice tool, it's really great for quick Mongo migrations!
Here is a small CLI bug:

Describe the bug
The API allows for an optional migration param, but the CLI requires it.

To Reproduce
Run pymongo-migrate downgrade -m folder -u URI: you get an error saying MIGRATION is required.

Expected behavior
The MIGRATION param should be optional: if none is provided, we downgrade all migrations in the folder.

Proposed fix
In src/cli.py, line 144-145, replace:

@click.argument("migration")
def downgrade(mongo_migrate, migration):

with

@click.argument("migration", required=False)
def downgrade(mongo_migrate, migration=None):

Async migrations

Calling a migration in the form of await mongo_migrate.migrate() would be awesome.

I could help in order to achieve that feature.

`generate` should say that file was created and show its path

generate command should say that file was created and show its path.

Is the migration graph stable?

I've noticed this sentence at the very end of the README:

For now only linear revision history is supported.

I'm not sure I understand what you mean by that. We've made some tests with non-linear dependency graphs, and our migrations seem to be applied correctly.

So, is this something we should worry about, or did you just mean there's no support for squash migrations?

Add print of time took to each migration to run.

On big dbs, migration scripts can take long time to run.
It will help to know which migration took how much time.

Running upgrade migration 'A'
Runtime 'A': min:sec.milisec
Running upgrade migration 'B'
Runtime 'B': min:sec.milisec

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

custom migration file template

As a user it would be nice if I could provide migration template.

Similar feature is already implemented by alembic: script.py.mako. Personally I would avoid having external dependencies such as Mako or Jinja for this feature.

Generate | double generating of the same name overwrite the existing script, instead of throw an error

When generating an existing script, I would expect that it will fail if this script name already exists. Instead, it overwrites the existing script, which also breaks the graph structure if it was the last one

To Reproduce

1. generate a script named "aaa"
2. Again, generate a script named "aaa"

Expected behavior
The second generates command should fail, with an appropriate message to the user.

This is the script content after the second generation:

"""
Migration description here!
"""
name = 'aaa'
dependencies = ['aaa']


def upgrade(db: "pymongo.database.Database"):
    pass


def downgrade(db: "pymongo.database.Database"):
    pass

Notice that the script has itself as a dependency, which breaks the graph structure.

Add example usage from Python

The README shows an example usage from the CLI, I thought it'd be nice to have an example of Python code too.

To show that you are delivering on

keeping it usable [...] as [a] python dependency

Can't override logger

The logger passed to MongoMigrate's constructor isn't used for all logging.

This is an issue with the 0.9.1 release and has since been fixed by this commit.

When can we expect a new release shipping this fix?