paltman-archive/nashvegas

Another database migration script runner for Django projects.

NO LONGER SUPPORTED: This project will receive no more updates and pull requests and issues will no longer be monitored. I recommend everyone begin to adopt the migrations support that is in Django 1.7 and use South until you are on Django 1.7.

Nashvegas

The purpose of this app is to enable a plug and play method for managing database changes.

Database migrations is a large topic with a lot of different approaches. This approach worked well for my needs and maybe it will for you as well.

Documentation

You can find the documentation in the docs/ folder of the repo or online at:

http://nashvegas.readthedocs.org

Installation

Installation is simple:

$ pip install nashvegas

Then add nashvegas to your INSTALLED_APPS in your Django settings.py file.

Support

You can either log issues on the Github issue tracker for this project or pop into #nashvegas on Freenode.

nashvegas's Issues

Inconsistency between NASHVEGAS docs & code

The docs show an example:

NASHVEGAS = {
"createdb": "createdb -U postgres -T template0 -E UTF8",
"dropdb": "dropdb -U postgres",
"pg_dump": "pg_dump -U postgres",
}

But this won't actually execute properly. The issue is that the code doesn't create the right commands. eg:
def setup_database(self):
command = "createdb %s" % self.name
if NASHVEGAS and "createdb" in settings.NASHVEGAS:
command = settings.NASHVEGAS["createdb"]
Popen(command.split()).wait()

If you don't specify the NASHVEGAS settings, it tries to execute a command like:
createdb mydb_compare
(where mydb_compare is generated, or passed as an option on the CLI)

If you specify NASHVEGAS["createdb"] as in the example, you get:
createdb -U postgres -T template0 -E UTF8
(no DB name provided)

I believe the "fix" is to have code similar to the following (for setup_database, teardown_database, and handle)
command = settings.NASHVEGAS["createdb"] + " %s" % self.name

Comments?

Add better (more robust) error handling

Don’t blow up if the db/ folder doesn’t exist, detect it and give the user a friendly message.

Improve --create option to generate more than just model adds

from jezdez: check out sqldiff we worked on for django-extensions

comparedb omits the database name as suffix for createdb command

The default createdb command for comparedb is:

command = "createdb %s" % self.name

But if the settings define a createdb command, it should be suffixed by the name too, which is not the case in:

command = settings.NASHVEGAS["createdb"]

https://github.com/paltman/nashvegas/blob/master/nashvegas/management/commands/comparedb.py#L28

Docs suggest that YYYYMMDD-01 is a valid identifier, but it is not.

identifier splits based on _ and then tries to cast to int.

Issue with versioning

It seems the version is being recorded as "0.8" in setup.py, when it should be at the very least, 0.8.0

Use python logging instead of print statements

[enhancement] NASHVEGAS_MIGRATIONS_PATH setting

It is currently possible to override the path to the migration dir with the -p option on commands. But it can be a little boring if you have to use it all the time so why not allow default migrations path override once-and-for-all via a setting like NASHVEGAS_MIGRATIONS_PATH.

Missing rollback feature

There is no way to rollback database schema. Sometimes you want step back to fix change set.

Make PEP 8 Compliant

This code could really use a cleanup.

Add ability to seed the Migration model with migrations already applied.

This solves the problem of installing nashvegas in a project that is already underway.

./manage.py upgradedb --seed migrations/*

Add option for running fixtures

if migrations are added then initial data migrations are loaded. however, there are times that you always want initial data to load (you updated fixtures but no migrations).

bug in migrations execution

There is a major bug in migrations execution. A set of migrations is executed in one transaction. Changes have to be commited in every iteration (for every migration/changeset).

Just try altering table structure by adding, then dropping columns, and try to set not null constraint on other field in same table (postgresql). There is no chance to execute that migration script. Splitting migration into separate files does not work.

Example:

alter table renting_request add column return_date date;
update renting_request set return_date = rent_date + period;
alter table renting_request drop column period;
alter table renting_request alter column return_date set not null;

Add ability to execute python scripts in addition to sql scripts

Nashvegas currently only will execute sql scripts against the database connection.

Adding support to execute arbitrary python scripts as well gives the tool a lot more flexibility. For example, doing one time data manipulation across a number of models, can be a lot easier to express in python using Models, rather than a bunch of SELECT and UPDATE and INSERT statements cobbled together in SQL.

Fix bug where migrations succeed but loading initial_data fixture fails

The bug is I don't think that should rollback migrations, a failure to load fixtures is independent from migrating schema in my opinion.

In addition, it appears that a failure to load fixtures is returning a 0 error code instead of non-zero thereby prohibiting proper response handling from tools that expect unix-standard behavior.

[Proposal] Replace syncdb with upgradedb --execute

Wanted to open this for discussion. I'd like to simplify the upgradedb process, and I can't think of any reason why you'd need syncdb if you're using nashvegas.

It's not clear in the docs that --seed 5 runs 1, 2, 3, and 4 but not 5.

I'm not sure if that is intentional or an off by one error.

Calling Django's version of syncdb should still be possible

Since bf38d07, calling syncdb with nashvegas installed calls upgradedb. When creating a new DB from a project that has a lot of migrations, the python migrations which use the project's models are not necessarily in sync with the DB at the time of the migration (for instance, different field names between the DB and the python models), and the project may be in a stage where the database can't even be sync'd.

The larger issue is forward-compatibility of migrations but this is not always possible. Anyway, the migration process should be:

• if the DB schema already exists, do upgradedb -x
• if the DB schema doesn't exist (1st deployment), use Django's version of syncdb and seed the migrations since there is no need to go through the whole migration process on an empty DB

Nashvegas used to warn the user when trying to run syncdb with nashvegas installed, and the usual workflow was to run upgradedb + syncdb on all deploys. This could work very well here, and upgradedb could do the following:

Check if the DB is being synced for the 1st time. If so: do syncdb + seed migrations
Else: upgradedb -x && syncdb

This way the user only uses upgradedb, gets a big fat warning when trying to use syncdb directly and upgradedb calls syncdb at the appropriate time.

Thoughts?
#46 implements the upgrade case but not the 1st sync case.

Create Versions Model

Currently, nashvegas uses a table called Versions, which is created via a SQL script. Making it model based makes it more django-ish as well as providing the ability to attach methods and reference it within an application.

Initial create throws exception

Ran a simple test to create some tables, all the tables got created, but got the following error:

Any insight as to why the connection is getting 'lost'?

domU-12-31-39-0E-26-41:/var/www/publisher/infographs> python ./manage.py upgradedb --execute --path etc/db/migrations/
Executing 0001_initial.sql... success
Emitting post sync signal.
Exception _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query') in <bound method Cursor.del of <MySQLdb.cursors.Cursor object at 0x1f3c990>> ignored
Traceback (most recent call last):
File "./manage.py", line 11, in
execute_manager(settings)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(_args, *_options.dict)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/core/management/base.py", line 220, in execute
output = self.handle(_args, *_options)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/nashvegas/management/commands/upgradedb.py", line 297, in handle
self.execute_migrations(show_traceback=True)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/db/transaction.py", line 340, in _commit_manually
leave_transaction_management(using=db)
File "/home/build/.virtualenv/publisher/lib/python2.6/site-packages/django/db/transaction.py", line 87, in leave_transaction_management
raise TransactionManagementError("Transaction managed block ended with pending COMMIT/ROLLBACK")
django.db.transaction.TransactionManagementError: Transaction managed block ended with pending COMMIT/ROLLBACK

ignore hidden files in db directory

If there are hidden files in the db directory, nashvegas will fail expecting migration file names to start with a digit.

See my fork for a patch.

Tests break when nashvegas is enabled.

Version: nashvegas==0.8b1

When I have nashvegas in my installed apps, my tables don't seem to get created. I have one .py migration which is a data migration.

./manage.py test streamer
Creating test database for alias 'default'...
Got an error creating the test database: (1007, "Can't create database 'test_gitstreams'; database exists")
Type 'yes' if you would like to try deleting the test database 'test_gitstreams', or 'no' to cancel: yes
Destroying old test database 'default'...
Executing migration '0001_populate_activity_repo.py' on 'default'....Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/commands/test.py", line 49, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/commands/test.py", line 72, in handle
    failures = test_runner.run_tests(test_labels)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/test/simple.py", line 381, in run_tests
    old_config = self.setup_databases()
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/test/simple.py", line 317, in setup_databases
    self.verbosity, autoclobber=not self.interactive)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/backends/creation.py", line 271, in create_test_db
    load_initial_data=False)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/__init__.py", line 150, in call_command
    return klass.execute(*args, **defaults)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/nashvegas/management/commands/syncdb.py", line 18, in handle_noargs
    verbosity=options.get("verbosity"),
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/__init__.py", line 150, in call_command
    return klass.execute(*args, **defaults)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 421, in handle
    self.execute_migrations()
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 298, in execute_migrations
    show_traceback=show_traceback
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 177, in _execute_migration
    execfile(migration, {}, module)
  File "/Users/justinlilly/src/hacks/gitstreams/gitstreams/migrations/0001_populate_activity_repo.py", line 7, in <module>
    total = qs.count()
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/models/query.py", line 351, in count
    return self.query.get_count(using=self.db)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/models/sql/query.py", line 418, in get_count
    number = obj.get_aggregation(using=using)[None]
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/models/sql/query.py", line 384, in get_aggregation
    result = query.get_compiler(using).execute_sql(SINGLE)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
    cursor.execute(sql, params)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 114, in execute
    return self.cursor.execute(query, args)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/Users/justinlilly/.virtualenvs/gitstreams/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.DatabaseError: (1146, "Table 'test_gitstreams.streamer_activity' doesn't exist")

Do I need to create initial schema migrations w/ nashvegas? The docs didn't really make it clear. I thought it would basically be syncdb + any changes that need to happen.

Better Error message when migration can not be loaded

I only have migrations that start with numbers (0042_something - 0050_something) but I'm still getting this message:

Traceback (most recent call last):
  File "manage.py", line 27, in <module>
    execute_from_command_line()
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 272, in handle
    self.execute_migrations(show_traceback=True)
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/django/db/transaction.py", line 338, in _commit_manually
    return func(*args, **kw)
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 135, in execute_migrations
    migrations = self._filter_down()
  File "/Users/issackelly/virtualenvs/po2-winwin/lib/python2.7/site-packages/nashvegas/management/commands/upgradedb.py", line 79, in _filter_down
    raise MigrationError("Invalid migration file prefix (must begin with a number)")
nashvegas.management.commands.upgradedb.MigrationError: Invalid migration file prefix (must begin with a number)

I'd really like to see what it's trying to parse, which migration, etc, that it can't determine is a number.

Initial schema followed by migrations in MySQL

I'm stabbing in the dark here but I can't get a seemingly simple mysql set to work at all with nashvegas. It gets tangled in transaction juggling.

I have

migrations/000_initial_schema.sql
migrations/001_some_alter_table.sql

And this doesn't work. There are no BEGIN or COMMIT or anything like that in any of the .sql files. As output I get this:

Exception _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query') in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor object at 0x102554ad0>> ignored
Executing 0001_webby-stage-auth_bug653180.sql... 

So, I thought perhaps it's not right that the cursor is re-created in each iteration of the loop so I moved it outside and tried again. Now I get this:

Rolled back all migrations.
Traceback (most recent call last):
  File "./manage.py", line 41, in <module>
    execute_manager(settings)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/private/tmp/elmos/1/elmo/vendor-local/lib/python/nashvegas/management/commands/upgradedb.py", line 309, in handle
    self.execute_migrations(show_traceback=True)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/transaction.py", line 217, in inner
    res = func(*args, **kwargs)
  File "/private/tmp/elmos/1/elmo/vendor-local/lib/python/nashvegas/management/commands/upgradedb.py", line 184, in execute_migrations
    cursor.execute(to_execute)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/backends/util.py", line 34, in execute
    return self.cursor.execute(sql, params)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)
  File "/Users/peterbe/virtualenvs/elmo/lib/python2.6/site-packages/MySQLdb/cursors.py", line 154, in execute
    charset = db.character_set_name()
django.db.utils.DatabaseError: (2014, "Commands out of sync; you can't run this command now")
Exception _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query') in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor object at 0x102540ad0>> ignored

One last time, I removed all my debugging modifications from upgradedb.py and moved away all files (migrations/*.sql) except the one called 000_initial_schema.sql and then I get this error:

Executing 0000_initial_schema.sql... Rolled back all migrations.
Traceback (most recent call last):
  File "./manage.py", line 41, in <module>
    execute_manager(settings)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/private/tmp/elmos/1/elmo/vendor-local/lib/python/nashvegas/management/commands/upgradedb.py", line 300, in handle
    self.execute_migrations(show_traceback=True)
  File "/private/tmp/elmos/1/elmo/vendor-local/lib/python/nashvegas/management/commands/upgradedb.py", line 192, in execute_migrations
    transaction.commit(using=self.db)
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/transaction.py", line 142, in commit
    connection.commit()
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/backends/__init__.py", line 201, in commit
    self._commit()
  File "/private/tmp/elmos/1/elmo/vendor/src/django/django/db/backends/__init__.py", line 46, in _commit
    return self.connection.commit()
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
Exception _mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now") in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor object at 0x102554990>> ignored

I should add; my file (000_initial_schema.sql) is pretty big and it creates indexes and a bunch of other things.

Add ContentType Creation/Update

`./manage.py syncdb` called functions to update and/or create records in the content types table which drives a lot of under the covers functionality (e.g. Generic Relations).

Add hooks to perform the same function calls as syncdb.

Remove any BEGIN/COMMIT from output

transactions are controlled via nashvegas so don't generate commit messages on --create output

Add option for running fixtures

if migrations are added then initial data migrations are loaded. however, there are times that you always want initial data to load (you updated fixtures but no migrations).

upgradedb --execute failing on SQLite

Hi,

I'm trying to set up Nashvegas in a project that is still in development, using SQLite.

I started by removing my current database file, and running:

django-admin.py upgradedb --create  > migrations/0001.sql

To get a fresh migration to start with. I then tried running:

django-admin.py upgradedb --execute --path migrations

This gives the following error:

Executing 0001.sql... failed
Traceback (most recent call last):
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/nashvegas/management/commands/upgradedb.py", line 183, in execute_migrations
    cursor.execute(to_execute)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/db/backends/util.py", line 34, in execute
    return self.cursor.execute(sql, params)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/db/backends/sqlite3/base.py", line 234, in execute
    return Database.Cursor.execute(self, query, params)
Warning: You can only execute one statement at a time.
Rolled back all migrations.
Traceback (most recent call last):
  File "/Users/jamie/.virtualenvs/dabapps/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/nashvegas/management/commands/upgradedb.py", line 297, in handle
    self.execute_migrations(show_traceback=True)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/django/db/transaction.py", line 217, in inner
    res = func(*args, **kwargs)
  File "/Users/jamie/.virtualenvs/dabapps/lib/python2.6/site-packages/nashvegas/management/commands/upgradedb.py", line 188, in execute_migrations
    raise MigrationError()
nashvegas.management.commands.upgradedb.MigrationError

The error being returned from the database backend seems to be:

    Warning: You can only execute one statement at a time.

I'm not really sure where to start with figuring this out.. Google doesn't seem to be helping much. Any ideas?

Thanks