3yourmind/django-replace-migrations

django-replace-migrations

This package is an extension to djangos makemigrations.py. It can be used to get rid of old migrations as an alternative to djangos squashmigration command.

Reasoning

In big django projects, migration files easily pile up and get an increasing problem. Django comes with the squashmigration command - however, it is hard to handle because of multiple reasons. Especially, it can not handle circular dependencies - they must be resolved manually and with great care.

One possible solution is to:

• Delete all existing migrations
• Run ./manage.py makemigrations, so that it creates new initial migrations
• Run ./manage.py migrate --fake [new migrations] or ./manage.py migrate --fake-initail on all servers.

This workflow might work fine, if you have only few (production) servers - however, it becomes hard, when you have many environments with different versions of your application.

With django-replace-migrations also creates new initial migrations, but also, additionally, adds the already existing migrations to the replace list of the new migration (That list is used by squashmigrations as well). By doing that, faking migrations is not needed anymore.

Warning

The new replacing migrations will not consider any RunPython or RunSQL operations. That might be acceptable depending on your use of those operations and if you need those to prepare a fresh database.

Installation

Before you install, read the workflow below. You need to have the app installed in your project only on a specific branch temporarily.

Run

pip install django-replace-migrations

and add django_replace_migrations to your list of installed apps.

Simple Workflow

If your apps are not depending on each other, you can use django-replace-migrations like this:

./manage.py makemigratons --replace-all --name replace [app1, app2, ...]

Note, that you will need to list all of your apps explicitly - otherwise django will also try to replace migrations from dependencies. While --name could be omitted, it is highly recommended to use it so that you can easily recognize the new migrations.

If for any of your apps there are not one but two or more migrations created, your apps are depending on each other (see below).

You can leave your old migrations in the codebase. Old versions will continue to use the old migrations, while fresh installations will use the newly created replace migration instead.

If you remove the old migrations later, you will need to update the dependencies in your other migrations and replace all occurrences of the old migration with the new replace migration. You can easily do that with try-and-error (migrate will fail and tell you which dependency is missing)

Workflow for depending apps

Due to the way django resolves the replace list, it can not handle circular dependencies within apps. To prevent an error during migration, you must delete the old migrations that you replaced.

If you have your application deployed on multiple servers, you must define down to which version, you will support upgrading and only replace those migrations.

Let’s assume that our current version of the application is 3.0 and we want to get rid of all migrations prior to 2.0.

The workflow for this would be:

• git checkout 2.0
• create a new branch git checkout -b 2-0-delete-migrations
• delete all existing migrations in your apps
• commit and note the commit hash
• git checkout 2.0
• create a new branch git checkout -b 2-0-replace-migrations
• Install django-replace-migration here.
• run ./manage.py makemigrations --replace-all --name replace_2_0 app1, app2, ... (How to get all apps)
• commit and note the commit hash
• git checkout [your main/feature branch]
• git cherry-pick [commit-hash from 2-0-delete-migrations]
• git cherry-pick [commit-hash from 2-0-replace-migrations]
• Go over every migration in your app which was not replaced, and check the dependencies array. You need to make sure that every dependency listed was not replaced in the process, and the array is not out of date. If that's the case, then you need to specify the newest replaced migration instead. For example, in the app A there were two replace migrations created 0001_replace, 0002_replace. Migration 0026 is referencing migration 00015, both from the app A, but 0015 was replaced. You need to swap 0015 with 0002_replace.

Now you have all migrations prior to 2.0 removed and replaced with new migrations.

Consequences

If your app is below 2.0 and you want to update to something after 2.0, you first need to update to 2.0

• upgrading from 1.0 to 1.5 will be possible
• upgrading from 2.0 to 3.0 will be possible
• upgrading from 1.0 to 3.0 will be not possible

makemigration.py compatibility

This package requires deep integration into makemigrations.py so that I needed to copy the whole makemigrations.py here. Currently the version of makemigrations.py is copied from Django 2.1, however it is also tested with Django 3.0 and works there as well. If you encounter problems, please write what version of Django you are using.

django-replace-migrations's Issues

ModuleNotFoundError: No module named 'django_migration_linter'

Prepare environment

Docker is the fastest/easiest way to get a quick disposable environment:

$ docker exec -it --rm python bash

Install dependencies

# pip install django django-replace-migrations
# django-admin startproject myapp .

Add "django_migration_linter" to INSTALLED_APPS in settings.py

Error

Running ./manage.py showmigrations or ./manage.py makemigrations or anything really I'll get the same error: ModuleNotFoundError: No module named 'django_migration_linter'

I tried replacing "django_replace_migrations" with "django_replace_migrations" but nothing works.
Shoot, I even tried this:

# python
Python 3.9.1 (default, Dec 18 2020, 05:16:04) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django_migration_linter
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'django_migration_linter'

So, what is the name of this package I should import? Is it django_migration_linter or possibly django_replace_migrations?

Missing top level package

When I install, this is the layout

/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/PKG-INFO
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/SOURCES.txt
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/dependency_links.txt
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/requires.txt
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/top_level.txt
/usr/lib/python3.8/site-packages/django_replace_migrations-0.0.1-py3.8.egg-info/version.txt
/usr/lib/python3.8/site-packages/management
/usr/lib/python3.8/site-packages/management/__init__.py
/usr/lib/python3.8/site-packages/management/__pycache__
/usr/lib/python3.8/site-packages/management/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/management/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/management/commands
/usr/lib/python3.8/site-packages/management/commands/__init__.py
/usr/lib/python3.8/site-packages/management/commands/__pycache__
/usr/lib/python3.8/site-packages/management/commands/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/management/commands/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/management/commands/__pycache__/makemigrations.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/management/commands/__pycache__/makemigrations.cpython-38.pyc
/usr/lib/python3.8/site-packages/management/commands/__pycache__/replacemigrations.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/management/commands/__pycache__/replacemigrations.cpython-38.pyc
/usr/lib/python3.8/site-packages/management/commands/makemigrations.py
/usr/lib/python3.8/site-packages/management/commands/replacemigrations.py
/usr/share/doc/packages/python3-django-replace-migrations
/usr/share/doc/packages/python3-django-replace-migrations/README.md
/usr/share/licenses/python3-django-replace-migrations
/usr/share/licenses/python3-django-replace-migrations/LICENSE

There is a missing package name , under which management should be.

I get a KeyError when I run manage.py makemigrations --replace-all (...)

I'm unable to follow the step where I makemigrations --replace-all.

Steps I took

Created a new branch, deleted ALL migrations and committed changes.

Created another branch. Used:

>>> from django.apps import apps
>>> print(" ".join(map(str, sorted({model._meta.app_label for model in apps.get_models()}))))
'accounts admin authtoken contenttypes sessions token (the rest are apps in my project)'

Ran

python manage.py makemigrations --replace-all --name replace_old_migration (everything from previous script)

All at once:

python \
   manage.py \
   makemigrations \
   --replace-all \
   --name replace_old_migrations \
   $(python manage.py shell -c "from django.apps import apps; print(' '.join(map(str, sorted({model._meta.app_label for model
in apps.get_models()}))))")

Error

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django_replace_migrations/management/commands/makemigrations.py", line 154, in handle
    loader.project_state(),
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 324, in project_state
    return self.graph.make_state(nodes=nodes, at_end=at_end, real_apps=list(self.unmigrated_apps))
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/graph.py", line 315, in make_state
    project_state = self.nodes[node].mutate_state(project_state, preserve=False)
KeyError: ('contenttypes', '0001_initial')

The previous script is only for convenience. I manually wrote out the makemigrations line with all the app listed and I still got the same error. I removed each app that the key error was complaining about until I was down to like three apps before it happily ran.

Other strategies attempted

Even if I add if model._meta.app_label not in {'admin', 'auth', 'authtoken', 'contenttypes', 'sessions', 'token'} filter (apps which are not mine) I still get a KeyError, just with a different app.

Many of my apps depend on other apps, and many of those apps depend on each other.
Do I need to separate squashing into separate groups/batches? How would I tell which apps go in which group/batch?

Versions

Python: 3.6.8
Django: 2.2.1
django-replace-migrations: 0.0.2