Upgrade from 2024.4.1 to >= 2024.4.2 Raises issue on SCIM users primary key.

[gcarrarom]

Upgrade from 2024.4.1 to >= 2024.4.2 Raises issue on SCIM users primary key.

[BeryJu]

Could you try to upgrade to 2024.4.1 and then to 2024.4.2? We test version upgrades as part of our CI pipeline, however we only test upgrades from the immediate previous version

[gcarrarom]

Thanks for the heads-up on that! I've upgraded without issues to 2024.4.1, but from 2024.4.1 to 2024.4.2 it fails with the same issue:

"pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:08.221343"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.signals", "pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:08.221450"}
=== Starting migration
Operations to perform:
  Apply all migrations: auth, authentik_blueprints, authentik_brands, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_rac, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_sources_scim, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_source, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions
Running migrations:
  Applying authentik_providers_scim.0007_scimgroup_scim_id_scimuser_scim_id_and_more...

Fixing primary key for SCIM users, this might take a couple of minutes...

Fixing primary key for SCIM groups, this might take a couple of minutes...
{"domain_url": null, "event": "releasing database lock", "level": "info", "logger": "lifecycle.migrate", "pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:10.671378"}
Failed to read config file: ./lifecycle/gunicorn.conf.py
Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/gunicorn/app/base.py", line 111, in get_config_from_filename
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/lifecycle/gunicorn.conf.py", line 147, in <module>
    run_migrations()
  File "/lifecycle/migrate.py", line 113, in run_migrations
    execute_from_command_line(["", "migrate_schemas"])
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/management/commands/migrate_schemas.py", line 63, in handle
    executor.run_migrations(tenants=[self.PUBLIC_SCHEMA_NAME])
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/standard.py", line 11, in run_migrations
    run_migrations(self.args, self.options, self.codename, self.PUBLIC_SCHEMA_NAME)
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/base.py", line 59, in run_migrations
    migrate_command_class(stdout=stdout, stderr=stderr).execute(*args, **options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
    alter_together(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 594, in alter_unique_together
    self._delete_composed_index(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 658, in _delete_composed_index
    self.execute(self._delete_constraint_sql(sql, model, constraint_names[0]))
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/__init__.py", line 644, in execute
    result = real_execute(self, sql, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist

[gcarrarom]

Anyone? Same issue trying to go to 2024.6.0...

[PracticalRyan]

I'm using a regular docker compose file upgrading from 2024.4.3 to 2024.6.0 and I'm getting a similar error related to Failed to read config file: ./lifecycle/gunicorn.conf.py I have already followed the PostgreSQL 16 update steps for 2024.6.0, but I'm still having issues. Reverting back works totally fine though but only breaks when updating.

Not sure if it's related though.

Error log

server-1      | {"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1720301419.2265337, "file": "/authentik/lib/default.yml"}
server-1      | {"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1720301419.2269688, "count": 15}
server-1      | {"error":"authentik starting","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2024-07-06T21:30:19Z"}
server-1      | {"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1720301420.3206322}
server-1      | {"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1720301420.3330467}
server-1      | {"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1720301420.3346312}
server-1      | {"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1720301420.334769}
server-1      | 2024-07-06 21:30:20 [info     ] waiting to acquire database lock
server-1      | 2024-07-06 21:30:20 [info     ] applying django migrations
server-1      | 2024-07-06 21:30:22 [info     ] releasing database lock
server-1      | Failed to read config file: ./lifecycle/gunicorn.conf.py
server-1      | Traceback (most recent call last):
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/gunicorn/app/base.py", line 111, in get_config_from_filename
server-1      |     spec.loader.exec_module(mod)
server-1      |   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
server-1      |   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
server-1      |   File "/lifecycle/gunicorn.conf.py", line 154, in <module>
server-1      |     run_migrations()
server-1      |   File "/lifecycle/migrate.py", line 112, in run_migrations
server-1      |     execute_from_command_line(["", "migrate_schemas"])
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
server-1      |     utility.execute()
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 382, in execute
server-1      |     settings.INSTALLED_APPS
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 89, in __getattr__
server-1      |     self._setup(name)
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 76, in _setup
server-1      |     self._wrapped = Settings(settings_module)
server-1      |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
server-1      |   File "/ak-root/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 190, in __init__
server-1      |     mod = importlib.import_module(self.SETTINGS_MODULE)
server-1      |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1      |   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
server-1      |     return _bootstrap._gcd_import(name[level:], package, level)
server-1      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1      |   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
server-1      |   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
server-1      |   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
server-1      |   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
server-1      |   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
server-1      |   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
server-1      |   File "/authentik/root/settings.py", line 317, in <module>
server-1      |     for replica in CONFIG.get_keys("postgresql.read_replicas"):
server-1      |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1      |   File "/authentik/lib/config.py", line 311, in get_keys
server-1      |     return attr.keys()
server-1      |            ^^^^^^^^^
server-1      | AttributeError: 'Attr' object has no attribute 'keys'
server-1      | {"error":"exit status 1","event":"gunicorn process died, restarting","level":"warning","logger":"authentik.router","timestamp":"2024-07-06T21:30:22Z"}
server-1      | {"error":"exit status 1","event":"gunicorn failed to start, restarting","level":"error","logger":"authentik.router","timestamp":"2024-07-06T21:30:22Z"}

[gcarrarom]

Yeah, doesn't seem to be the same issue. I'm running PSQL 16 as well, version 16.3. Could it be something in the version of PostGreSQL?

[PracticalRyan]

I have no idea what that issue was, but loosely following this guide. I was able to fix the issue. I first created a dump of the database, downed and deleted all containers, create temporary bind mounts to empty directories. (At first I just deleted the DB volume but it still errored, so it had to be more than just the DB, which is why I deleted everything from the containers.)

After doing all this, docker compose up and you'll get the initial Authentik instance with no data whatsoever. Now that it runs on the latest version, I moved back all my bind mounts to their original places, restarted, and everything still worked! But since all my data was gone, I just applied the DB dump to the empty database and now it runs normally.

[gcarrarom]

Alright, thanks for letting me know. I've fixed this issue by manually doing the migrations from:
https://github.com/goauthentik/authentik/tree/main/authentik/providers/scim/migrations

I don't know how that migration was broken, but manually migrating tables and altering them in a DB is not ideal.
Once I've done the steps on those python migration scripts, and I've added the migration names in the django_migrations table, I now have version 2024.6.1 up and running.

I do not recommend someone else doing this without understanding how those scripts happen and how they are supposed to run. But it did work.

I've tried running a new version of authentik in a fresh DB and migrating the data over, but the number of constraints and the new IDs were way too much for me to migrate over. I'll close for now, but I'm not happy that I had to do this.