django-configurations and django-celery > 3

[jazzband]

django-configurations and django-celery > 3

[jezdez]

How are you starting celery? If you've correctly modified your manage.py and use manage.py celery worker the importer should be installed automatically.

If that still fails, try adding 'configurations.management' to the CELERY_IMPORTS configuration (http://celery.github.com/celery/configuration.html#celery-imports) as importing that module installs the importer automatically.

[jeanlucmongrain]

my manage.py is this:

import os

if __name__ == '__main__':
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings.local')
    os.environ.setdefault('DJANGO_CONFIGURATION', 'ProjectSettings')
    from configurations.management import execute_from_command_line
    execute_from_command_line(sys.argv)

and I launch celery with a simple manage.py celeryd -B

I tried the CELERY_IMPORTS trick and no luck.

just to see, I tried adding it to CELERY_INCLUDE too.

the problem is no celeryd itself, but the the sub-processes it create, here is a part of the console output, you can see that the settings are properly initialize, because I can see django db logs.

but it fail after billard spawn processes.

2012-09-23 14:29:44,369 DEBUG    django.db.backends util.execute:44 (0.001) SELECT "djkombu_message"."id", "djkombu_message"."visible", "djkombu_message"."sent_at", "djkombu_message"."payload", "djkombu_message"."queue_id" FROM "djkombu_message" WHERE ("djkombu_message"."queue_id" = 1  AND "djkombu_message"."visible" = true ) ORDER BY "djkombu_message"."sent_at" ASC, "djkombu_message"."id" ASC LIMIT 1; args=(1, True)
2012-09-23 14:29:44,369 DEBUG    django.db.backends util.execute:44 (0.001) SELECT "djkombu_message"."id", "djkombu_message"."visible", "djkombu_message"."sent_at", "djkombu_message"."payload", "djkombu_message"."queue_id" FROM "djkombu_message" WHERE ("djkombu_message"."queue_id" = 1  AND "djkombu_message"."visible" = true ) ORDER BY "djkombu_message"."sent_at" ASC, "djkombu_message"."id" ASC LIMIT 1; args=(1, True)
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/billiard/forking.py", line 525, in main
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/billiard/forking.py", line 525, in main
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/billiard/forking.py", line 525, in main
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/billiard/forking.py", line 525, in main
    self = load(from_parent)
  File "/sw/lib/python2.7/pickle.py", line 1378, in load
    self = load(from_parent)
  File "/sw/lib/python2.7/pickle.py", line 1378, in load
    self = load(from_parent)
  File "/sw/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/sw/lib/python2.7/pickle.py", line 858, in load
    return Unpickler(file).load()
  File "/sw/lib/python2.7/pickle.py", line 858, in load
    return Unpickler(file).load()
  File "/sw/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/sw/lib/python2.7/pickle.py", line 1133, in load_reduce
    dispatch[key](self)
  File "/sw/lib/python2.7/pickle.py", line 1133, in load_reduce
    dispatch[key](self)
  File "/sw/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 129, in _unpickle_app
    return pickler()(cls, *args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 108, in __call__
    value = func(*args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 129, in _unpickle_app
    self.prepare(app, **kwargs)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 112, in prepare
    return pickler()(cls, *args)
  File "/Users/patate/sandbox/sg/boilerplate/env/lib/python2.7/site-packages/celery/app/utils.py", line 108, in __call__
    app.conf.update(kwargs['changes'])

followed by a lot more traceback and the django-configurations ImproperlyConfigured error.

[jeanlucmongrain]

just to make it easier to troubleshoot this, I created this project that reproduce the problem:

https://github.com/bclermont/django-celery-issue186

[jeanlucmongrain]

Asksol said that using entry-point could make that work: celery/django-celery#186 (comment)

Any idea on how to implement this?

Is that something django-configurations should do, or at least, provide an easy way (such as a function plugged somewhere) to cover those cases.

[Lacrymology]

I don't know if this is a celery or configurations thing, but if on top of the settings file (before from configurations import Settings) you do this:

from configurations import importer
if not importer.installed:
    importer.install()

it gets fixed. The thing is that celery 3.0's forks/threads don't load manage.py so modifying it doesn't do the trick

[Lacrymology]

I just checked, and adding this to the settings file doesn't remove the need of running the same code on manage.py

[apmorton]

for the life of me I cannot make this workaround work..

I have even tried creating a fresh virtualenv and installing localshop (which uses django-configurations + django-celery) and even that suffers a similar issue.

I'm not sure if the issue is related to windows, but thats the only thing I can think up at this point. I will test on a linux box later tonight to be sure.

(labman-ny) PS> python manage.py celeryd -c 1 --no-execv
[<SettingsImporter for 'labman_ny.settings.Dev'>]

 -------------- celery@xxxxx v3.0.19 (Chiastic Slide)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker:      django://localhost//
- ** ---------- .> app:         default:0x39ceef0 (djcelery.loaders.DjangoLoader)
- ** ---------- .> concurrency: 1 (processes)
- *** --- * --- .> events:      OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery:      exchange:celery(direct) binding:celery


C:\Users\amorton\Envs\labman-ny\lib\site-packages\djcelery\loaders.py:133: UserWarning: Using settings.DEBUG leads to a
memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2013-04-29 15:23:38,280: WARNING/MainProcess] C:\Users\amorton\Envs\labman-ny\lib\site-packages\djcelery\loaders.py:133
: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '

[2013-04-29 15:23:38,282: WARNING/MainProcess] celery@xxxxx ready.
[<SettingsImporter for 'labman_ny.settings.Dev'>]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\billiard\forking.py", line 530, in main
    self = load(from_parent)
  File "C:\Python27\Lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\Lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\Lib\pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\utils.py", line 140, in _unpickle_app
    return pickler()(cls, *args)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\utils.py", line 117, in __call__
    self.prepare(app, **kwargs)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\utils.py", line 121, in prepare
    app.conf.update(kwargs['changes'])
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\kombu\utils\__init__.py", line 294, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\base.py", line 494, in conf
    return self._get_config()
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\base.py", line 358, in _get_config
    s = Settings({}, [self.prepare_config(self.loader.conf),
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\kombu\utils\__init__.py", line 294, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\app\base.py", line 506, in loader
    return get_loader_cls(self.loader_cls)(app=self)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\celery\loaders\__init__.py", line 23, in get_loader_cls
    return symbol_by_name(loader, LOADER_ALIASES)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\kombu\utils\__init__.py", line 80, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "C:\Python27\Lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\djcelery\loaders.py", line 15, in <module>
    from django import db
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\django\db\__init__.py", line 11, in <module>
    if settings.DATABASES and DEFAULT_DB_ALIAS not in settings.DATABASES:
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\django\conf\__init__.py", line 53, in __getattr__
    self._setup(name)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\django\conf\__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "C:\Users\amorton\Envs\labman-ny\lib\site-packages\django\conf\__init__.py", line 152, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
[2013-04-29 15:23:39,079: ERROR/MainProcess] Process 'PoolWorker-1' pid:5936 exited with exitcode 1

Running other manage.py commands works fine, so the issue is isolated to celery workers

python manage.py celery report also works fine, and produces the correct settings for the configuration specified

Although on a related note, you cannot pass --configuration to any python manage.py celery <sub> command

[apmorton]

pardon my stupidity, but after playing with just a few more options and looking into the source, the issue is indeed related to celery on windows.

the default pool method (processes) apparently wipes out whatever setup was done on windows specifically.

manually specifying -P solo works perfectly.

I would imagine any of the others (except processes) should work if you have the required additional modules installed.

[coagulant]

I'm stuck with latest celeryd_detach not working with django-configurations.

Celery workers are started with default celery settings, so I'm getting this in logs:

[2013-07-31 14:59:15,126: ERROR/MainProcess] consumer: Cannot connect to amqp://[email protected]:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds...

Here how I get this behavior (this command gets executed to launch celery detached according to celery docs):

/home/lava/envs/myproject/bin/python /home/lava/src/myproject/manage.py celery multi start myproject --uid=celery --workdir=/home/myuser/src/myproject --pidfile="/var/run/celery/celeryd_%n.pid" --logfile="/var/log/celery/celeryd_%n.log" --loglevel="DEBUG" --cmd="-m celery.bin.celeryd_detach"

I've got the discussed above import hook enabled in myproject/myproject/settings.py, however this doesn't help.

from configurations import importer
importer.install()

Note, that running via manage.py works as expected:

python manage.py celeryd_detach --loglevel=DEBUG --logfile=celeryd_v_test.log

My guess is that djcelery.loaders.DjangoLoader somehow is not working nice with my setup.

I'll try to provide reproducable example a bit later, just wanted to share my issue in case someone already had faced something like that.

[rizumu]

I am working on a project where importlib.import_module is being used in a few places and I am hitting these errors when switching over to django-configurations. One call was local so I simply moved its call out of the manage.py where it was previously and into to the urls.py, but the other is in a 3rd party app. https://github.com/pinax/django-user-accounts/blob/master/account/conf.py#L13

I already have importer.install() at the top of my settings file for celery. Any suggestions on how to resolve this, in the 3rd party app, locally, or within django-configurations? Maybe the 1.7 AppConfig.ready will provide a solution? https://docs.djangoproject.com/en/dev/ref/applications/#django.apps.AppConfig.ready