celery/billiard

Multiprocessing Pool Extensions

Version

5.4.0 (opalescent)

Web

https://docs.celeryq.dev/en/stable/index.html

Download

https://pypi.org/project/celery/

Source

https://github.com/celery/celery/

Keywords

task, queue, job, async, rabbitmq, amqp, redis, python, distributed, actors

Donations

This project relies on your generous donations.

If you are using Celery to create a commercial product, please consider becoming our backer or our sponsor to ensure Celery's future.

For enterprise

Available as part of the Tidelift Subscription.

The maintainers of celery and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use. Learn more.

What's a Task Queue?

Task queues are used as a mechanism to distribute work across threads or machines.

A task queue's input is a unit of work, called a task, dedicated worker processes then constantly monitor the queue for new work to perform.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.

Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there's node-celery for Node.js, a PHP client, gocelery, gopher-celery for Go, and rusty-celery for Rust.

Language interoperability can also be achieved by using webhooks in such a way that the client enqueues an URL to be requested by a worker.

What do I need?

Celery version 5.3.5 runs on:

• Python (3.8, 3.9, 3.10, 3.11, 3.12)
• PyPy3.9+ (v7.3.12+)

This is the version of celery which will support Python 3.8 or newer.

If you're running an older version of Python, you need to be running an older version of Celery:

• Python 3.7: Celery 5.2 or earlier.
• Python 3.6: Celery 5.1 or earlier.
• Python 2.7: Celery 4.x series.
• Python 2.6: Celery series 3.1 or earlier.
• Python 2.5: Celery series 3.0 or earlier.
• Python 2.4: Celery series 2.2 or earlier.

Celery is a project with minimal funding, so we don't support Microsoft Windows but it should be working. Please don't open any issues related to that platform.

Celery is usually used with a message broker to send and receive messages. The RabbitMQ, Redis transports are feature complete, but there's also experimental support for a myriad of other solutions, including using SQLite for local development.

Celery can run on a single machine, on multiple machines, or even across datacenters.

Get Started

If this is the first time you're trying to use Celery, or you're new to Celery v5.4.x coming from previous versions then you should read our getting started tutorials:

• First steps with Celery

  Tutorial teaching you the bare minimum needed to get started with Celery.

• Next steps

  A more complete overview, showing more features.

You can also get started with Celery by using a hosted broker transport CloudAMQP. The largest hosting provider of RabbitMQ is a proud sponsor of Celery.

Celery is...

• Simple

  Celery is easy to use and maintain, and does not need configuration files.

  It has an active, friendly community you can talk to for support, like at our mailing-list, or the IRC channel.

  Here's one of the simplest applications you can make:

  from celery import Celery
  
  app = Celery('hello', broker='amqp://guest@localhost//')
  
  @app.task
  def hello():
      return 'hello world'

• Highly Available

  Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Primary/Primary or Primary/Replica replication.

• Fast

  A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ, py-librabbitmq, and optimized settings).

• Flexible

  Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, broker transports, and much more.

It supports...

• Message Transports

  • RabbitMQ, Redis, Amazon SQS

• Concurrency

  • Prefork, Eventlet, gevent, single threaded (solo)

• Result Stores

  • AMQP, Redis
  • memcached
  • SQLAlchemy, Django ORM
  • Apache Cassandra, IronCache, Elasticsearch

• Serialization

  • pickle, json, yaml, msgpack.
  • zlib, bzip2 compression.
  • Cryptographic message signing.

Framework Integration

Celery is easy to integrate with web frameworks, some of which even have integration packages:

Django	not needed
Pyramid	pyramid_celery
Pylons	celery-pylons
Flask	not needed
web2py	web2py-celery
Tornado	tornado-celery
FastAPI	not needed

The integration packages aren't strictly necessary, but they can make development easier, and sometimes they add important hooks like closing database connections at fork.

Documentation

The latest documentation is hosted at Read The Docs, containing user guides, tutorials, and an API reference.

最新的中文文档托管在 https://www.celerycn.io/ 中，包含用户指南、教程、API接口等。

Installation

You can install Celery either via the Python Package Index (PyPI) or from source.

To install using pip:

:

$ pip install -U Celery

Bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

You can specify these in your requirements or on the pip command-line by using brackets. Multiple bundles can be specified by separating them by commas.

:

$ pip install "celery[redis]"

$ pip install "celery[redis,auth,msgpack]"

The following bundles are available:

Serializers

celery[auth]

for using the auth security serializer.

celery[msgpack]

for using the msgpack serializer.

celery[yaml]

for using the yaml serializer.

Concurrency

celery[eventlet]

for using the eventlet pool.

celery[gevent]

for using the gevent pool.

Transports and Backends

celery[amqp]

for using the RabbitMQ amqp python library.

celery[redis]

for using Redis as a message transport or as a result backend.

celery[sqs]

for using Amazon SQS as a message transport.

celery[tblib]

for using the task_remote_tracebacks feature.

celery[memcache]

for using Memcached as a result backend (using pylibmc)

celery[pymemcache]

for using Memcached as a result backend (pure-Python implementation).

celery[cassandra]

for using Apache Cassandra/Astra DB as a result backend with the DataStax driver.

celery[azureblockblob]

for using Azure Storage as a result backend (using azure-storage)

celery[s3]

for using S3 Storage as a result backend.

celery[gcs]

for using Google Cloud Storage as a result backend.

celery[couchbase]

for using Couchbase as a result backend.

celery[arangodb]

for using ArangoDB as a result backend.

celery[elasticsearch]

for using Elasticsearch as a result backend.

celery[riak]

for using Riak as a result backend.

celery[cosmosdbsql]

for using Azure Cosmos DB as a result backend (using pydocumentdb)

celery[zookeeper]

for using Zookeeper as a message transport.

celery[sqlalchemy]

for using SQLAlchemy as a result backend (supported).

celery[pyro]

for using the Pyro4 message transport (experimental).

celery[slmq]

for using the SoftLayer Message Queue transport (experimental).

celery[consul]

for using the Consul.io Key/Value store as a message transport or result backend (experimental).

celery[django]

specifies the lowest version possible for Django support.

You should probably not use this in your requirements, it's here for informational purposes only.

Downloading and installing from source

Download the latest version of Celery from PyPI:

https://pypi.org/project/celery/

You can install it by doing the following:

:

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install

The last command must be executed as a privileged user if you aren't currently using a virtualenv.

Using the development version

With pip

The Celery development version also requires the development versions of kombu, amqp, billiard, and vine.

You can install the latest snapshot of these using the following pip commands:

:

$ pip install https://github.com/celery/celery/zipball/main#egg=celery
$ pip install https://github.com/celery/billiard/zipball/main#egg=billiard
$ pip install https://github.com/celery/py-amqp/zipball/main#egg=amqp
$ pip install https://github.com/celery/kombu/zipball/main#egg=kombu
$ pip install https://github.com/celery/vine/zipball/main#egg=vine

With git

Please see the Contributing section.

Getting Help

Mailing list

For discussions about the usage, development, and future of Celery, please join the celery-users mailing list.

IRC

Come chat with us on IRC. The #celery channel is located at the Libera Chat network.

Bug tracker

If you have any suggestions, bug reports, or annoyances please report them to our issue tracker at https://github.com/celery/celery/issues/

Wiki

https://github.com/celery/celery/wiki

Credits

Contributors

This project exists thanks to all the people who contribute. Development of celery happens at GitHub: https://github.com/celery/celery

You're highly encouraged to participate in the development of celery. If you don't like GitHub (for some reason) you're welcome to send regular patches.

Be sure to also read the Contributing to Celery section in the documentation.

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

billiard's Issues

Worker dies with exitcode 20 when using Popen() in task (2.7.3.24)

When using subprocess.Popen in a celery task. The worker dies with code 20 before the process returns. The offending version is 2.7.3.24, .23 works.

23:04:49 celery.1 | Process 'PoolWorker-1' pid:15622 exited with exitcode 20 23:05:02 celery.1 | Task modelconvert.tasks.convert_model[2064382f-ce91-45ec-9a45-d957c1c29f24] raised exception: WorkerLostError('Worker exited prematurely (exitcode: 20).',) 23:05:02 celery.1 | Traceback (most recent call last): 23:05:02 celery.1 | File "/Users/andi/.virtualenvs/mc/lib/python2.7/site-packages/billiard/pool.py", line 944, in _join_exited_workers 23:05:02 celery.1 | lost_ret, )) 23:05:02 celery.1 | WorkerLostError: Worker exited prematurely (exitcode: 20).

Relevant code an project is here:
https://github.com/x3dom/pipeline/blob/master/modelconvert/tasks.py#L389
https://github.com/x3dom/pipeline

Python 2.7.3/Mac, Ubuntu 12.04 x64
Python 2.6.1/Mac

Poll Result error causes celeryd to crash

Unrecoverable error: 'TypeError(\'__init__() takes exactly 3 arguments (2 given)\', <class \'requests.packages.urllib3.exceptions.MaxRetryError\'>, ("HTTPConnectionPool(host=\'www.timesfreepress.com\', port=80): Max retries exceeded with url: /news/2012/sep/08/dade-county-gets-new-extension-agriculture-agent/?utm_source=twitterfeed&utm_medium=twitter",))'

Stacktrace (most recent call last):

  File "celery/worker/__init__.py", line 353, in start
    component.start()
  File "celery/worker/consumer.py", line 369, in start
    self.consume_messages()
  File "celery/worker/consumer.py", line 453, in consume_messages
    readers[fileno](fileno, event)
  File "billiard/pool.py", line 660, in handle_event
    self._it.next()
  File "billiard/pool.py", line 633, in _process_result
    ready, task = poll(timeout)
  File "billiard/pool.py", line 1034, in _poll_result
    return True, self._quick_get()

This happened in a celery task that was making an HTTP request with requests, which raised an exception and it caused celeryd to stop running entirely. You have to manually restart after this. Versions are billiard==2.7.3.12 and celery==3.0.9

Pool dont work on freebsd 8.1: OSError: [Errno 23] Too many open files in system

This simple script fail on freebsd 8.1 maybe others version too. Multiprocessing is working.

import multiprocessing as mp

import billiard as mp
p = mp.Pool(16)
import time
p.apply_async(time.sleep, 1)

traceback:
Traceback (most recent call last):
File "test_multi.py", line 3, in
p = mp.Pool(16)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/init.py", line 253, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/pool.py", line 799, in init
self._create_worker_process()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/pool.py", line 856, in _create_worker_process
sentinel = Event()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/init.py", line 229, in Event
return Event()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 342, in init
self._cond = Condition(Lock())
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 223, in init
self._sleeping_count = Semaphore(0)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 138, in init
SemLock.init(self, SEMAPHORE, value, SEM_VALUE_MAX)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 87, in init
sl = self._semlock = _billiard.SemLock(kind, value, maxvalue)
OSError: [Errno 23] Too many open files in system

Can not create Manager instances

Does billiard support "Manager" feature of multiprocessing?

$ python -c "from billiard import Manager;m = Manager()"
Traceback (most recent call last):
File "", line 1, in
File "c:\Python27.32\lib\site-packages\billiard-2.7.3.23-py2.7-win32.egg\billiard__init__.py", line 87, in Manager
from .managers import SyncManager
File "c:\Python27.32\lib\site-packages\billiard-2.7.3.23-py2.7-win32.egg\billiard\managers.py", line 443, in
'xmlrpclib': (connection.XmlListener, connection.XmlClient),
AttributeError: 'module' object has no attribute 'XmlListener'

einfo.Frame doesn't read frame's f_locals and impairs raven/sentry

Are there plans to fix this? Previous versions of raven were able to log the f_locals information. A quick and dirty hack would be to use raven's ability to walk the f_locals information and make it JSON serializable.

In Frame.__init__:

try:
    from raven.utils.stacks import get_stack_info
except ImportError:
    pass
else:
    self.f_locals = get_stack_info([(frame, 1)])[0]['vars']

Having the variables in the context is very useful in Sentry.

AttributeError: 'str' object has no attribute 'append'

This is with a61149b

[2013-04-24 21:33:19,846: ERROR/MainProcess] Process PoolWorker-100
Traceback (most recent call last):
  File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/process.py", line 257, in _bootstrap
    self.run()
  File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/process.py", line 106, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/pool.py", line 303, in worker
    ready, task = poll(1.0)
  File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/pool.py", line 263, in poll
    return True, get()
  File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/queues.py", line 341, in get
    return recv()
AttributeError: 'str' object has no attribute 'append'

fails to build with sem_handle problem

pip install billiard gives me the following output:

x86_64-pc-linux-gnu-gcc -pthread -O2 -pipe -march=native -fPIC -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_billiard -I/usr/include/python2.7 -c Modules/_billiard/semaphore.c -o build-2.7/temp.linux-x86_64-2.7/Modules/_billiard/semaphore.o
Modules/_billiard/semaphore.c:15:5: error: expected specifier-qualifier-list before ‘SEM_HANDLE’
Modules/_billiard/semaphore.c:209:35: error: expected ‘)’ before ‘*’ token
... and a bunch more - i think/hope that's the relevant part though.

i use gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) and i'm trying to install billiard on a vserver (eventhough i don't really think that's relevant)

thanks

celery 3.0.1 and billiard-2.7.3.10 causing pain on Heroku

I have exactly this issue (see celery/django-celery#154 and celery/celery#856) on a heroku instance - I had upgraded from 2.4.2 (?) Billiard (billiard-2.7.3.10) is the problem, I get a _billiard.so lib that crashes on startup - and worse, there is no way for me to nuke the .so within the Heroku build-set....

We really need a billiard-2.7.3.11 very soon.....very?

Crashes with broken pipe

This makes our workers crash/hang every hour or so.

Traceback (most recent call last):
File "../lib/python2.7/site-packages/billiard/process.py", line 273, in _bootstrap
self.run()
File "../lib/python2.7/site-packages/billiard/process.py", line 122, in run
self._target(*self._args, **self._kwargs)
File "../lib/python2.7/site-packages/billiard/pool.py", line 302, in worker
put((ACK, (job, i, time.time(), pid)))
File "../lib/python2.7/site-packages/billiard/queues.py", line 377, in put
return send(obj)
IOError: [Errno 32] Broken pipe

celery==3.0.7
django-celery==3.0.4
kombu==2.4.3
billiard==2.7.3.12

We're using the SQS backend.

1.1.1: sqlalchemy: 'NoneType' object has no attribute 'get_dialect'

I just upgraded to 1.1.1 and everything stopped working. Any clues?

  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 91, in execute_safe
    return self.execute(*args, **kwargs)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 106, in execute
    return super(WorkerTaskTrace, self).execute()
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/execute/trace.py", line 59, in execute
    retval = self._trace()
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/execute/trace.py", line 73, in _trace
    return handler(trace.retval, trace.exc_type, trace.tb, trace.strtb)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 111, in handle_success
    self.task.backend.mark_as_done(self.task_id, retval)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/base.py", line 43, in mark_as_done
    return self.store_result(task_id, result, status=states.SUCCESS)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/base.py", line 157, in store_result
    return self._store_result(task_id, result, status, traceback)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/database.py", line 25, in _store_result
    session = self.ResultSession()
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/database.py", line 21, in ResultSession
    return ResultSession(dburi=self.dburi, **self.engine_options)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 34, in ResultSession
    engine, session = create_session(dburi, **kwargs)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 23, in create_session
    engine = get_engine(dburi, **kwargs)
  File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 18, in get_engine
    _ENGINES[dburi] = create_engine(dburi, **kwargs)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/engine/__init__.py", line 241, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/engine/strategies.py", line 46, in create
    dialect_cls = u.get_dialect()
AttributeError: 'NoneType' object has no attribute 'get_dialect'

Billiard 2.7.3.10 breaks backwards compatibility with python 2.5 (atexit.register)

Apparently billiard changed to use atexit.register as a decorator (at some point) for _exit_function in billiard/util.py, but atexit.register does not return a function until python 2.6, making it unusable as a decorator in earlier versions.

Suggested fix:
Remove it as a decorator and instead run an explicit call to the function (atexit.register(_exit_function)). Has the same result with backwards compatibility.

Calling NoneType recv_payload()

The fix for #53 is causing a new error.

Pool process error: TypeError("'NoneType' object is not callable",)
Traceback (most recent call last):
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 262, in run
sys.exit(self.workloop())
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 312, in workloop
req = wait_for_job()
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 405, in receive
ready, req = _receive(1.0)
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 376, in _recv
return True, loads(get_payload())
File "venv/lib/python3.3/site-packages/billiard/queues.py", line 346, in get_payload
return recv_payload()
TypeError: 'NoneType' object is not callable

Can't compile C extension

When I try to set up billiard from master branch python compiles C extension and fails with this error:

pipe_connection.obj : error LNK2019: unresolved external symbol __Billiard_conn_send_offset referenced in function _Billiard_conn_poll

When I replace static with extern in socket_connection.c for _Billiard_conn_send_offset function, everything compiles fine. I'm not sending PR for this because I'm not sure if this problem is local to my machine.

Pool worker does not detect when parent has gone away

Worker processes spawned by the Pool run poll(1.0) catching (IOError, EOFError), but neither seems to be triggered when the parent exits. As a result of that, if the parent process that started the Pool goes away, the child processes do not exit.

Seems like it all boils down to Pipe.poll() never raising the error when the other side of the pipe is gone.

see: https://gist.github.com/3701493

A workaround to force the Pool processes to exit when the parent is gone could be checking that os.getppid() returns '1'.

python3 build fails on 2.7.3.13

[mrunge@turing billiard-2.7.3.13]$ python3 setup.py
File "setup.py", line 57
raise BuildFailed, BuildFailed(), sys.exc_info()[2]
^
SyntaxError: invalid syntax

Version 2.7.3.12 builds fine

Segfault in pool.py:239

This problem is related to ticket https://github.com/ask/celery/issues/690.
When I set python trace in celeryd, i've got the last line in log(just before segfault)

[2012-06-04 15:09:03,551: WARNING/PoolWorker-2] call, /home/www/www/local/lib/python2.7/site-packages/billiard/pool.py:239

I tried to start celery several times and the result was the same.
Command — /home/www/www/bin/celeryd -B --logfile /home/www/www/celery.log --schedule=/home/www/www/celerybeat-schedule --concurrency=1

Used this method to trace — http://stackoverflow.com/questions/2663841/python-tracing-a-segmentation-fault

Error when starting worker (Python 3.3)

Python 3.3, Django 1.5.1, django-celery/kombu/celery/billiard from master

python manage.py celery worker --loglevel=info

 -------------- [email protected] v3.1.0rc3 (Cipater)
---- **** -----
--- * ***  * -- Darwin-12.3.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker:      amqp://guest@localhost:5672//
- ** ---------- .> app:         default:0x106cd4590 (djcelery.loaders.DjangoLoader)
- ** ---------- .> concurrency: 4 (processes)
- *** --- * --- .> events:      OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . musicdiscover.tasks.add

[2013-06-08 14:37:31,477: ERROR/MainProcess] Unrecoverable error: AttributeError("'Connection' object has no attribute 'recv_payload'",)
Traceback (most recent call last):
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/worker/__init__.py", line 189, in start
    self.blueprint.start(self)
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/bootsteps.py", line 352, in start
    return self.obj.start()
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/base.py", line 112, in start
    self.on_start()
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 461, in on_start
    **self.options)
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 236, in __init__
    for _ in range(processes))
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 236, in <genexpr>
    for _ in range(processes))
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 267, in create_process_queues
    inq, outq, synq = _SimpleQueue(), _SimpleQueue(), None
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/billiard/queues.py", line 317, in __init__
    self._make_methods()
  File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/billiard/queues.py", line 332, in _make_methods
    recv_payload = self._reader.recv_payload
AttributeError: 'Connection' object has no attribute 'recv_payload'

WorkerLostError: Worker exited prematurely (exitcode: 155)

I needed maxtasksprechild in Pool for python2.6, and googled up the billiard library. After installing version 2.7.3.31 with pip, I simply changed import multiprocessing to import billiard as multiprocessing and added maxtasksperchild=1 (no other changes). But now the app fails with:

Exception: Traceback (most recent call last): File "/home/radim/workspace/vew/server/lib/python2.6/site-packages/billiard/pool.py", line 948, in _join_exited_workers lost_ret, )) WorkerLostError: Worker exited prematurely (exitcode: 155).

It didn't use to fail before (=with built-in multiprocessing and no maxtasksperchild). Any ideas what's wrong? Where to look for problems?

Invalid issue

never mind

billiard.exceptions.WorkerLostError: Could not start worker processes

I'm currently not running the latest version of celery due to this bug
celery/celery#1301
As I am now i can only use the celery --help, celery help commands but I can't seem to start celery -A tasks worker --loglevel=info
I'm following the first step celery tutorial since I'm still learning allot.
The rabbitmq server is running and my tasks.py file is exaclty the same as in the tutorial.
Here is my output.

[gert@localhost ~]$ celery -A tasks worker --loglevel=info

-------------- [email protected] v3.1.0rc1 (Cipater)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: amqp://guest@localhost:5672//

• ** ---------- . app: tasks:0x7f75054a1c50
• ** ---------- . concurrency: 2 (processes)
• ** ---------- . events: OFF (enable -E to monitor this worker)
• ** ----------
• *** --- * --- [Queues]
  -- ******* ---- . celery exchange=celery(direct) key=celery
  --- ***** -----

[Tasks]
. tasks.add

[2013-04-12 14:39:22,761: ERROR/MainProcess] Process PoolWorker-1
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 255, in _bootstrap
self.run()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 104, in run
self._target(_self._args, *_self._kwargs)
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/pool.py", line 262, in worker
reset_signals()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/common.py", line 19, in reset_signals
signal.signal(i, handler)
OSError: [Errno 22] Invalid argument
[2013-04-12 14:39:22,772: ERROR/MainProcess] Process PoolWorker-2
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 255, in _bootstrap
self.run()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 104, in run
self._target(_self._args, *_self._kwargs)
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/pool.py", line 262, in worker
reset_signals()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/common.py", line 19, in reset_signals
signal.signal(i, handler)
OSError: [Errno 22] Invalid argument
[2013-04-12 14:39:22,789: INFO/MainProcess] Connected to amqp://[email protected]:5672//
[2013-04-12 14:39:22,806: INFO/MainProcess] mingle: searching for neighbors
[2013-04-12 14:39:23,822: INFO/MainProcess] mingle: synced with [email protected]
[2013-04-12 14:39:23,823: ERROR/MainProcess] Process 'PoolWorker-2' pid:23560 exited with exitcode 1
[2013-04-12 14:39:23,824: ERROR/MainProcess] Process 'PoolWorker-1' pid:23559 exited with exitcode 1
[2013-04-12 14:39:23,824: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/init.py", line 205, in start
self.namespace.start(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 352, in start
return self.obj.start()
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/consumer.py", line 195, in start
ns.start(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/consumer.py", line 664, in start
c.loop(*c.loop_args())
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/loops.py", line 30, in asynloop
with hub as hub:
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/hub.py", line 199, in enter
self.init()
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/hub.py", line 147, in init
callback(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/components.py", line 121, in on_poll_init
raise WorkerLostError('Could not start worker processes')
billiard.exceptions.WorkerLostError: Could not start worker processes

Hope you guys can figure it out.

PicklingError: Can't pickle built-in method recv of _billiard.PipeConnection object

This happens on Windows using Python 2.7.3:

[2013-07-11 22:29:19,724: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <built-in method recv of _billiard.PipeConnection object at 0x034B7090>: it's not found as __main__.recv",)
Traceback (most recent call last):
  File "D:\Dev\Python\lib\site-packages\celery\worker\__init__.py", line 191, in start
    self.blueprint.start(self)
  File "D:\Dev\Python\lib\site-packages\celery\bootsteps.py", line 119, in start
    step.start(parent)
  File "D:\Dev\Python\lib\site-packages\celery\bootsteps.py", line 352, in start
    return self.obj.start()
  File "D:\Dev\Python\lib\site-packages\celery\concurrency\base.py", line 112, in start
    self.on_start()
  File "D:\Dev\Python\lib\site-packages\celery\concurrency\processes.py", line 461, in on_start
    **self.options)
  File "D:\Dev\Python\lib\site-packages\billiard\pool.py", line 914, in __init__
    self._create_worker_process(i)
  File "D:\Dev\Python\lib\site-packages\billiard\pool.py", line 1004, in _create_worker_process
    w.start()
  File "D:\Dev\Python\lib\site-packages\billiard\process.py", line 125, in start
    self._popen = Popen(self)
  File "D:\Dev\Python\lib\site-packages\billiard\forking.py", line 352, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "D:\Dev\Python\lib\site-packages\billiard\forking.py", line 146, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "D:\Dev\Python\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "D:\Dev\Python\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "D:\Dev\Python\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "D:\Dev\Python\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "D:\Dev\Python\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "D:\Dev\Python\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "D:\Dev\Python\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "D:\Dev\Python\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <built-in method recv of _billiard.PipeConnection object at 0x034B7090>: it's not found as __main__.recv

Billiard 2.7.3.10 issues with bytes / strings

Trying to run billiard 2.7.3.10 with Celery in Python 3.2. I get the following error (suggested fix below):
[2012-07-25 10:29:27,504: ERROR/MainProcess] Unrecoverable error: TypeError('string argument without an encoding',)
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/worker/init.py", line 350, in start
component.start()
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/pool.py", line 760, in init
self._setup_queues()
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/pool.py", line 990, in _setup_queues
self._inqueue = SimpleQueue()
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/queues.py", line 339, in init
self._reader, self._writer = Pipe(duplex=False)
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/init.py", line 113, in Pipe
from .connection import Pipe
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/connection.py", line 426, in
CHALLENGE = bytes('#CHALLENGE#')
TypeError: string argument without an encoding

By specifying an encoding in lines 426-428 of connection.py, it got past this error. Not sure if other things will pop up elsewhere, however.

other options might be utf_32, utf_16

CHALLENGE = bytes('#CHALLENGE#','ascii')

WELCOME = bytes('#WELCOME#','ascii')

FAILURE = bytes('#FAILURE#','ascii')

rebuild semlock seems to fail

I'm trying to run celery in Windows (64bit) and when i try to start a worker

celery.exe -A tasks worker

i get the following Traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\site-packages\billiard-2.7.3.15-py2.7-win-amd64.egg\billiard\forking.py", line 525, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 1217, in load_build
    setstate(state)
  File "C:\Python27\lib\site-packages\billiard-2.7.3.15-py2.7-win-amd64.egg\billiard\synchronize.py", line 110, in __setstate__
    self._semlock = _billiard.SemLock._rebuild(*state)
TypeError: function takes exactly 4 arguments (3 given)

invalid ticket

Typo in docs/conf.py

docs/conf.py:38:
copyright = u'2009, Opera Softare (WebTeam)'
should be
copyright = u'2009, Opera Software (WebTeam)'

Supervisor process dying

I'm seeing this error when using autoscaling on celery.

[2013-08-08 21:25:17,717: ERROR/MainProcess] Thread 'Supervisor' crashed: StopIteration()
Traceback (most recent call last):
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 346, in run
    return self.body()
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 398, in body
    pool._maintain_pool()
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1054, in _maintain_pool
    self._repopulate_pool(joined)
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1039, in _repopulate_pool
    self._create_worker_process(self._avail_index())
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1045, in _avail_index
    return next(i for i in range(self._processes) if i not in indices)
StopIteration

It results in this error from the workers:

[2013-08-08 21:25:18,593: ERROR/MainProcess] Process PoolWorker-5
Traceback (most recent call last):
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/process.py", line 271, in _bootstrap
    self.run()
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 321, in worker
    put((READY, (job, i, (False, einfo))))
  File "/var/www/env/local/lib/python2.7/site-packages/billiard/queues.py", line 363, in put
    return send(obj)
OSError: [Errno 32] Broken pipe

billiard==2.7.3.31
celery==3.0.21
django-celery==3.0.21
kombu==2.5.12

pool.py uses os.kill (incompatible <=python 2.5)

Looks like pool.py has a few os.kill calls in it, even though the start has the _kill defined to deal with that for windows compatibility.

Port tests from multiprocessing

The testsuite doesn't run currently because the tests are not ported from multiprocessing (see #29).

Can not import celery.task by Jython(2.5 or 2.7) on ubuntu

Can you help me out of this issue, or did I miss something.

@ubuntu:/home/jython2.5.4rc1$ ./jython
Jython 2.5.4rc1 (2.5:723492dbab02, Feb 8 2013, 09:50:55)
[Java HotSpot(TM) Server VM (Oracle Corporation)] on java1.7.0_13
Type "help", "copyright", "credits" or "license" for more information.

import celery.task
Traceback (most recent call last):
File "", line 1, in
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/task/init.py", line 57, in
lambda: current_app.tasks['celery.backend_cleanup']
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 173, in recreate_module
new_module.dict.update(dict((mod, get_compat_module(new_module, mod))
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 173, in
new_module.dict.update(dict((mod, get_compat_module(new_module, mod))
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 188, in get_compat_module
module = sys.modules[fqdn] = import_module(attrs)
File "/home/jython2.5.4rc1/Lib/site-packages/importlib-1.0.2-py2.5.egg/importlib/init.py", line 37, in import_module
import(name)
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/canvas.py", line 24, in
from celery.result import AsyncResult, GroupResult
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/result.py", line 23, in
from .app import app_or_default
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/app/init.py", line 25, in
from .base import Celery, AppPickler # noqa
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/app/base.py", line 21, in
from billiard import forking as _forking
File "/home/jython2.5.4rc1/Lib/site-packages/billiard-2.7.3.21-py2.5.egg/billiard/forking.py", line 17, in
from ._ext import Connection, PipeConnection, win32
ImportError: cannot import name Connection

Setup.py:run_setup throws SystemExit if no compiler is found

Installing billiard on Windows fails if there is no C compiler present.

The current setup.py attempts to handle this by first running run_setup(True) and catching any Exceptions thrown. Unfortunately SystemExit is thrown somewhere if no compiler is found, thus the finally block (which attempts to build without the C extension) is never hit.

This can be fixed by changing Exception to BaseException, which catches SystemExit, however this is not a good idea for a few reasons. Not sure how else to fix this.

get_command_line doesn't set up Django environment?

http://stackoverflow.com/questions/11646279/celery-error-no-module-named-billiard-forking-how-to-diagnose/11647902#11647902

I'm getting this error in a Django app that otherwise runs ok:

ImportError: No module named billiard.forking

It comes down to get_command_line running a command line without the full Django environment. If I either add the billiard path to PYTHONPATH, or "pip install billiard", that starts a game of whack-a-mole:

[2012-07-25 21:05:27,690: DEBUG/MainProcess] Consumer: Connection established. 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 470, in main
    prepare(preparation_data)
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 610, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named django

Here's the Celery report:

software -> celery:3.0.3 (Chiastic Slide) kombu:2.3.0 py:2.6.5
            billiard:2.7.3.10 django:1.4.0.final.0
platform -> system:Linux arch:32bit, ELF imp:CPython
loader   -> djcelery.loaders.DjangoLoader
settings -> transport:django results:database

_wrapped: <django.conf.Settings object at 0x901b46c>

The code is https://github.com/stevage/mytardis

billiard in python 3.3 on Windows

hello,

I am using python 3.3 and I am trying to run celery. Now I ran Celery and I get the following error:

File "C:\Python33\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\for
king.py", line 276, in
import _subprocess
ImportError: No module named '_subprocess'

I have searched this on the web and I found that I must change it to _winapi, Is this right? After this I get another error, AttributeError: 'int' object has no attribute 'Detach' what can I do to fix this problem?
thanks in advance

ImportError: No module named billiard.forking with Buildout [now Windows only]

I have seen #26 and #10 but I am not using Django so the Django layout hack doesn't help.

I am attempting to use celery with a pyramid project that is managed with buildout. Starting celeryd leads to ImportError in the spawned billiard processes. A quick read through the code looks like it passes sys.path to the child processes. However, because the command that is run is "'from billiard.forking import main; main()'" it is trying to import billiard.forking before unpickling the path.

I can solve this by setting PYTHONPATH before calling:

prog = 'from billiard.forking import main; main()'
os.environ['PYTHONPATH'] = ':'.join(sys.path)
return [_python_exe, '-c', prog, '--billiard-fork']

If the forked process then sets sys.path to the unpickled parent one would it harm anything to just put it in billiard?

There may well be good reason not to. If so I would propose making it optional.

I was originally thinking of having a --with-buildout option, but there may be a more general reason for preserving sys.path in PYTHONPATH when calling.

I am not familiar enough with celery to know if how you pass options from celery to billiards. So as a guess, maybe something like this in celeryd's options:

...
Option('--preserve-syspath', action='store_true', default=False, dest='preserve_path'),
Option('--with-buildout', action='store_true', default=False, dest='preserve_path'),
...

and then checking for it properly here:

SHA: 3c75659

Thoughts?

Installation fails on Snow Leopard

Hi, I'm trying to install billiard 2.7.3.12 as part of an app I'm working on. We're using PIP (with virtualenv) to handle package management. Other packages install fine, but when it gets to billiard, I get this error:

Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe

It might not be billiard's fault, but any help would be very appreciated.

Here's a more verbose output:

Running setup.py install for billiard
building '_billiard' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.6-universal-2.6/Modules/_billiard/multiprocessing.o
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/as/ppc/as or /usr/bin/../local/libexec/as/ppc/as) for architecture ppc not installed
Installed assemblers are:
/usr/bin/../libexec/as/x86_64/as for architecture x86_64
/usr/bin/../libexec/as/i386/as for architecture i386
/usr/bin/../libexec/as/arm/as for architecture arm
Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe
compilation terminated.
lipo: can't open input file: /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-//cc7zhCqu.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1
Complete output from command /Users/chad/Sites/post/ENV/bin/python -c "import setuptools;file='/Users/chad/Sites/post/ENV/build/billiard/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-/pip-t5W0cF-record/install-record.txt --single-version-externally-managed --install-headers /Users/chad/Sites/post/ENV/bin/../include/site/python2.6:
running install

running build

running build_py

creating build

creating build/lib.macosx-10.6-universal-2.6

creating build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/init.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/_ext.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/_win.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/common.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/compat.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/connection.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/einfo.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/exceptions.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/forking.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/heap.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/managers.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/pool.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/process.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/queues.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/reduction.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/sharedctypes.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/synchronize.py -> build/lib.macosx-10.6-universal-2.6/billiard

copying billiard/util.py -> build/lib.macosx-10.6-universal-2.6/billiard

creating build/lib.macosx-10.6-universal-2.6/billiard/dummy

copying billiard/dummy/init.py -> build/lib.macosx-10.6-universal-2.6/billiard/dummy

copying billiard/dummy/connection.py -> build/lib.macosx-10.6-universal-2.6/billiard/dummy

creating build/lib.macosx-10.6-universal-2.6/billiard/tests

copying billiard/tests/init.py -> build/lib.macosx-10.6-universal-2.6/billiard/tests

copying billiard/tests/test_multiprocessing.py -> build/lib.macosx-10.6-universal-2.6/billiard/tests

running build_ext

building '_billiard' extension

creating build/temp.macosx-10.6-universal-2.6

creating build/temp.macosx-10.6-universal-2.6/Modules

creating build/temp.macosx-10.6-universal-2.6/Modules/_billiard

gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.6-universal-2.6/Modules/_billiard/multiprocessing.o

/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/as/ppc/as or /usr/bin/../local/libexec/as/ppc/as) for architecture ppc not installed

Installed assemblers are:

/usr/bin/../libexec/as/x86_64/as for architecture x86_64

/usr/bin/../libexec/as/i386/as for architecture i386

/usr/bin/../libexec/as/arm/as for architecture arm

Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe

compilation terminated.

lipo: can't open input file: /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-//cc7zhCqu.out (No such file or directory)

error: command 'gcc-4.2' failed with exit status 1

---

Command /Users/chad/Sites/post/ENV/bin/python -c "import setuptools;file='/Users/chad/Sites/post/ENV/build/billiard/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-/pip-t5W0cF-record/install-record.txt --single-version-externally-managed --install-headers /Users/chad/Sites/post/ENV/bin/../include/site/python2.6 failed with error code 1 in /Users/chad/Sites/post/ENV/build/billiard
Storing complete log in /Users/chad/.pip/pip.log

error in pool.py file because of _note() missing

Just repporting an error that I solved in adding these lines in the pool.py file:

Into the class LaxBoundedSemaphore, added the following line:

def _note(self, *arg, **args): pass

It seems that with the last version of python (3.3), the _note function in the class Semaphore has been removed. Semaphore is a class of the threading library.

Django old layout hack - assumes top-level settings module

When running celery in my Django project, a warning is displayed about setup_environ. I tracked it down to billiard/forking.py at the following code section (line 437):

try:
    settings_name = os.environ['DJANGO_SETTINGS_MODULE']
except KeyError:
    return  # not using Django.

try:
    project_name, _ = settings_name.split('.', 1)
except ValueError:
    return  # not modified by setup_environ

The reason why this is occurring for me is because I have moved out my settings module into a package of it's own, separating the common code from the development/production-specific configuration.

I am using Django 1.4.1, so the warning message was confusing (what setup_environ?). I'm not sure if there would be an easy way to detect what is needed, without false-positives. Django's settings module is easy to extend and fit to one's needs, and it'd be nice to not limit that.

For now, I can stop the warning by adding the DJANGO_PROJECT_DIR environment variable directly in manage.py:

os.environ.setdefault("DJANGO_PROJECT_DIR",
    os.path.dirname(os.path.realpath(__file__)))

Incompatibility with Python3.3 threading module

Hello! I attempted to install and use billiard (2.7.3.18) with the latest version of Celery, which I installed from source from github this evening (3.1.0rc1). Calling 'celery worker' at the commandline errors out in billiard with the following stack trace, pasted below. Based on http://docs.python.org/3/whatsnew/3.3.html#threading, it looks like billiard needs to be updated for Python3.3 to reflect the fact that Semaphore and Condition are no longer factory functions returning a class instance, but are now instead classes.

Please let me know if I can provide any additional information! Thanks very much for taking a look.

Traceback (most recent call last):
File "/home/gideon/gctssfc/venv/bin/celery", line 9, in
load_entry_point('celery==3.1.0rc1', 'console_scripts', 'celery')()
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/main.py", line 28, in main
main()
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 76, in main
cmd.execute_from_commandline(argv)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 700, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 255, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 692, in handle_argv
return self.execute(command, argv)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 667, in execute
argv[1:], command=argv[0])
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/worker.py", line 149, in run_from_argv
return super(worker, self).run_from_argv(prog_name, argv, command)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 259, in run_from_argv
sys.argv if argv is None else argv, command)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 296, in handle_argv
return self(_args, *_options)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 227, in call
ret = self.run(_args, *_kwargs)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/worker.py", line 162, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/concurrency/init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/kombu-2.5.0rc1-py3.3.egg/kombu/utils/init.py", line 80, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/home/gideon/gctssfc/venv/lib/python3.3/importlib/init.py", line 88, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1577, in _gcd_import
File "", line 1558, in _find_and_load
File "", line 1525, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1023, in load_module
File "", line 1004, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 869, in _load_module
File "", line 313, in _call_with_frames_removed
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/concurrency/processes.py", line 17, in
from billiard.pool import Pool, RUN, CLOSE
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/billiard/pool.py", line 72, in
_Semaphore = threading._Semaphore
AttributeError: 'module' object has no attribute '_Semaphore'

'SimpleQueue' object has no attribute '_wlock' in billiard.queue

Just a note that to get to this point i have installed celery, amqp and billiard from the github dev versions.

Current relevant packages
amqp==1.2.0*
anyjson==0.3.3
billiard==3.3.0.0rc1*
celery==3.1.0rc3*
kombu==2.5.12
*cloned from github and installed with python setup.py install

This is running on a fresh install of WinPython 3.3 on Windows 7.
From the First Steps of Celery docs
>celery -A tasks worker --loglevel=info

[2013-07-03 10:07:41,352: ERROR/MainProcess] Unrecoverable error: AttributeError("'SimpleQueue' object has no attribute 'wlock'",)
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\worker__init_.py", line 192, in start
self.blueprint.start(self)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\bootsteps.py", line 119, in start
step.start(parent)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\bootsteps.py", line 352, in start
return self.obj.start()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\concurrency\base.py", line 112, in start
self.on_start()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\concurrency\processes.py", line 461, in on_start
**self.options)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\pool.py", line 876, in init
self._setup_queues()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\pool.py", line 1198, in _setup_queues
self._inqueue = SimpleQueue()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\queues.py", line 372, in init
self._make_methods()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\queues.py", line 352, in _make_methods
if self._wlock is None:
AttributeError: 'SimpleQueue' object has no attribute '_wlock'

billiard-2.7.3.19 - Python 2.5 - connection.py: SyntaxError: 'import *' not allowed with 'from .'

Edit - edit - edit, I need to test stuff better...

This is the problem anyway:

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import billiard
>>> import billiard.connection
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python25\lib\site-packages\billiard-2.7.3.19-py2.5-win32.egg\billiard\connection.py", line 9
    from ._connection import *
SyntaxError: 'import *' not allowed with 'from .'

billiard 2.7.3.14 python3 tests fail (syntax error)

[mrunge@turing billiard-2.7.3.14]$ python3 setup.py test
running test
running build_py
running egg_info
creating /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info
writing /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/PKG-INFO
writing top-level names to /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/top_level.txt
writing dependency_links to /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/dependency_links.txt
writing manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
reading manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'Lib'
writing manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
running build_ext
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-branch' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-html' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-package' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-exclude' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
Failure: SyntaxError (invalid syntax (process.py, line 252)) ... ERROR

ERROR: Failure: SyntaxError (invalid syntax (process.py, line 252))

Traceback (most recent call last):
File "/usr/lib/python3.2/site-packages/nose/failure.py", line 37, in runTest
raise self.exc_class(self.exc_val).with_traceback(self.tb)
File "/usr/lib/python3.2/site-packages/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/python3.2/site-packages/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/python3.2/site-packages/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/billiard/init.py", line 61, in
from .process import Process, current_process, active_children
File "", line None
SyntaxError: invalid syntax (process.py, line 252)

---

Ran 1 test in 0.001s

FAILED (errors=1)
Error in atexit._run_exitfuncs:
TypeError: 'NoneType' object is not callable

python is python3.2

Tests doesn't run

Hi,

I tried to run the testsuite without luck. I tried with Python 2.7 from Ubuntu 13.04.

1. the Makefile seems to be outdated:

$ make test
find Lib/ ( -name '.py[co]' -or -name '.so' ) -exec rm {} ;
find: `Lib/': No such file or directory
make: *** [clean] Error 1

2. 2. python2.7 test_multiprocessing.py
      Exception in thread Thread-5:
      Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
      self.run()
      File "/usr/lib/python2.7/threading.py", line 505, in run
      self.__target(_self.__args, *_self.__kwargs)
      File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 271, in worker
      signal.signal(SIG_SOFT_TIMEOUT, soft_timeout_sighandler)
      ValueError: signal only works in main thread

Traceback (most recent call last):
File "test_multiprocessing.py", line 2071, in
main()
File "test_multiprocessing.py", line 2068, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
File "test_multiprocessing.py", line 2036, in test_main
ThreadsMixin.pool = billiard.dummy.Pool(4)
File "/usr/lib/python2.7/dist-packages/billiard/dummy/init.py", line 156, in Pool
return ThreadPool(processes, initializer, initargs)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 1597, in init
Pool.init(self, processes, initializer, initargs)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 839, in init
self._create_worker_process(i)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 911, in _create_worker_process
self._poolctrl[w.pid] = sentinel
AttributeError: 'DummyProcess' object has no attribute 'pid'

Maybe I do something wrong? Or is the testsuite broken?

TIA Tom

Error installing on OS X 10.8.2

I'm getting an error trying to install billiard on OS X 10.8.2 related to multiprocessing.c

Requirement already satisfied (use --upgrade to upgrade): celery in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages
Downloading/unpacking billiard>=2.7.3.14 (from celery)
  Running setup.py egg_info for package billiard

    warning: no files found matching '*.py' under directory 'Lib'
Downloading/unpacking python-dateutil>=1.5,<2.0 (from celery)
  Running setup.py egg_info for package python-dateutil

Downloading/unpacking kombu>=2.4.7,<3.0 (from celery)
  Running setup.py egg_info for package kombu

Requirement already satisfied (use --upgrade to upgrade): anyjson>=0.3.3 in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages (from kombu>=2.4.7,<3.0->celery)
Requirement already satisfied (use --upgrade to upgrade): amqplib>=1.0.2 in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages (from kombu>=2.4.7,<3.0->celery)
Installing collected packages: billiard, python-dateutil, kombu
  Running setup.py install for billiard
    building '_billiard' extension
    gcc -fno-strict-aliasing -fno-common -dynamic -Os -w -pipe -march=native -Qunused-arguments -mmacosx-version-min=10.7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/lib -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.7-x86_64-2.7/Modules/_billiard/multiprocessing.o
    Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -march= switch
    Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -mtune= switch
    error: command 'gcc' failed with exit status 1
    Complete output from command /Users/igor/.virtualenvs/guide/bin/python -c "import setuptools;__file__='/Users/igor/.virtualenvs/guide/build/billiard/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/vg/2kjxrrkn0s52zhqgbg13nq440000gn/T/pip-OUd7qO-record/install-record.txt --single-version-externally-managed --install-headers /Users/igor/.virtualenvs/guide/bin/../include/site/python2.7:
    running install

running build

running build_py

running build_ext

building '_billiard' extension

gcc -fno-strict-aliasing -fno-common -dynamic -Os -w -pipe -march=native -Qunused-arguments -mmacosx-version-min=10.7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/lib -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.7-x86_64-2.7/Modules/_billiard/multiprocessing.o

Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -march= switch

Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -mtune= switch

error: command 'gcc' failed with exit status 1

----------------------------------------
Command /Users/igor/.virtualenvs/guide/bin/python -c "import setuptools;__file__='/Users/igor/.virtualenvs/guide/build/billiard/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/vg/2kjxrrkn0s52zhqgbg13nq440000gn/T/pip-OUd7qO-record/install-record.txt --single-version-externally-managed --install-headers /Users/igor/.virtualenvs/guide/bin/../include/site/python2.7 failed with error code 1 in /Users/igor/.virtualenvs/guide/build/billiard
Storing complete log in /Users/igor/.pip/pip.log

Compiling now mandatory?

Hello!

I am using the latest celery 3.1rc3 version with the latest billiard, amqp and kombu.

It appears that billiard is now required to be compiled. I am receiving this error:

Traceback (most recent call last):
File "/vendor/billiard/queues.py", line 332, in _make_methods
recv_payload = self._reader.recv_payload
AttributeError: '_multiprocessing.Connection' object has no attribute 'recv_payload'

_billiard seems to have the method recv_payload whereas multiprocessing does not. This is causing billiard to fail if it is not compiled. Is compiling billiard now a requirement?

Thanks!

-tony

Python 3 import issues -ImportError: cannot import name WAIT_ABANDONED_0

I am working the First Steps tutorial, but running into issues with the Python3 imports.

It looks like some of the _winapi imports are in the win32con or win32event modules.

My setup
I'm running Python 3.3 on Windows 7
I have installed celery, kombu, billiard and py-amqp (as amqp) from git with

>git clone <lib>
>cd <lib>
>python setup.py develop

giving me
> pip freeze
...
-e git+https://github.com/celery/py-amqp.git@735c2b409f6e1180f9401132bae2a95bdd3ab34f#egg=amqp-dev
anyjson==0.3.3
billiard==3.3.0.0rc1
-e git+https://github.com/celery/celery.git@35ef4a557c2adffd4778e795fe64f2d558dc01cf#egg=celery-dev
-e git+https://github.com/celery/kombu.git@97a67498d86d5f84fff8706a4525b960b6945853#egg=kombu-dev
pymongo==2.5.2
python-dateutil==2.1
pytz==2013b
six==1.3.0
...

Now running the celery worker
C:\Users\gav\src\python\proj>celery -A tasks worker -l debug

Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 29, in
from _winapi import (
ImportError: cannot import name WAIT_ABANDONED_0

The problem lies here,

from _winapi import (
       WAIT_OBJECT_0,
       WAIT_ABANDONED_0,
       WAIT_TIMEOUT,
       INFINITE,
   )

If I change this to:

import win32con
from win32con import (
    WAIT_OBJECT_0,
    WAIT_ABANDONED_0,
    WAIT_TIMEOUT,
)    
import win32event
from win32event import INFINITE
import _winapi

then run the worker
C:\Users\gav\src\python\proj>celery -A tasks worker -l debug
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 977, in
from . import reduction
File "c:\users\gav\src\billiard\build\lib\billiard\reduction.py", line 21, in
from billiard.connection import Client, Listener
ImportError: cannot import name Client

Moving to reduction.py
Changing the import of Client and Listener

if sys.version_info[0] == 3:
#    from billiard.connection import Client, Listener
    from billiard._connection3 import Client, Listener

C:\Users\gav\src\python\proj>celery -A tasks worker -l debug
Traceback (most recent call last):
File "c:\users\gav\src\billiard\build\lib\billiard_ext.py", line 21, in
Connection = _billiard.Connection
AttributeError: 'module' object has no attribute 'Connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 977, in
from . import reduction
File "c:\users\gav\src\billiard\build\lib\billiard\reduction.py", line 27, in
from ._ext import _billiard, win32
File "c:\users\gav\src\billiard\build\lib\billiard_ext.py", line 23, in
from billiard.connection import Connection # noqa
ImportError: cannot import name Connection

I will leave this issue here, and continue following the trail of imports a little later.

Circular dependency between imports

Because of this circular dependency:

File "C:\Python33\lib\site-packages\billiard\connection.py", line 9, in <module>
  from . import _connection3 as connection
File "C:\Python33\lib\site-packages\billiard\_connection3.py", line 960, in <module>
  from . import reduction
File "C:\Python33\lib\site-packages\billiard\reduction.py", line 21, in <module>
  from billiard.connection import Client, Listener

the last line in connection.py sys.modules[__name__] = connection doesn't execute, so billiard.connection module doesn't get overridden.

LaxBoundedSemaphore broken on Python 3.3

On Python 3.3 Threading.Semaphore does not accept the verbose argument:

Traceback (most recent call last):
  File "/home/alga/src/celery/celery/worker/__init__.py", line 205, in start
    self.namespace.start(self)
  File "/home/alga/src/celery/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/alga/src/celery/celery/bootsteps.py", line 352, in start
    return self.obj.start()
  File "/home/alga/src/celery/celery/concurrency/base.py", line 112, in start
    self.on_start()
  File "/home/alga/src/celery/celery/concurrency/processes.py", line 81, in on_start
    **self.options)
  File "/home/alga/src/eggs/billiard-2.7.3.23-py3.3.egg/billiard/pool.py", line 835, in __init__
    self._putlock = semaphore or LaxBoundedSemaphore(self._processes)
  File "/home/alga/src/eggs/billiard-2.7.3.23-py3.3.egg/billiard/pool.py", line 136, in __init__
    _Semaphore.__init__(self, value, verbose)
TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given

Semaphore Lock Files Not Being Deleted

We are on Billiard 2.7.3.9. Billiard is creating semaphore lock files, but they don't seem to always be clearing out. Every few weeks, this seems to use up our entire disk, then causing entire server to crash. Here is the the stack trace:

[2012-07-11 16:42:07,735: (celery.worker) ERROR] Unrecoverable error: OSError(28, 'No space left on device')
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/worker/init.py", line 350, in start
component.start()
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/pool.py", line 760, in init
self._setup_queues()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/pool.py", line 990, in _setup_queues
self._inqueue = SimpleQueue()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/queues.py", line 340, in init
self._rlock = Lock()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/synchronize.py", line 171, in init
SemLock.init(self, SEMAPHORE, 1, 1)
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/synchronize.py", line 85, in init
kind, value, maxvalue, self._make_name(), unlink_immediately)
OSError: [Errno 28] No space left on device

Unrecoverable error in celery worker

After the update to Celery 3.0.3 (from 2.4.6) today, we had all workers crash after running for roughly 12 hours. The traceback was the following:

Jul 27 19:12:40 78.108.125.18 celery[19503:140072151082752]: [celery.worker] Unrecoverable error: TypeError('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ()) Traceback (most recent call last): File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/init.py", line 350, in start component.start() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 360, in start self.consume_messages() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 430, in consume_messages readers[fileno](fileno, event) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 635, in handle_event self._it.next() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 608, in _process_result ready, task = poll(timeout) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 997, in _poll_result return True, self._quick_get() TypeError: ('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ())

It seems it is trying to initiate an exception of ours without any arguments (besides self), which then fails and subsequently stop the worker.

billard 2.7.3.10 was installed.

Tests fail with core dump

python setup.py test (using python2)
....
test_connection (billiard.tests.test_multiprocessing.WithProcessesTestConnection) ... Segmentation fault (core dumped)

What information do you need to debug? version is 2.7.3.14

unexpected keyword argument 'allow_restart'

using celery = 3.0.5 and billiard = 2.7.3.11 i get following error by starting celeryd:

[2012-08-01 23:16:12,776: ERROR/MainProcess] Unrecoverable error: TypeError("init() got an unexpected keyword argument 'allow_restart'",)
Traceback (most recent call last):
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/worker/init.py", line 353, in start
component.start()
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
TypeError: init() got an unexpected keyword argument 'allow_restart'