celery / billiard Goto Github PK
View Code? Open in Web Editor NEWMultiprocessing Pool Extensions
License: Other
Multiprocessing Pool Extensions
License: Other
docs/conf.py:38:
copyright = u'2009, Opera Softare (WebTeam)'
should be
copyright = u'2009, Opera Software (WebTeam)'
This simple script fail on freebsd 8.1 maybe others version too. Multiprocessing is working.
import billiard as mp
p = mp.Pool(16)
import time
p.apply_async(time.sleep, 1)
traceback:
Traceback (most recent call last):
File "test_multi.py", line 3, in
p = mp.Pool(16)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/init.py", line 253, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/pool.py", line 799, in init
self._create_worker_process()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/pool.py", line 856, in _create_worker_process
sentinel = Event()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/init.py", line 229, in Event
return Event()
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 342, in init
self._cond = Condition(Lock())
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 223, in init
self._sleeping_count = Semaphore(0)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 138, in init
SemLock.init(self, SEMAPHORE, value, SEM_VALUE_MAX)
File "./lib/python2.6/site-packages/billiard-2.7.3.10-py2.6-freebsd-8.1-RELEASE-i386.egg/billiard/synchronize.py", line 87, in init
sl = self._semlock = _billiard.SemLock(kind, value, maxvalue)
OSError: [Errno 23] Too many open files in system
Hello! I attempted to install and use billiard (2.7.3.18) with the latest version of Celery, which I installed from source from github this evening (3.1.0rc1). Calling 'celery worker' at the commandline errors out in billiard with the following stack trace, pasted below. Based on http://docs.python.org/3/whatsnew/3.3.html#threading, it looks like billiard needs to be updated for Python3.3 to reflect the fact that Semaphore and Condition are no longer factory functions returning a class instance, but are now instead classes.
Please let me know if I can provide any additional information! Thanks very much for taking a look.
Traceback (most recent call last):
File "/home/gideon/gctssfc/venv/bin/celery", line 9, in
load_entry_point('celery==3.1.0rc1', 'console_scripts', 'celery')()
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/main.py", line 28, in main
main()
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 76, in main
cmd.execute_from_commandline(argv)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 700, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 255, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 692, in handle_argv
return self.execute(command, argv)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/celery.py", line 667, in execute
argv[1:], command=argv[0])
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/worker.py", line 149, in run_from_argv
return super(worker, self).run_from_argv(prog_name, argv, command)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 259, in run_from_argv
sys.argv if argv is None else argv, command)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 296, in handle_argv
return self(_args, *_options)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/base.py", line 227, in call
ret = self.run(_args, *_kwargs)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bin/worker.py", line 162, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/concurrency/init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/kombu-2.5.0rc1-py3.3.egg/kombu/utils/init.py", line 80, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/home/gideon/gctssfc/venv/lib/python3.3/importlib/init.py", line 88, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1577, in _gcd_import
File "", line 1558, in _find_and_load
File "", line 1525, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1023, in load_module
File "", line 1004, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 869, in _load_module
File "", line 313, in _call_with_frames_removed
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/concurrency/processes.py", line 17, in
from billiard.pool import Pool, RUN, CLOSE
File "/home/gideon/gctssfc/venv/lib/python3.3/site-packages/billiard/pool.py", line 72, in
_Semaphore = threading._Semaphore
AttributeError: 'module' object has no attribute '_Semaphore'
This is with a61149b
[2013-04-24 21:33:19,846: ERROR/MainProcess] Process PoolWorker-100
Traceback (most recent call last):
File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/process.py", line 257, in _bootstrap
self.run()
File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/process.py", line 106, in run
self._target(*self._args, **self._kwargs)
File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/pool.py", line 303, in worker
ready, task = poll(1.0)
File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/pool.py", line 263, in poll
return True, get()
File "/home/ionel/projects/celery-bug-1316/.ve/local/lib/python2.7/site-packages/billiard/queues.py", line 341, in get
return recv()
AttributeError: 'str' object has no attribute 'append'
python setup.py test (using python2)
....
test_connection (billiard.tests.test_multiprocessing.WithProcessesTestConnection) ... Segmentation fault (core dumped)
What information do you need to debug? version is 2.7.3.14
I have exactly this issue (see celery/django-celery#154 and celery/celery#856) on a heroku instance - I had upgraded from 2.4.2 (?) Billiard (billiard-2.7.3.10) is the problem, I get a _billiard.so lib that crashes on startup - and worse, there is no way for me to nuke the .so within the Heroku build-set....
We really need a billiard-2.7.3.11 very soon.....very?
On Python 3.3 Threading.Semaphore
does not accept the verbose
argument:
Traceback (most recent call last):
File "/home/alga/src/celery/celery/worker/__init__.py", line 205, in start
self.namespace.start(self)
File "/home/alga/src/celery/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/alga/src/celery/celery/bootsteps.py", line 352, in start
return self.obj.start()
File "/home/alga/src/celery/celery/concurrency/base.py", line 112, in start
self.on_start()
File "/home/alga/src/celery/celery/concurrency/processes.py", line 81, in on_start
**self.options)
File "/home/alga/src/eggs/billiard-2.7.3.23-py3.3.egg/billiard/pool.py", line 835, in __init__
self._putlock = semaphore or LaxBoundedSemaphore(self._processes)
File "/home/alga/src/eggs/billiard-2.7.3.23-py3.3.egg/billiard/pool.py", line 136, in __init__
_Semaphore.__init__(self, value, verbose)
TypeError: __init__() takes from 1 to 2 positional arguments but 3 were given
Can you help me out of this issue, or did I miss something.
@ubuntu:/home/jython2.5.4rc1$ ./jython
Jython 2.5.4rc1 (2.5:723492dbab02, Feb 8 2013, 09:50:55)
[Java HotSpot(TM) Server VM (Oracle Corporation)] on java1.7.0_13
Type "help", "copyright", "credits" or "license" for more information.
import celery.task
Traceback (most recent call last):
File "", line 1, in
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/task/init.py", line 57, in
lambda: current_app.tasks['celery.backend_cleanup']
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 173, in recreate_module
new_module.dict.update(dict((mod, get_compat_module(new_module, mod))
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 173, in
new_module.dict.update(dict((mod, get_compat_module(new_module, mod))
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/compat.py", line 188, in get_compat_module
module = sys.modules[fqdn] = import_module(attrs)
File "/home/jython2.5.4rc1/Lib/site-packages/importlib-1.0.2-py2.5.egg/importlib/init.py", line 37, in import_module
import(name)
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/canvas.py", line 24, in
from celery.result import AsyncResult, GroupResult
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/result.py", line 23, in
from .app import app_or_default
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/app/init.py", line 25, in
from .base import Celery, AppPickler # noqa
File "/home/jython2.5.4rc1/Lib/site-packages/celery-3.0.15-py2.5.egg/celery/app/base.py", line 21, in
from billiard import forking as _forking
File "/home/jython2.5.4rc1/Lib/site-packages/billiard-2.7.3.21-py2.5.egg/billiard/forking.py", line 17, in
from ._ext import Connection, PipeConnection, win32
ImportError: cannot import name Connection
I'm seeing this error when using autoscaling on celery.
[2013-08-08 21:25:17,717: ERROR/MainProcess] Thread 'Supervisor' crashed: StopIteration()
Traceback (most recent call last):
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 346, in run
return self.body()
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 398, in body
pool._maintain_pool()
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1054, in _maintain_pool
self._repopulate_pool(joined)
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1039, in _repopulate_pool
self._create_worker_process(self._avail_index())
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 1045, in _avail_index
return next(i for i in range(self._processes) if i not in indices)
StopIteration
It results in this error from the workers:
[2013-08-08 21:25:18,593: ERROR/MainProcess] Process PoolWorker-5
Traceback (most recent call last):
File "/var/www/env/local/lib/python2.7/site-packages/billiard/process.py", line 271, in _bootstrap
self.run()
File "/var/www/env/local/lib/python2.7/site-packages/billiard/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/var/www/env/local/lib/python2.7/site-packages/billiard/pool.py", line 321, in worker
put((READY, (job, i, (False, einfo))))
File "/var/www/env/local/lib/python2.7/site-packages/billiard/queues.py", line 363, in put
return send(obj)
OSError: [Errno 32] Broken pipe
billiard==2.7.3.31
celery==3.0.21
django-celery==3.0.21
kombu==2.5.12
Are there plans to fix this? Previous versions of raven were able to log the f_locals information. A quick and dirty hack would be to use raven's ability to walk the f_locals information and make it JSON serializable.
In Frame.__init__
:
try:
from raven.utils.stacks import get_stack_info
except ImportError:
pass
else:
self.f_locals = get_stack_info([(frame, 1)])[0]['vars']
Having the variables in the context is very useful in Sentry.
We are on Billiard 2.7.3.9. Billiard is creating semaphore lock files, but they don't seem to always be clearing out. Every few weeks, this seems to use up our entire disk, then causing entire server to crash. Here is the the stack trace:
[2012-07-11 16:42:07,735: (celery.worker) ERROR] Unrecoverable error: OSError(28, 'No space left on device')
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/worker/init.py", line 350, in start
component.start()
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/usr/local/lib/python2.6/dist-packages/celery-2.6.0rc5-py2.6.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/pool.py", line 760, in init
self._setup_queues()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/pool.py", line 990, in _setup_queues
self._inqueue = SimpleQueue()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/queues.py", line 340, in init
self._rlock = Lock()
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/synchronize.py", line 171, in init
SemLock.init(self, SEMAPHORE, 1, 1)
File "/usr/local/lib/python2.6/dist-packages/billiard-2.7.3.9-py2.6-linux-i686.egg/billiard/synchronize.py", line 85, in init
kind, value, maxvalue, self._make_name(), unlink_immediately)
OSError: [Errno 28] No space left on device
When using subprocess.Popen
in a celery task. The worker dies with code 20 before the process returns. The offending version is 2.7.3.24, .23 works.
23:04:49 celery.1 | Process 'PoolWorker-1' pid:15622 exited with exitcode 20 23:05:02 celery.1 | Task modelconvert.tasks.convert_model[2064382f-ce91-45ec-9a45-d957c1c29f24] raised exception: WorkerLostError('Worker exited prematurely (exitcode: 20).',) 23:05:02 celery.1 | Traceback (most recent call last): 23:05:02 celery.1 | File "/Users/andi/.virtualenvs/mc/lib/python2.7/site-packages/billiard/pool.py", line 944, in _join_exited_workers 23:05:02 celery.1 | lost_ret, )) 23:05:02 celery.1 | WorkerLostError: Worker exited prematurely (exitcode: 20).
Relevant code an project is here:
https://github.com/x3dom/pipeline/blob/master/modelconvert/tasks.py#L389
https://github.com/x3dom/pipeline
Python 2.7.3/Mac, Ubuntu 12.04 x64
Python 2.6.1/Mac
[mrunge@turing billiard-2.7.3.14]$ python3 setup.py test
running test
running build_py
running egg_info
creating /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info
writing /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/PKG-INFO
writing top-level names to /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/top_level.txt
writing dependency_links to /home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/dependency_links.txt
writing manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
reading manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'Lib'
writing manifest file '/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/build/lib/billiard.egg-info/SOURCES.txt'
running build_ext
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-branch' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-html' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-package' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
/usr/lib/python3.2/site-packages/nose/config.py:259: RuntimeWarning: Option 'cover3-exclude' in config file 'setup.cfg' ignored: excluded by runtime environment
warn(msg, RuntimeWarning)
Failure: SyntaxError (invalid syntax (process.py, line 252)) ... ERROR
Traceback (most recent call last):
File "/usr/lib/python3.2/site-packages/nose/failure.py", line 37, in runTest
raise self.exc_class(self.exc_val).with_traceback(self.tb)
File "/usr/lib/python3.2/site-packages/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/python3.2/site-packages/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/python3.2/site-packages/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/mrunge/fedora/python-billiard/billiard-2.7.3.14/billiard/init.py", line 61, in
from .process import Process, current_process, active_children
File "", line None
SyntaxError: invalid syntax (process.py, line 252)
Ran 1 test in 0.001s
FAILED (errors=1)
Error in atexit._run_exitfuncs:
TypeError: 'NoneType' object is not callable
python is python3.2
I'm trying to run celery in Windows (64bit) and when i try to start a worker
celery.exe -A tasks worker
i get the following Traceback:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\site-packages\billiard-2.7.3.15-py2.7-win-amd64.egg\billiard\forking.py", line 525, in main
self = load(from_parent)
File "C:\Python27\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "C:\Python27\lib\pickle.py", line 858, in load
dispatch[key](self)
File "C:\Python27\lib\pickle.py", line 1217, in load_build
setstate(state)
File "C:\Python27\lib\site-packages\billiard-2.7.3.15-py2.7-win-amd64.egg\billiard\synchronize.py", line 110, in __setstate__
self._semlock = _billiard.SemLock._rebuild(*state)
TypeError: function takes exactly 4 arguments (3 given)
When I try to set up billiard from master branch python compiles C extension and fails with this error:
pipe_connection.obj : error LNK2019: unresolved external symbol __Billiard_conn_send_offset referenced in function _Billiard_conn_poll
When I replace static
with extern
in socket_connection.c for _Billiard_conn_send_offset function, everything compiles fine. I'm not sending PR for this because I'm not sure if this problem is local to my machine.
Hi,
I tried to run the testsuite without luck. I tried with Python 2.7 from Ubuntu 13.04.
$ make test
find Lib/ ( -name '.py[co]' -or -name '.so' ) -exec rm {} ;
find: `Lib/': No such file or directory
make: *** [clean] Error 1
Traceback (most recent call last):
File "test_multiprocessing.py", line 2071, in
main()
File "test_multiprocessing.py", line 2068, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
File "test_multiprocessing.py", line 2036, in test_main
ThreadsMixin.pool = billiard.dummy.Pool(4)
File "/usr/lib/python2.7/dist-packages/billiard/dummy/init.py", line 156, in Pool
return ThreadPool(processes, initializer, initargs)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 1597, in init
Pool.init(self, processes, initializer, initargs)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 839, in init
self._create_worker_process(i)
File "/usr/lib/python2.7/dist-packages/billiard/pool.py", line 911, in _create_worker_process
self._poolctrl[w.pid] = sentinel
AttributeError: 'DummyProcess' object has no attribute 'pid'
Maybe I do something wrong? Or is the testsuite broken?
TIA Tom
Apparently billiard changed to use atexit.register
as a decorator (at some point) for _exit_function
in billiard/util.py
, but atexit.register
does not return a function until python 2.6, making it unusable as a decorator in earlier versions.
Suggested fix:
Remove it as a decorator and instead run an explicit call to the function (atexit.register(_exit_function)
). Has the same result with backwards compatibility.
Python 3.3, Django 1.5.1, django-celery/kombu/celery/billiard from master
python manage.py celery worker --loglevel=info
-------------- [email protected] v3.1.0rc3 (Cipater)
---- **** -----
--- * *** * -- Darwin-12.3.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker: amqp://guest@localhost:5672//
- ** ---------- .> app: default:0x106cd4590 (djcelery.loaders.DjangoLoader)
- ** ---------- .> concurrency: 4 (processes)
- *** --- * --- .> events: OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. musicdiscover.tasks.add
[2013-06-08 14:37:31,477: ERROR/MainProcess] Unrecoverable error: AttributeError("'Connection' object has no attribute 'recv_payload'",)
Traceback (most recent call last):
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/worker/__init__.py", line 189, in start
self.blueprint.start(self)
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/bootsteps.py", line 352, in start
return self.obj.start()
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/base.py", line 112, in start
self.on_start()
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 461, in on_start
**self.options)
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 236, in __init__
for _ in range(processes))
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 236, in <genexpr>
for _ in range(processes))
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/celery/concurrency/processes.py", line 267, in create_process_queues
inq, outq, synq = _SimpleQueue(), _SimpleQueue(), None
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/billiard/queues.py", line 317, in __init__
self._make_methods()
File "/Users/nikitagrishko/.virtualenvs/m4h/lib/python3.3/site-packages/billiard/queues.py", line 332, in _make_methods
recv_payload = self._reader.recv_payload
AttributeError: 'Connection' object has no attribute 'recv_payload'
Hello!
I am using the latest celery 3.1rc3 version with the latest billiard, amqp and kombu.
It appears that billiard is now required to be compiled. I am receiving this error:
Traceback (most recent call last):
File "/vendor/billiard/queues.py", line 332, in _make_methods
recv_payload = self._reader.recv_payload
AttributeError: '_multiprocessing.Connection' object has no attribute 'recv_payload'
_billiard seems to have the method recv_payload whereas multiprocessing does not. This is causing billiard to fail if it is not compiled. Is compiling billiard now a requirement?
Thanks!
-tony
pip install billiard gives me the following output:
x86_64-pc-linux-gnu-gcc -pthread -O2 -pipe -march=native -fPIC -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_billiard -I/usr/include/python2.7 -c Modules/_billiard/semaphore.c -o build-2.7/temp.linux-x86_64-2.7/Modules/_billiard/semaphore.o
Modules/_billiard/semaphore.c:15:5: error: expected specifier-qualifier-list before ‘SEM_HANDLE’
Modules/_billiard/semaphore.c:209:35: error: expected ‘)’ before ‘*’ token
... and a bunch more - i think/hope that's the relevant part though.
i use gcc version 4.5.3 (Gentoo 4.5.3-r2 p1.1, pie-0.4.7) and i'm trying to install billiard on a vserver (eventhough i don't really think that's relevant)
thanks
I just upgraded to 1.1.1 and everything stopped working. Any clues?
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 91, in execute_safe
return self.execute(*args, **kwargs)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 106, in execute
return super(WorkerTaskTrace, self).execute()
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/execute/trace.py", line 59, in execute
retval = self._trace()
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/execute/trace.py", line 73, in _trace
return handler(trace.retval, trace.exc_type, trace.tb, trace.strtb)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/worker/job.py", line 111, in handle_success
self.task.backend.mark_as_done(self.task_id, retval)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/base.py", line 43, in mark_as_done
return self.store_result(task_id, result, status=states.SUCCESS)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/base.py", line 157, in store_result
return self._store_result(task_id, result, status, traceback)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/database.py", line 25, in _store_result
session = self.ResultSession()
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/backends/database.py", line 21, in ResultSession
return ResultSession(dburi=self.dburi, **self.engine_options)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 34, in ResultSession
engine, session = create_session(dburi, **kwargs)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 23, in create_session
engine = get_engine(dburi, **kwargs)
File "/home/jono/work/sites/labinventory/labinventory/extra/celery/db/session.py", line 18, in get_engine
_ENGINES[dburi] = create_engine(dburi, **kwargs)
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/engine/__init__.py", line 241, in create_engine
return strategy.create(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/engine/strategies.py", line 46, in create
dialect_cls = u.get_dialect()
AttributeError: 'NoneType' object has no attribute 'get_dialect'
I have seen #26 and #10 but I am not using Django so the Django layout hack doesn't help.
I am attempting to use celery with a pyramid project that is managed with buildout. Starting celeryd leads to ImportError in the spawned billiard processes. A quick read through the code looks like it passes sys.path to the child processes. However, because the command that is run is "'from billiard.forking import main; main()'" it is trying to import billiard.forking before unpickling the path.
I can solve this by setting PYTHONPATH before calling:
prog = 'from billiard.forking import main; main()'
os.environ['PYTHONPATH'] = ':'.join(sys.path)
return [_python_exe, '-c', prog, '--billiard-fork']
If the forked process then sets sys.path to the unpickled parent one would it harm anything to just put it in billiard?
There may well be good reason not to. If so I would propose making it optional.
I was originally thinking of having a --with-buildout option, but there may be a more general reason for preserving sys.path in PYTHONPATH when calling.
I am not familiar enough with celery to know if how you pass options from celery to billiards. So as a guess, maybe something like this in celeryd's options:
...
Option('--preserve-syspath', action='store_true', default=False, dest='preserve_path'),
Option('--with-buildout', action='store_true', default=False, dest='preserve_path'),
...
and then checking for it properly here:
SHA: 3c75659
Thoughts?
using celery = 3.0.5 and billiard = 2.7.3.11 i get following error by starting celeryd:
[2012-08-01 23:16:12,776: ERROR/MainProcess] Unrecoverable error: TypeError("init() got an unexpected keyword argument 'allow_restart'",)
Traceback (most recent call last):
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/worker/init.py", line 353, in start
component.start()
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/Users/altipard/inquant/develop/python/iqfinder/buildout/eggs/celery-3.0.5-py2.6.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
TypeError: init() got an unexpected keyword argument 'allow_restart'
Installing billiard on Windows fails if there is no C compiler present.
The current setup.py attempts to handle this by first running run_setup(True) and catching any Exceptions thrown. Unfortunately SystemExit is thrown somewhere if no compiler is found, thus the finally block (which attempts to build without the C extension) is never hit.
This can be fixed by changing Exception to BaseException, which catches SystemExit, however this is not a good idea for a few reasons. Not sure how else to fix this.
Does billiard support "Manager" feature of multiprocessing?
$ python -c "from billiard import Manager;m = Manager()"
Traceback (most recent call last):
File "", line 1, in
File "c:\Python27.32\lib\site-packages\billiard-2.7.3.23-py2.7-win32.egg\billiard__init__.py", line 87, in Manager
from .managers import SyncManager
File "c:\Python27.32\lib\site-packages\billiard-2.7.3.23-py2.7-win32.egg\billiard\managers.py", line 443, in
'xmlrpclib': (connection.XmlListener, connection.XmlClient),
AttributeError: 'module' object has no attribute 'XmlListener'
The testsuite doesn't run currently because the tests are not ported from multiprocessing (see #29).
This happens on Windows using Python 2.7.3:
[2013-07-11 22:29:19,724: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <built-in method recv of _billiard.PipeConnection object at 0x034B7090>: it's not found as __main__.recv",)
Traceback (most recent call last):
File "D:\Dev\Python\lib\site-packages\celery\worker\__init__.py", line 191, in start
self.blueprint.start(self)
File "D:\Dev\Python\lib\site-packages\celery\bootsteps.py", line 119, in start
step.start(parent)
File "D:\Dev\Python\lib\site-packages\celery\bootsteps.py", line 352, in start
return self.obj.start()
File "D:\Dev\Python\lib\site-packages\celery\concurrency\base.py", line 112, in start
self.on_start()
File "D:\Dev\Python\lib\site-packages\celery\concurrency\processes.py", line 461, in on_start
**self.options)
File "D:\Dev\Python\lib\site-packages\billiard\pool.py", line 914, in __init__
self._create_worker_process(i)
File "D:\Dev\Python\lib\site-packages\billiard\pool.py", line 1004, in _create_worker_process
w.start()
File "D:\Dev\Python\lib\site-packages\billiard\process.py", line 125, in start
self._popen = Popen(self)
File "D:\Dev\Python\lib\site-packages\billiard\forking.py", line 352, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "D:\Dev\Python\lib\site-packages\billiard\forking.py", line 146, in dump
ForkingPickler(file, protocol).dump(obj)
File "D:\Dev\Python\lib\pickle.py", line 224, in dump
self.save(obj)
File "D:\Dev\Python\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "D:\Dev\Python\lib\pickle.py", line 419, in save_reduce
save(state)
File "D:\Dev\Python\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "D:\Dev\Python\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "D:\Dev\Python\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "D:\Dev\Python\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "D:\Dev\Python\lib\pickle.py", line 748, in save_global
(obj, module, name))
PicklingError: Can't pickle <built-in method recv of _billiard.PipeConnection object at 0x034B7090>: it's not found as __main__.recv
After the update to Celery 3.0.3 (from 2.4.6) today, we had all workers crash after running for roughly 12 hours. The traceback was the following:
Jul 27 19:12:40 78.108.125.18 celery[19503:140072151082752]: [celery.worker] Unrecoverable error: TypeError('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ()) Traceback (most recent call last): File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/init.py", line 350, in start component.start() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 360, in start self.consume_messages() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 430, in consume_messages readers[fileno](fileno, event) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 635, in handle_event self._it.next() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 608, in _process_result ready, task = poll(timeout) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 997, in _poll_result return True, self._quick_get() TypeError: ('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ())
It seems it is trying to initiate an exception of ours without any arguments (besides self), which then fails and subsequently stop the worker.
billard 2.7.3.10 was installed.
I'm getting this error in a Django app that otherwise runs ok:
ImportError: No module named billiard.forking
It comes down to get_command_line running a command line without the full Django environment. If I either add the billiard path to PYTHONPATH, or "pip install billiard", that starts a game of whack-a-mole:
[2012-07-25 21:05:27,690: DEBUG/MainProcess] Consumer: Connection established.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 470, in main
prepare(preparation_data)
File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 610, in prepare
file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named django
Here's the Celery report:
software -> celery:3.0.3 (Chiastic Slide) kombu:2.3.0 py:2.6.5
billiard:2.7.3.10 django:1.4.0.final.0
platform -> system:Linux arch:32bit, ELF imp:CPython
loader -> djcelery.loaders.DjangoLoader
settings -> transport:django results:database
_wrapped: <django.conf.Settings object at 0x901b46c>
The code is https://github.com/stevage/mytardis
When running celery in my Django project, a warning is displayed about setup_environ
. I tracked it down to billiard/forking.py
at the following code section (line 437):
try:
settings_name = os.environ['DJANGO_SETTINGS_MODULE']
except KeyError:
return # not using Django.
try:
project_name, _ = settings_name.split('.', 1)
except ValueError:
return # not modified by setup_environ
The reason why this is occurring for me is because I have moved out my settings
module into a package of it's own, separating the common code from the development/production-specific configuration.
I am using Django 1.4.1, so the warning message was confusing (what setup_environ
?). I'm not sure if there would be an easy way to detect what is needed, without false-positives. Django's settings module is easy to extend and fit to one's needs, and it'd be nice to not limit that.
For now, I can stop the warning by adding the DJANGO_PROJECT_DIR
environment variable directly in manage.py
:
os.environ.setdefault("DJANGO_PROJECT_DIR",
os.path.dirname(os.path.realpath(__file__)))
hello,
I am using python 3.3 and I am trying to run celery. Now I ran Celery and I get the following error:
File "C:\Python33\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\for
king.py", line 276, in
import _subprocess
ImportError: No module named '_subprocess'
I have searched this on the web and I found that I must change it to _winapi, Is this right? After this I get another error, AttributeError: 'int' object has no attribute 'Detach' what can I do to fix this problem?
thanks in advance
I'm currently not running the latest version of celery due to this bug
celery/celery#1301
As I am now i can only use the celery --help, celery help commands but I can't seem to start celery -A tasks worker --loglevel=info
I'm following the first step celery tutorial since I'm still learning allot.
The rabbitmq server is running and my tasks.py file is exaclty the same as in the tutorial.
Here is my output.
[gert@localhost ~]$ celery -A tasks worker --loglevel=info
-------------- [email protected] v3.1.0rc1 (Cipater)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: amqp://guest@localhost:5672//
[Tasks]
. tasks.add
[2013-04-12 14:39:22,761: ERROR/MainProcess] Process PoolWorker-1
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 255, in _bootstrap
self.run()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 104, in run
self._target(_self._args, *_self._kwargs)
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/pool.py", line 262, in worker
reset_signals()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/common.py", line 19, in reset_signals
signal.signal(i, handler)
OSError: [Errno 22] Invalid argument
[2013-04-12 14:39:22,772: ERROR/MainProcess] Process PoolWorker-2
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 255, in _bootstrap
self.run()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/process.py", line 104, in run
self._target(_self._args, *_self._kwargs)
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/pool.py", line 262, in worker
reset_signals()
File "/usr/lib/python3.3/site-packages/billiard-3.3.0.0rc1-py3.3.egg/billiard/common.py", line 19, in reset_signals
signal.signal(i, handler)
OSError: [Errno 22] Invalid argument
[2013-04-12 14:39:22,789: INFO/MainProcess] Connected to amqp://[email protected]:5672//
[2013-04-12 14:39:22,806: INFO/MainProcess] mingle: searching for neighbors
[2013-04-12 14:39:23,822: INFO/MainProcess] mingle: synced with [email protected]
[2013-04-12 14:39:23,823: ERROR/MainProcess] Process 'PoolWorker-2' pid:23560 exited with exitcode 1
[2013-04-12 14:39:23,824: ERROR/MainProcess] Process 'PoolWorker-1' pid:23559 exited with exitcode 1
[2013-04-12 14:39:23,824: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)
Traceback (most recent call last):
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/init.py", line 205, in start
self.namespace.start(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 352, in start
return self.obj.start()
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/consumer.py", line 195, in start
ns.start(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/consumer.py", line 664, in start
c.loop(*c.loop_args())
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/loops.py", line 30, in asynloop
with hub as hub:
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/hub.py", line 199, in enter
self.init()
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/hub.py", line 147, in init
callback(self)
File "/usr/lib/python3.3/site-packages/celery-3.1.0rc1-py3.3.egg/celery/worker/components.py", line 121, in on_poll_init
raise WorkerLostError('Could not start worker processes')
billiard.exceptions.WorkerLostError: Could not start worker processes
Hope you guys can figure it out.
This problem is related to ticket https://github.com/ask/celery/issues/690.
When I set python trace in celeryd, i've got the last line in log(just before segfault)
[2012-06-04 15:09:03,551: WARNING/PoolWorker-2] call, /home/www/www/local/lib/python2.7/site-packages/billiard/pool.py:239
I tried to start celery several times and the result was the same.
Command — /home/www/www/bin/celeryd -B --logfile /home/www/www/celery.log --schedule=/home/www/www/celerybeat-schedule --concurrency=1
Used this method to trace — http://stackoverflow.com/questions/2663841/python-tracing-a-segmentation-fault
Unrecoverable error: 'TypeError(\'__init__() takes exactly 3 arguments (2 given)\', <class \'requests.packages.urllib3.exceptions.MaxRetryError\'>, ("HTTPConnectionPool(host=\'www.timesfreepress.com\', port=80): Max retries exceeded with url: /news/2012/sep/08/dade-county-gets-new-extension-agriculture-agent/?utm_source=twitterfeed&utm_medium=twitter",))'
Stacktrace (most recent call last):
File "celery/worker/__init__.py", line 353, in start
component.start()
File "celery/worker/consumer.py", line 369, in start
self.consume_messages()
File "celery/worker/consumer.py", line 453, in consume_messages
readers[fileno](fileno, event)
File "billiard/pool.py", line 660, in handle_event
self._it.next()
File "billiard/pool.py", line 633, in _process_result
ready, task = poll(timeout)
File "billiard/pool.py", line 1034, in _poll_result
return True, self._quick_get()
This happened in a celery task that was making an HTTP request with requests, which raised an exception and it caused celeryd to stop running entirely. You have to manually restart after this. Versions are billiard==2.7.3.12
and celery==3.0.9
Hi, I'm trying to install billiard 2.7.3.12 as part of an app I'm working on. We're using PIP (with virtualenv) to handle package management. Other packages install fine, but when it gets to billiard, I get this error:
Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe
It might not be billiard's fault, but any help would be very appreciated.
Running setup.py install for billiard
building '_billiard' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.6-universal-2.6/Modules/_billiard/multiprocessing.o
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/as/ppc/as or /usr/bin/../local/libexec/as/ppc/as) for architecture ppc not installed
Installed assemblers are:
/usr/bin/../libexec/as/x86_64/as for architecture x86_64
/usr/bin/../libexec/as/i386/as for architecture i386
/usr/bin/../libexec/as/arm/as for architecture arm
Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe
compilation terminated.
lipo: can't open input file: /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-//cc7zhCqu.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1
Complete output from command /Users/chad/Sites/post/ENV/bin/python -c "import setuptools;file='/Users/chad/Sites/post/ENV/build/billiard/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-/pip-t5W0cF-record/install-record.txt --single-version-externally-managed --install-headers /Users/chad/Sites/post/ENV/bin/../include/site/python2.6:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.6-universal-2.6
creating build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/init.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/_ext.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/_win.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/common.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/compat.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/connection.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/einfo.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/exceptions.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/forking.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/heap.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/managers.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/pool.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/process.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/queues.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/reduction.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/sharedctypes.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/synchronize.py -> build/lib.macosx-10.6-universal-2.6/billiard
copying billiard/util.py -> build/lib.macosx-10.6-universal-2.6/billiard
creating build/lib.macosx-10.6-universal-2.6/billiard/dummy
copying billiard/dummy/init.py -> build/lib.macosx-10.6-universal-2.6/billiard/dummy
copying billiard/dummy/connection.py -> build/lib.macosx-10.6-universal-2.6/billiard/dummy
creating build/lib.macosx-10.6-universal-2.6/billiard/tests
copying billiard/tests/init.py -> build/lib.macosx-10.6-universal-2.6/billiard/tests
copying billiard/tests/test_multiprocessing.py -> build/lib.macosx-10.6-universal-2.6/billiard/tests
running build_ext
building '_billiard' extension
creating build/temp.macosx-10.6-universal-2.6
creating build/temp.macosx-10.6-universal-2.6/Modules
creating build/temp.macosx-10.6-universal-2.6/Modules/_billiard
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.6-universal-2.6/Modules/_billiard/multiprocessing.o
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/as/ppc/as or /usr/bin/../local/libexec/as/ppc/as) for architecture ppc not installed
Installed assemblers are:
/usr/bin/../libexec/as/x86_64/as for architecture x86_64
/usr/bin/../libexec/as/i386/as for architecture i386
/usr/bin/../libexec/as/arm/as for architecture arm
Modules/_billiard/multiprocessing.c:318: fatal error: error writing to -: Broken pipe
compilation terminated.
lipo: can't open input file: /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-//cc7zhCqu.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1
Command /Users/chad/Sites/post/ENV/bin/python -c "import setuptools;file='/Users/chad/Sites/post/ENV/build/billiard/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/8U/8UyYeW9EF8ONfi0HOuBXA++++TI/-Tmp-/pip-t5W0cF-record/install-record.txt --single-version-externally-managed --install-headers /Users/chad/Sites/post/ENV/bin/../include/site/python2.6 failed with error code 1 in /Users/chad/Sites/post/ENV/build/billiard
Storing complete log in /Users/chad/.pip/pip.log
This makes our workers crash/hang every hour or so.
Traceback (most recent call last):
File "../lib/python2.7/site-packages/billiard/process.py", line 273, in _bootstrap
self.run()
File "../lib/python2.7/site-packages/billiard/process.py", line 122, in run
self._target(*self._args, **self._kwargs)
File "../lib/python2.7/site-packages/billiard/pool.py", line 302, in worker
put((ACK, (job, i, time.time(), pid)))
File "../lib/python2.7/site-packages/billiard/queues.py", line 377, in put
return send(obj)
IOError: [Errno 32] Broken pipe
celery==3.0.7
django-celery==3.0.4
kombu==2.4.3
billiard==2.7.3.12
We're using the SQS backend.
I'm getting an error trying to install billiard on OS X 10.8.2 related to multiprocessing.c
Requirement already satisfied (use --upgrade to upgrade): celery in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages
Downloading/unpacking billiard>=2.7.3.14 (from celery)
Running setup.py egg_info for package billiard
warning: no files found matching '*.py' under directory 'Lib'
Downloading/unpacking python-dateutil>=1.5,<2.0 (from celery)
Running setup.py egg_info for package python-dateutil
Downloading/unpacking kombu>=2.4.7,<3.0 (from celery)
Running setup.py egg_info for package kombu
Requirement already satisfied (use --upgrade to upgrade): anyjson>=0.3.3 in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages (from kombu>=2.4.7,<3.0->celery)
Requirement already satisfied (use --upgrade to upgrade): amqplib>=1.0.2 in /Users/igor/.virtualenvs/guide/lib/python2.7/site-packages (from kombu>=2.4.7,<3.0->celery)
Installing collected packages: billiard, python-dateutil, kombu
Running setup.py install for billiard
building '_billiard' extension
gcc -fno-strict-aliasing -fno-common -dynamic -Os -w -pipe -march=native -Qunused-arguments -mmacosx-version-min=10.7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/lib -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.7-x86_64-2.7/Modules/_billiard/multiprocessing.o
Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -march= switch
Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -mtune= switch
error: command 'gcc' failed with exit status 1
Complete output from command /Users/igor/.virtualenvs/guide/bin/python -c "import setuptools;__file__='/Users/igor/.virtualenvs/guide/build/billiard/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/vg/2kjxrrkn0s52zhqgbg13nq440000gn/T/pip-OUd7qO-record/install-record.txt --single-version-externally-managed --install-headers /Users/igor/.virtualenvs/guide/bin/../include/site/python2.7:
running install
running build
running build_py
running build_ext
building '_billiard' extension
gcc -fno-strict-aliasing -fno-common -dynamic -Os -w -pipe -march=native -Qunused-arguments -mmacosx-version-min=10.7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/lib -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_BROKEN_SEM_GETVALUE=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_billiard -I/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c Modules/_billiard/multiprocessing.c -o build/temp.macosx-10.7-x86_64-2.7/Modules/_billiard/multiprocessing.o
Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -march= switch
Modules/_billiard/multiprocessing.c:1: error: bad value (native) for -mtune= switch
error: command 'gcc' failed with exit status 1
----------------------------------------
Command /Users/igor/.virtualenvs/guide/bin/python -c "import setuptools;__file__='/Users/igor/.virtualenvs/guide/build/billiard/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/vg/2kjxrrkn0s52zhqgbg13nq440000gn/T/pip-OUd7qO-record/install-record.txt --single-version-externally-managed --install-headers /Users/igor/.virtualenvs/guide/bin/../include/site/python2.7 failed with error code 1 in /Users/igor/.virtualenvs/guide/build/billiard
Storing complete log in /Users/igor/.pip/pip.log
The fix for #53 is causing a new error.
Pool process error: TypeError("'NoneType' object is not callable",)
Traceback (most recent call last):
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 262, in run
sys.exit(self.workloop())
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 312, in workloop
req = wait_for_job()
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 405, in receive
ready, req = _receive(1.0)
File "venv/lib/python3.3/site-packages/billiard/pool.py", line 376, in _recv
return True, loads(get_payload())
File "venv/lib/python3.3/site-packages/billiard/queues.py", line 346, in get_payload
return recv_payload()
TypeError: 'NoneType' object is not callable
Worker processes spawned by the Pool run poll(1.0) catching (IOError, EOFError), but neither seems to be triggered when the parent exits. As a result of that, if the parent process that started the Pool goes away, the child processes do not exit.
Seems like it all boils down to Pipe.poll() never raising the error when the other side of the pipe is gone.
see: https://gist.github.com/3701493
A workaround to force the Pool processes to exit when the parent is gone could be checking that os.getppid() returns '1'.
[mrunge@turing billiard-2.7.3.13]$ python3 setup.py
File "setup.py", line 57
raise BuildFailed, BuildFailed(), sys.exc_info()[2]
^
SyntaxError: invalid syntax
Version 2.7.3.12 builds fine
Edit - edit - edit, I need to test stuff better...
This is the problem anyway:
Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import billiard
>>> import billiard.connection
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python25\lib\site-packages\billiard-2.7.3.19-py2.5-win32.egg\billiard\connection.py", line 9
from ._connection import *
SyntaxError: 'import *' not allowed with 'from .'
I needed maxtasksprechild
in Pool
for python2.6, and googled up the billiard
library. After installing version 2.7.3.31
with pip
, I simply changed import multiprocessing
to import billiard as multiprocessing
and added maxtasksperchild=1
(no other changes). But now the app fails with:
Exception: Traceback (most recent call last): File "/home/radim/workspace/vew/server/lib/python2.6/site-packages/billiard/pool.py", line 948, in _join_exited_workers lost_ret, )) WorkerLostError: Worker exited prematurely (exitcode: 155).
It didn't use to fail before (=with built-in multiprocessing
and no maxtasksperchild
). Any ideas what's wrong? Where to look for problems?
I am working the First Steps tutorial, but running into issues with the Python3 imports.
It looks like some of the _winapi imports are in the win32con or win32event modules.
My setup
I'm running Python 3.3 on Windows 7
I have installed celery, kombu, billiard and py-amqp (as amqp) from git with
>git clone <lib>
>cd <lib>
>python setup.py develop
giving me
> pip freeze
...
-e git+https://github.com/celery/py-amqp.git@735c2b409f6e1180f9401132bae2a95bdd3ab34f#egg=amqp-dev
anyjson==0.3.3
billiard==3.3.0.0rc1
-e git+https://github.com/celery/celery.git@35ef4a557c2adffd4778e795fe64f2d558dc01cf#egg=celery-dev
-e git+https://github.com/celery/kombu.git@97a67498d86d5f84fff8706a4525b960b6945853#egg=kombu-dev
pymongo==2.5.2
python-dateutil==2.1
pytz==2013b
six==1.3.0
...
Now running the celery worker
C:\Users\gav\src\python\proj>celery -A tasks worker -l debug
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 29, in
from _winapi import (
ImportError: cannot import name WAIT_ABANDONED_0
The problem lies here,
from _winapi import (
WAIT_OBJECT_0,
WAIT_ABANDONED_0,
WAIT_TIMEOUT,
INFINITE,
)
If I change this to:
import win32con
from win32con import (
WAIT_OBJECT_0,
WAIT_ABANDONED_0,
WAIT_TIMEOUT,
)
import win32event
from win32event import INFINITE
import _winapi
then run the worker
C:\Users\gav\src\python\proj>celery -A tasks worker -l debug
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 977, in
from . import reduction
File "c:\users\gav\src\billiard\build\lib\billiard\reduction.py", line 21, in
from billiard.connection import Client, Listener
ImportError: cannot import name Client
Moving to reduction.py
Changing the import of Client and Listener
if sys.version_info[0] == 3:
# from billiard.connection import Client, Listener
from billiard._connection3 import Client, Listener
C:\Users\gav\src\python\proj>celery -A tasks worker -l debug
Traceback (most recent call last):
File "c:\users\gav\src\billiard\build\lib\billiard_ext.py", line 21, in
Connection = _billiard.Connection
AttributeError: 'module' object has no attribute 'Connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\Scripts\celery-script.py", line 9, in
load_entry_point('celery==3.1.0rc3', 'console_scripts', 'celery')()
File "c:\users\gav\src\celery\celery__main__.py", line 30, in main
main()
File "c:\users\gav\src\celery\celery\bin\celery.py", line 78, in main
cmd.execute_from_commandline(argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 729, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "c:\users\gav\src\celery\celery\bin\base.py", line 300, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "c:\users\gav\src\celery\celery\bin\celery.py", line 721, in handle_argv
return self.execute(command, argv)
File "c:\users\gav\src\celery\celery\bin\celery.py", line 677, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "c:\users\gav\src\celery\celery\bin\worker.py", line 155, in run_from_argv
return self(args, *options)
File "c:\users\gav\src\celery\celery\bin\base.py", line 263, in call
ret = self.run(args, *kwargs)
File "c:\users\gav\src\celery\celery\bin\worker.py", line 168, in run
pool_cls = (concurrency.get_implementation(pool_cls) or
File "c:\users\gav\src\celery\celery\concurrency__init.py", line 26, in get_implementation
return symbol_by_name(cls, ALIASES)
File "c:\users\gav\src\kombu\kombu\utils__init.py", line 82, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\importlib__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1586, in _gcd_import
File "", line 1567, in _find_and_load
File "", line 1534, in _find_and_load_unlocked
File "", line 586, in _check_name_wrapper
File "", line 1024, in load_module
File "", line 1005, in load_module
File "", line 562, in module_for_loader_wrapper
File "", line 870, in _load_module
File "", line 313, in call_with_frames_removed
File "c:\users\gav\src\celery\celery\concurrency\processes.py", line 27, in
from billiard import pool as pool
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1831, in
class ThreadPool(Pool):
File "c:\users\gav\src\billiard\build\lib\billiard\pool.py", line 1833, in ThreadPool
from billiard.dummy import Process as DummyProcess
File "c:\users\gav\src\billiard\build\lib\billiard\dummy__init.py", line 56, in
from billiard.connection import Pipe
File "c:\users\gav\src\billiard\build\lib\billiard\connection.py", line 6, in
from . import _connection3 as connection
File "c:\users\gav\src\billiard\build\lib\billiard_connection3.py", line 977, in
from . import reduction
File "c:\users\gav\src\billiard\build\lib\billiard\reduction.py", line 27, in
from ._ext import _billiard, win32
File "c:\users\gav\src\billiard\build\lib\billiard_ext.py", line 23, in
from billiard.connection import Connection # noqa
ImportError: cannot import name Connection
I will leave this issue here, and continue following the trail of imports a little later.
never mind
Because of this circular dependency:
File "C:\Python33\lib\site-packages\billiard\connection.py", line 9, in <module>
from . import _connection3 as connection
File "C:\Python33\lib\site-packages\billiard\_connection3.py", line 960, in <module>
from . import reduction
File "C:\Python33\lib\site-packages\billiard\reduction.py", line 21, in <module>
from billiard.connection import Client, Listener
the last line in connection.py sys.modules[__name__] = connection
doesn't execute, so billiard.connection module doesn't get overridden.
Just repporting an error that I solved in adding these lines in the pool.py file:
Into the class LaxBoundedSemaphore, added the following line:
def _note(self, *arg, **args): pass
It seems that with the last version of python (3.3), the _note function in the class Semaphore has been removed. Semaphore is a class of the threading library.
Just a note that to get to this point i have installed celery, amqp and billiard from the github dev versions.
Current relevant packages
amqp==1.2.0*
anyjson==0.3.3
billiard==3.3.0.0rc1*
celery==3.1.0rc3*
kombu==2.5.12
*cloned from github and installed with python setup.py install
This is running on a fresh install of WinPython 3.3 on Windows 7.
From the First Steps of Celery docs
>celery -A tasks worker --loglevel=info
[2013-07-03 10:07:41,352: ERROR/MainProcess] Unrecoverable error: AttributeError("'SimpleQueue' object has no attribute 'wlock'",)
Traceback (most recent call last):
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\worker__init_.py", line 192, in start
self.blueprint.start(self)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\bootsteps.py", line 119, in start
step.start(parent)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\bootsteps.py", line 352, in start
return self.obj.start()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\concurrency\base.py", line 112, in start
self.on_start()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\celery-3.1.0rc3-py3.3.egg\celery\concurrency\processes.py", line 461, in on_start
**self.options)
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\pool.py", line 876, in init
self._setup_queues()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\pool.py", line 1198, in _setup_queues
self._inqueue = SimpleQueue()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\queues.py", line 372, in init
self._make_methods()
File "C:\WinPython-64bit-3.3.2.1\python-3.3.2.amd64\lib\site-packages\billiard-3.3.0.0rc1-py3.3.egg\billiard\queues.py", line 352, in _make_methods
if self._wlock is None:
AttributeError: 'SimpleQueue' object has no attribute '_wlock'
Looks like pool.py has a few os.kill calls in it, even though the start has the _kill defined to deal with that for windows compatibility.
Trying to run billiard 2.7.3.10 with Celery in Python 3.2. I get the following error (suggested fix below):
[2012-07-25 10:29:27,504: ERROR/MainProcess] Unrecoverable error: TypeError('string argument without an encoding',)
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/worker/init.py", line 350, in start
component.start()
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/concurrency/base.py", line 109, in start
self.on_start()
File "/usr/local/lib/python3.2/dist-packages/celery-3.0.3-py3.2.egg/celery/concurrency/processes/init.py", line 84, in on_start
**self.options)
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/pool.py", line 760, in init
self._setup_queues()
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/pool.py", line 990, in _setup_queues
self._inqueue = SimpleQueue()
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/queues.py", line 339, in init
self._reader, self._writer = Pipe(duplex=False)
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/init.py", line 113, in Pipe
from .connection import Pipe
File "/usr/local/lib/python3.2/dist-packages/billiard-2.7.3.10-py3.2.egg/billiard/connection.py", line 426, in
CHALLENGE = bytes('#CHALLENGE#')
TypeError: string argument without an encoding
By specifying an encoding in lines 426-428 of connection.py, it got past this error. Not sure if other things will pop up elsewhere, however.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.