Monitor not Drawing/Displaying Traffic for Remote Worker

[mher]

Monitor not Drawing/Displaying Traffic for Remote Worker

[cutso]

for late, i have the same problem, but solved it well.
when most guy create virtual host and user, they forgot to set a tag. rabbitmq user's tag.

so, just check your rabbitmq user's tag, make sure it is setting to administrator or management.

you can use rabbitmq-plugin command or it's web management.

Thank you!

[sterrell]

sorry - closed on accident.

[mher]

flower gets all info from broker. Technically there is no difference a worker is launched on the local or remote host.

Can you confirm the stats (number of completed tasks) for all workers and queues are correct? How do you launch workers, flower? Can you show your config?

[sterrell]

There are only a few tasks getting executed in my prototype, so I have confirmed that the number of completed tasks for each queue is correct. The Workers tab shows both workers, their individual queues, and the correct number of completed tasks. So, it can be seen that Flower is getting notified of all the tasks with no problem.

I used <server1> in place of real server name below.

Here's how I launch each worker:

Server 1
(rabbitMQ already running on server1)
/etc/init.d/celeryd to start (unmodified from generic-init.d/celeryd in github)
/etc/default/celeryd settings:

CELERYD_NODES="w1"
CELERYD_OPTS="--time-limit=3600 --workdir=/home/ubuntu/celery --app=proj -l info -Q ANALYTICS"
CELERYD_CHDIR="/home/ubuntu/celery/"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_MULTI=/usr/local/bin/celeryd-multi
CELERYCTL=/usr/local/bin/celeryctl
CELERYD_USER="ubuntu"
CELERYD_GROUP="ubuntu"
CELERY_CREATE_DIRS=1

Applicable contents from server1's proj/celery.py:

from celery import Celery

celery = Celery('proj.celery',
                broker='amqp://guest@<server1>//',
                backend='amqp://guest@<server1>//',
                include=['app.end2end'])

Server 2 - almost identical as server1
/etc/init.d/celeryd to start (unmodified from generic-init.d/celeryd in github)
/etc/default/celeryd settings:

CELERYD_NODES="w1"
CELERYD_OPTS="--time-limit=3600 --workdir=/home/ubuntu/celery --app=proj -l info -Q VIZ --broker=amqp://guest@<server1>//"
CELERYD_CHDIR="/home/ubuntu/celery/"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_MULTI=/usr/local/bin/celeryd-multi
CELERYCTL=/usr/local/bin/celeryctl
CELERYD_USER="ubuntu"
CELERYD_GROUP="ubuntu"
CELERY_CREATE_DIRS=1

Applicable contents from server2's proj/celery.py:

from celery import Celery

celery = Celery('proj.celery',
                broker='amqp://guest@<server1>//',
                backend='amqp://guest@<server1>//',
                include=['forward'])

Everything for the server2 (VIZ queue) worker looks fine on Flower except the lack of the tasks getting plotted, so my gut tells me this might be display bug. I am displaying this on Mac OS Safari in case that makes a difference.

Please let me know if you need any Flower screenshots.

Thanks!

[mher]

In your configuration both workers have the same name: CELERYD_NODES="w1". Renaming the second worker CELERYD_NODES="w2" should fix the issue.

[sterrell]

That was a good idea. However, I tried changing the name to w99 and got the same results. Furthermore, the startup script appends the hostname to the end of the w1 or w99 (as can be seen by the ps -elf and the Flower UI itself), so I don't think that is part of the problem. For example, ps -elf reveals -n w99.ip-10-202-141-97 was used in the actual in the command.

The problem has worsened in a way that is unrelated to the name change. Now I am not getting the traffic graphed for the server1 worker, either. Yet, the tasks are still showing up as completed successfully on the Tasks tab. This extension of the original problem started when I started killing/restarting everything. :(

At least now I know that the problem I'm having is not limited to remote workers. I'll try restarting rabbitMQ and the browser as well... But I'm not sure I'll have much more time to troubleshoot since we're starting a new work sprint today.

[sterrell]

rabbitMQ and browser restart did not help.

[amezhenin]

I have the same issue with flower. "Broker" and "Monitor" tabs doesn't work for remote broker(RabbitMQ + Redis as result backend). I configure Celery with Django's settings.py. Should I post it?

[mher]

Are you using --broker_api option?

[amezhenin]

No, I use

python manage.py celery flower --broker=amqp://<remote_host>

[mher]

--broker_api option is required for Broker tab. Flower uses RabbitMQ management api to get info about queues. Also, make sure RabbitMQ management plugin is enabled (see flower log for instructions).

python manage.py celery flower --broker=amqp:// --broker_api=http://guest:guest@localhost:15672/api/

[amezhenin]

My 15672 is not open to Internet, all requests to management plugin is proxied by nginx.

--broker_api=https://<remote_host>/rabbitmq/api/

But this doesn't help, I see new messes in log:

[E 130516 13:15:45 state:101] Failed to inspect the broker: 401 Client Error: None

I'm interested in 'Monitor' tab, more then in 'Broker' tab, btw.

[mher]

RabbitMQ management plugin should be accessible from flower. Make sure you are using valid username/password in --broker_api.

[mher]

Monitor tab should work without --broker_api. Are there other error messages in the log?

[amezhenin]

There is only 1 error:

$ python manage.py celery flower --broker=amqp://<remote_host>
...
[E 130516 14:34:19 web:1732] Could not open static file 'favicon.ico'
...
[I 130516 14:38:12 web:1514] 200 GET /monitor/succeeded-tasks?lastquery=1368700689.369 (127.0.0.1) 37.38ms
[I 130516 14:38:12 web:1514] 200 GET /monitor/completion-time?lastquery=1368700689.448 (127.0.0.1) 21.07ms
[I 130516 14:38:12 web:1514] 200 GET /monitor/failed-tasks?lastquery=1368700689.468 (127.0.0.1) 30.19ms
[I 130516 14:38:15 web:1514] 200 GET /monitor/succeeded-tasks?lastquery=1368700692.378 (127.0.0.1) 30.82ms
[I 130516 14:38:15 web:1514] 200 GET /monitor/completion-time?lastquery=1368700692.445 (127.0.0.1) 34.25ms
[I 130516 14:38:15 web:1514] 200 GET /monitor/failed-tasks?lastquery=1368700692.48 (127.0.0.1) 29.61ms
[I 130516 14:38:18 web:1514] 200 GET /monitor/succeeded-tasks?lastquery=1368700695.37 (127.0.0.1) 21.57ms
[I 130516 14:38:18 web:1514] 200 GET /monitor/completion-time?lastquery=1368700695.449 (127.0.0.1) 23.70ms

[dmaniloff]

Hello,

I'm experiencing the same issue and my log looks exactly the same. When firing new tasks I can see them going through in celery's log and I also see flower's graph time-scale changing but no data is displayed.

Thanks in advance.

[mher]

@dmaniloff Are you using RabbitMQ?

[dmaniloff]

Yes I am.
On Jun 3, 2013 2:01 PM, "Mher Movsisyan" [email protected] wrote:

@dmaniloff https://github.com/dmaniloff Are you using RabbitMQ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/66#issuecomment-18859100
.

[mher]

Please provide the RabbitMQ version along with the command line you are using to launch flower.

[JuneKelly]

I'm experiencing the same issue, but with MongoDB as the broker.
I have two machines set up as celeryd workers, and one other machine hosting a celerybeat process and the flower app. The mongoDB broker is on another cluster of machines again. Overall the setup is working, tasks are distributed to the workers and they are executed without trouble.

In flower, the Tasks tab shows started/completed tasks properly but the Monitor tab never show's any activity in the various graphs. flower was started with the following command:

$ celery flower --config=config.test 

the results are the same when calling flower like:

$ celery flower --broker=url/for/broker

The config module contains the BROKER_URL setting.

I have looked at the browser dev tools network activity, and it shows that each of the requests to the /monitor/succeeded-tasks (etc) receives a response like:

{"celery_worker.devcelery2": 0, "batch_scheduler.devcelery0": 0, "celery_worker.devcelery1": 0, "flower.devcelery0": 0}

so it seems that the problem would be in the flower server rather than the client side.

In addition, when this same celery setup is run locally, (with both the broker, celery workers and flower running on the local machine), the flower monitor tab does work, and shows activity properly.

If there is any other info that would be relevant I can look into it.

[mher]

I think the problem is in clock synchronization. Timestamp is used to query the latest stats. Can you try to sync the clocks on your nodes?

[JuneKelly]

Thanks for the tip,

We saw that the clocks on the servers had drifted a bit from each other. We res-ynced the clocks and as they came into sync the events started showing up in the Monitor graph.

However, the Monitor would only work when the hosts were within one second of each other. So it looks like we have an issue with clock drift in our environment.

One thought, would it be possible for flower to account for a drift of say up to a few seconds? From my observation it looked like the monitor would not pick up events if they were out of sync by more than one second.

Thanks for the help!

[ask]

One thought, would it be possible for flower to account for a drift of say up to a few seconds? From my observation it looked like the monitor would not pick up events if they were out of sync by more than one second.

Seems weird that events behind a second does not show.

Events in the upcoming Celery 3.1 uses a logical clock instead of the timestamp
to sort events (though it falls back to the timestamp if the clocks collide)

Ask Solem
twitter.com/asksol

[mher]

On Jul 11, 2013, at 7:19 PM, Shane Kilkelly wrote:

One thought, would it be possible for flower to account for a drift of say up to a few seconds? From my observation it looked like the monitor would not pick up events if they were out of sync by more than one second.

I've added updateInterval option

http://localhost:5555/monitor?updateInterval=5000

Sets the update interval to 5 seconds (the default is 3)

[mher]

I'm closing this issue. Please reopen it if synching clocks doesn't help.

[SeanHayes]

I have the same problem, no info displayed at all on any Flower tab.
RabbitMQ 3.0.4
Django==1.6
celery==3.1.2
django-celery==3.1.1
flower==0.6.0

I'm running:
./manage.py celery flower --address=0.0.0.0 --port=5555 --basic_auth=user:password --broker=amqp://guest@:5672/ --broker_api=http://guest:guest@:5672/api/ --persistent --log_to_stderr

Clocks on both servers (app server where I run flower and the celery/RabbitMQ server) are synced.

[mher]

@SeanHayes Have you tried #204 patch?

[SeanHayes]

@mher I'm now using flower 0.7.2 which I think cntains those changes, and it's still not working.

Does flower only show data for tasks that execute while flower is running? Or does it show historical data too?

[SeanHayes]

I discovered that running "/usr/local/bin/celery flower --broker=amqp://whatever" works but "./manage.py celery flower --broker=amqp://whatever" doesn't.

[mpaolino]

I had this same issue once, make sure the clock in all your machines is setup correctly. Run a NTP client on all of them.

[mevinbabuc]

I have the same issue. In the developer console i can see this error.
Uncaught TypeError: Cannot read property 'length' of undefined from d3

[fccoelho]

I am having this issue when running everything on the same machine, so it can't be a clock synchrony issue... I only get the queued tasks graph. not the other three...