parallels/rq-dashboard

Flask-based web front-end for monitoring RQ queues

rq-dashboard's Issues

Poll less frequently when nothing is happening

Crash when lazy initialise with flask app

I was trying to integrate rq-dashboard into my flask app, with the following lines:

from rq_dashboard import RQDashboard
dashboard = RQDashboard()                    # crash here!!

then initialise the app from create_app function:

dashboard.init_app(app)

However, it crash immediately on creation. I believe this has been fixed with commit: 84b9cf2

My question is, when are you going to release this piece of code that fixed 8 months ago?

› python manage.py runserver
Traceback (most recent call last):
  File "manage.py", line 7, in <module>
    import application
  File "/Users/jiequanli/Dropbox/Code/shopping2/application/__init__.py", line 4, in <module>
    import models
  File "/Users/jiequanli/Dropbox/Code/shopping2/application/models/__init__.py", line 2, in <module>
    from inventory import *
  File "/Users/jiequanli/Dropbox/Code/shopping2/application/models/inventory/__init__.py", line 3, in <module>
    from item import *
  File "/Users/jiequanli/Dropbox/Code/shopping2/application/models/inventory/item.py", line 6, in <module>
    from application.extensions import db
  File "/Users/jiequanli/Dropbox/Code/shopping2/application/extensions.py", line 48, in <module>
    dashboard = RQDashboard()
  File "/Users/jiequanli/Dropbox/Code/shopping2/env/lib/python2.7/site-packages/rq_dashboard/__init__.py", line 13, in __init__
    self.app.auth_handler = auth_handler
AttributeError: 'NoneType' object has no attribute 'auth_handler'

Add tooltips for the Compact/Empty buttons

When I run the rq-dashboard I keep getting 500 error

I can't seem to get this working, I ran the example and even did rq-dashboard on the terminal and I keep getting a 500 error.

Any ideas why?

Make a single top-level API call that enables it to integrate into your very own Flask app

Implement requeue buttons

Switch from using times to arrow package

Rewrite the queue data model

The current model of serialising jobs and putting them on queues directly poses some limitations on the advanced stuff we want to build into RQ. Mainly:

• jobs aren't addressable individually anymore (which is just as bad as the Celery behaviour)
• due to the above, messages can't be pulled off from queues, and can't be easily requeued (at least not in random order)
• tracking jobs isn't possible easily

We can make a simple change to the persistence model for jobs, namely by only storing the job ID, and putting all of the job data inside separate Redis keys (one key per job), containing the serialised (pickled) properties as values.

This solves the following problems:

• this fundamentally makes jobs addressable directly (i.e. their properties can be grabbed directly if you know their job ID)
• jobs can be taken off from queues ("unqueued") by simply removing their job key (workers could be changed to ignore any job ID on a queue if there exists no corresponding job key)
• job status information can eventually be added directly inside the job key (whatever implementation is suitable here)
• requeueing jobs is now extremely easy, as it will be equivalent to pushing the job ID onto any queue

ReverseProxie support

Please add ReverseProxie support, to make it compatible with http servers used as frontend.
http://flask.pocoo.org/snippets/35/

Support multiple Redis connections at the same time

This is kind of a thought experiment, really. I'm not sure whether this would really be useful, even. But if there is a good use case for this, it should be interesting to add this. The UI could present a drop down list to select which Redis connection to show the RQ details for.

Support for Tornado Web Server

Are there any plans to support integration with Tornado web server?

'-c' flag to pass an rq config file

It would be great if we could pass a config file akin to 'rqworker -c settings'. Thanks for this great project!

Allow requeueing

Switch to using Backbone.js for the front-end

Adjust bootstrap's default layout

Requires #11, first.

Cant run rq-dashboard - DistributionNotFound times message

Trying to run rq-dashboard.... after successful installation... but errors out... Any idea?

$ rq-dashboard
Traceback (most recent call last):
File "/usr/local/bin/rq-dashboard", line 5, in
from pkg_resources import load_entry_point
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 2603, in
working_set.require(requires)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
pkg_resources.DistributionNotFound: times

"Empty all queues" button

nice to have!

How do I block the dashboard via an auth system

i.e. in flask-admin I can hide views to users who don't have a certain role, is there a built in way to block access in rq-dashboard?

strangeness when using blueprint

I took a screencast of this issue, you can see it here.

From the same machine, rqinfo runs fine while the RQDashboard blueprint hooked into an existing flask app is very flaky.

Notably, when running the dashboard as a standalone app ($ rq-dashboard -u …) the dashboard shows none of this wonkiness.

paginate dashboard items

Currently, say if I have many jobs (10's of thousands) in a certain queue the dashboard attempts to load every job that is inside of that queue. This causes rq-dash to crash because it attempts to load too many jobs at once. I believe it would have a better UX if the job list was paginated.

The worker list would also be nice to have a count of number of workers and paginated as well.

Display finished and pending jobs

It would be useful to be able to show jobs that have finished and jobs that depend on other jobs to finish before they can be queued. The jobs in different groups (queued, succeeded, failed, waiting-to-be-queued) should probably be shown separately.

Stop polling when the page is invisible

For example when the user puts the browser tab in the background.

Show all info on a single page

Add the server/port to connect to as an argument to the console script

Show the failure queue's contents when opening the dashboard, if it is non-empty

Run rq-dashboard from heroku

Hello,

Any way to monitor the rq-dashboard running on heroku in an easy way?

Do you think that something like this would make sense:

https://github.com/jorilallo/celery-flower-heroku

Switch from optparse to click for command line argument passing

New version release

Would love to get a new official release, specifically for the worker display fix #63 Happy to patch it locally for now, but for production lets get a 3.4 ver out there when possible.

Thanks!

Issue with Apache mod_wsgi and threading

Hi,

And thanks for this valuable tool.

As part of a bigger app, I wanted to deploy rq-dashboard through Apache/mod_wsgi, but the function rq.connections.get_current_connection() returns None in a - appearently - random manner. This is an example of trace of what returns this function.

[Tue Dec 17 15:32:22 2013] [error] <redis.client.Redis object at 0x7f172896a050>
[Tue Dec 17 15:32:22 2013] [error] None
[Tue Dec 17 15:32:22 2013] [error]  None
[Tue Dec 17 15:32:24 2013] [error] None
[Tue Dec 17 15:32:24 2013] [error] None
[Tue Dec 17 15:32:24 2013] [error]  None
[Tue Dec 17 15:32:27 2013] [error] <redis.client.Redis object at 0x7f172896a050>
[Tue Dec 17 15:32:27 2013] [error] <redis.client.Redis object at 0x7f172896a050>
[Tue Dec 17 15:32:27 2013] [error]  None
[Tue Dec 17 15:32:29 2013] [error] None
[Tue Dec 17 15:32:29 2013] [error] <redis.client.Redis object at 0x7f172896a050>
[Tue Dec 17 15:32:29 2013] [error]  None
[Tue Dec 17 15:32:31 2013] [error] None

Here is an example of traceback I can read in the error log :

Traceback (most recent call last):
  File "/home/loix-devint/loix.buildout/parts/rq-dashboard-wsgiscript/rq-dashboard.wsgi", line 57, in application
    return _application(environ, start_response)
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/loix-devint/loix.buildout/eggs/rq_dashboard-0.3.3-py2.7.egg/rq_dashboard/dashboard.py", line 100, in overview
    failed = Queue('failed')
  File "/home/loix-devint/loix.buildout/eggs/rq-0.3.12-py2.7.egg/rq/queue.py", line 48, in __init__
    self.connection = resolve_connection(connection)
  File "/home/loix-devint/loix.buildout/eggs/rq-0.3.12-py2.7.egg/rq/connections.py", line 65, in resolve_connection
    'Could not resolve a Redis connection.')
NoRedisConnectionException: Could not resolve a Redis connection.

This can be "fixed" only when limiting the number of threads to "1" in the Apache config part of the WSGI app.

I suspect there is an issue in the initialization part that makes the thread local stack of connections in rq.connections._connection_stack from rq_dashboard.dashboard.setup_rq_connection()

The Apache config part :

  <VirtualHost *:80>
    ServerName rq-dashboard.tld
    ServerAdmin [email protected]

    # Note the "threads=1"
    WSGIDaemonProcess rq-dashboard.tld user=loix-devint group=apache threads=1

    SetEnv RQ_DASHBOARD_SETTINGS /home/loix-devint/loix.buildout/etc/rq_dashboard_settings.py

    WSGIScriptAlias / /home/loix-devint/loix.buildout/parts/rq-dashboard-wsgiscript/rq-dashboard.wsgi
    <Directory /home/loix-devint/loix.buildout/parts/rq-dashboard-wsgiscript>
      WSGIProcessGroup rq-dashboard.tld
      WSGIApplicationGroup %{GLOBAL}
      Order deny,allow
      Allow from all
    </Directory>
    LogLevel debug
    ErrorLog /home/loix-devint/loix.buildout/var/log/httpd/rq-dashboard/errors.log
    CustomLog /home/loix-devint/loix.buildout/var/log/httpd/rq-dashboard/access.log common
  </VirtualHost>

The /home/loix-devint/loix.buildout/etc/rq_dashboard_settings.py file:

REDIS_URL = "redis://localhost:6379/0"
RQ_POLL_INTERVAL = 2500

The wsgi bootstraper file /home/loix-devint/loix.buildout/parts/rq-dashboard-wsgiscript/rq-dashboard.wsgi

import sys
import os

sys.path[0:0] = [
    '/home/loix-devint/loix.buildout/eggs/rq_dashboard-0.3.3-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/times-0.6.2-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/simplejson-3.3.1-py2.7-linux-x86_64.egg',
    '/home/loix-devint/loix.buildout/eggs/redis-2.8.0-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/Flask-0.10.1-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/rq-0.3.12-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/python_dateutil-2.2-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/pytz-2013.8-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/itsdangerous-0.23-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/Jinja2-2.7.1-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/Werkzeug-0.9.4-py2.7.egg',
    '/home/loix-devint/loix.buildout/eggs/six-1.4.1-py2.7.egg',
    '/usr/local/lib/python2.7/site-packages/MarkupSafe-0.18-py2.7-linux-x86_64.egg',
]

# Apache + WSGI env vars pushed to os.environ
envvars = [
    'RQ_DASHBOARD_SETTINGS',
]

_application = None
def application(environ, start_response):
    global _application
    for envvar in envvars:
        if envvar in environ:
            os.environ[envvar] = environ[envvar]
    if _application is None:
        from rq_dashboard.app import app as _application
    return _application(environ, start_response)

Any hint would be appreciated.

@Annakan follows this too...

Don't format times for Europe/Amsterdam

Make it as easy to integrate in Django as it is in Flask

Make Pull frequency configurable

The default pull seems fine in development environment,but too frequent in business environment.

suggestion of new feature: Restart function

It would be useful to have a restart method, to avoid unecessary deploy.

raise RuntimeError('working outside of application context')

This appears to be a Flash issue. The redis instance is in another domain. This is my suspect. Or maybe is something in my configuration. Any ideas? Thanks in advance

  File "<stdin>", line 1, in <module>
  File "/app/.heroku/python/lib/python2.7/site-packages/werkzeug/local.py", line 336, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/app/.heroku/python/lib/python2.7/site-packages/werkzeug/local.py", line 295, in _get_current_object
    return self.__local()
  File "/app/.heroku/python/lib/python2.7/site-packages/flask/globals.py", line 26, in _find_app
    raise RuntimeError('working outside of application context')

Unable to see data on dashboard

I have made a default setup as told by you on my site. But I dont see the data on the dashboard.

I can access the redis using the default connection parameters , but I cant see data on dashboard.

How can I debug or check the error?

environment variables

def setup_rq_connection():
if current_app.config.get('REDIS_URL'):

I do not know Flash, but I believe it would be better to get REDIS_URL from os.getenv instead of Flash current_app.

With this, we can integrate RQ-Dashboard inside other frameworks easily.

db variable

What is the 'db' variable?

Is your type (int) correct?

parser.add_option('-D', '--redis-database', dest='redis_database', type='int',
                  metavar='DB',
                  help='database of Redis server')

Start using less.js

Upgrade to Twitter Bootstrap 2.0

Create a PyPI package/installer for it

Upgrade rq

The dashboard is currently pinned to rq >=0.3.8, I think pinning to at least rq >=0.4.0 would be wise, due to bugfixes that rq-dashboard does not benefit from.

500 Error when accessing /rq route.

Does it support redis connections via unix sockets? That's the only reason I can come up with for this error?

ERROR
/usr/local/lib/python2.7/dist-packages/flask/app.py: 1423
app
log_exception
2014-07-28 07:24:37,202
Msg: Exception on /rq/ [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python2.7/dist-packages/rq_dashboard/dashboard.py", line 100, in overview
failed = Queue('failed')
File "/usr/local/lib/python2.7/dist-packages/rq/queue.py", line 59, in __init__
self.connection = resolve_connection(connection)
File "/usr/local/lib/python2.7/dist-packages/rq/connections.py", line 70, in resolve_connection
raise NoRedisConnectionException('Could not resolve a Redis connection.')
NoRedisConnectionException: Could not resolve a Redis connection.

Home link does not use the url_prefix

It should be a link to "/rq" or "/#{url_prefix}", but it appears to be a simple "/"

Internal Server Error?

Failed to start:

"The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."

make rq dashboard multi-tenant?

is it possible to integrate rq dashboard to a flask app that has login and register user function? so a user would login and see their own set of jobs in the rq dashboard? a login would be required to access the rq dashboard.

Integration with rq-scheduler?

Hello,

Do you know if there is any work on integrating rq-dashboard with rq-scheduler? Would you accept a PR?

For instance, if a conditional import for rq_scheduler works, then a new section would be added to the web interface containing details on scheduled jobs. It would be optional.

It may also be possible to take the blueprint from this package and use it in a new flask app that contains the rq-scheduler functionality (as a separate package or something, if you don't want rq-scheduler code to live in this project).

Anyway, just wanted to see what other folks thought, since it'd be nice to see both in one web interface.

Rename project to rq-dashboard

Suggested feature: authorization hook

Is there a way to prevent access to unauthorized users, maybe by calling back to the parent app?

Show completed jobs and possibily inspect results

Would it be possible ? Any ETA on this ?

not retuen json when get fail jobs

[root@master rq-dashboard]# curl -v "http://127.0.0.1:9181//jobs/failed/1.json"
* About to connect() to 127.0.0.1 port 9181 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 9181 (#0)
> GET //jobs/failed/1.json HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: 127.0.0.1:9181
> Accept: */*
>
* Closing connection #0
* Failure when receiving data from the peer
curl: (56) Failure when receiving data from the peer
[root@master rq-dashboard]# curl -v "http://127.0.0.1:9181//jobs/failed/1.json"
* About to connect() to 127.0.0.1 port 9181 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 9181 (#0)
> GET //jobs/failed/1.json HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: 127.0.0.1:9181
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 16407
< Server: Werkzeug/0.9.4 Python/2.6.6
< Date: Fri, 25 Apr 2014 16:11:32 GMT
<
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="initial-scale=1.0">
  <title>RQ dashboard</title>

  <link rel="stylesheet" href="/static/css/bootstrap.min.css">
  <link href="/static/css/main.css" rel="stylesheet">
</head>

<body>

<div class="container">



<div class="row">
    <div class="span6">
        <div class="section">

            <h1>Queues</h1>
            <p class="fixed intro">This list below contains all the registered queues with the number of jobs currently in the queue.  Select a queue from above to view all jobs currently pending on the queue.</p>

            <table id="queues" class="table table-bordered">
                <thead>
                    <tr>
                        <th>Queue</th>
                        <th class="narrow">Jobs</th>
                    </tr>
                </thead>
                <tbody>
                    <tr data-role="loading-placeholder">
                        <td colspan="2">Loading...</td>
                    </tr>
                </tbody>
            </table>

            <script name="queue-row" type="text/template">
                <tr data-role="queue" <% if (d.name === 'failed' && d.count > 0) { %> class="failed"<% } %>>
                    <td><i class="icon-inbox" style="opacity: .5;"></i> <a href="<%= d.url %>"><%= d.name %></a></td>
                    <td class="narrow"><%= d.count %></td>
                </tr>
            </script>

            <script name="no-queues-row" type="text/template">
                <tr>
                    <td colspan="3">No queues.</td>
                </tr>
            </script>

        </div>
    </div>

    <div class="span6">
        <div class="section">

        <h1>Workers</h1>
        <p class="fixed intro">This list below contains all the registered workers.</p>

        <table id="workers" class="table table-bordered">
            <thead>
                <tr>
                    <th style="width:48px">State</th>
                    <th>Worker</th>
                    <th>Queues</th>
                </tr>
            </thead>
            <tbody>
                <tr data-role="loading-placeholder">
                    <td colspan="3">Loading...</td>
                </tr>
            </tbody>
        </table>

        <script name="worker-row" type="text/template">
            <tr data-role="worker">
                <td><i class="icon-<%= d.state %>"></i></td>
                <td><%= d.name %></td>
                <td><%= d.queues.join(', ') %></td>
            </tr>
        </script>

        <script name="no-workers-row" type="text/template">
            <tr>
                <td colspan="3">No workers.</td>
            </tr>
        </script>

        </div>
    </div>
</div>

<div class="row">
    <div class="span12">
        <div class="section">

        <h1>Jobs on <strong class="failed">failed</strong></h1>
        <p class="intro">
        <a href="/queue/failed/empty" id="empty-btn" class="btn btn-danger btn-small" style="float: right" data-toggle="tooltip" title="Remove all jobs from this queue (<b>destructive</b>)" data-html=true ><i class="icon-trash icon-white"></i> Empty</a>
        <a href="/queue/failed/compact" id="compact-btn" class="btn btn-small" style="float: right; margin-right: 8px;" data-toggle="tooltip" title="Remove all stale jobs from this queue (non-destructive)" ><i class="icon-resize-small"></i> Compact</a>
        <a href="/requeue-all" id="requeue-all-btn" class="btn btn-small" style="float: right; margin-right: 8px;"><i class="icon-retweet"></i> Requeue All</a>
        This list below contains all the registered jobs on queue <strong>failed</strong>, sorted by age (oldest on top).</p>

        <table id="jobs" class="table table-bordered">
            <thead>
                <tr>
                    <th>Name</th>
                    <th>Age</th>
                    <th class="narrow">Actions</th>
                </tr>
            </thead>
            <tbody>
                <tr data-role="loading-placeholder">
                    <td colspan="2">Loading...</td>
                </tr>
            </tbody>
        </table>

        <script name="job-row" type="text/template">
            <tr data-role="job" data-job-id="<%= d.id %>">
                <td>
                    <i class="icon-file" style="opacity: .5;"></i>
                    <span class="description"><%= $('<div/>').text(d.description).html() %></span>
                    <% if (d.exc_info) { %>
                        <span class="origin">from <strong><%= d.origin %></strong></span>
                    <% } %>
                    <div class="job_id"><%= d.id %></div>
                    <% if (d.exc_info) { %>

and the server return 200

10.58.104.79 - - [26/Apr/2014 00:11:31] "GET /queues.json HTTP/1.1" 200 -
127.0.0.1 - - [26/Apr/2014 00:11:32] "GET //jobs/failed/1.json HTTP/1.1" 200 -
10.58.104.79 - - [26/Apr/2014 00:11:32] "GET /workers.json HTTP/1.1" 200 -