codeforamerica/open311dashboard

Visualizing and analyzing Open311 data on one unified, customizable dashboard

Open311 Dashboard

We are creating an application to help people visualize and analyze 311 data on one unified dashboard.

Non-Python Requirements

You will need all of the libraries to run GeoDjango. If you are so inclined, you can install most of these with homebrew. KyngChaos may also be a useful resource for MacOS X users.

• GEOS
• GDAL
• PROJ.4
• PostGIS
• Mapnik2 (not explicitly required but this generates map tiles, see below.)

Getting Started

1. Set up a database using the PostGIS spatial database template.
2. Rename settings_local.example.py to settings_local.py and set your db/secretkey variables.
3. pip install -r requirements.txt
4. python manage.py syncdb creates the database schema.
5. python manage.py update_db imports initial data.
6. Generate GeoJSON files with python manage.py geojson
7. git submodule update --init adds bootstrap submodule
8. python manage.py runserver

A Word on Mapnik

Mapnik2's Wiki has instructions for installing Mapnik2 on Mac OSX and Linux. Dane Springmeyer recently released Mapnik2 binaries for Mac OSX.

Database Update Script

To update data from the Open311 API, run python manage.py update_db. This command has a few options that may be useful. The first argument is the end date, defaults to yesterday. The second argument is the number of days to download previous to that date.

• Ex: python manage.py updatedb 2011-07-01 30 will download the entire month of June.

To set up automatic updates (midnight every night):

1. crontab -e
2. Insert 0 0 * * * /path/to/dir/manage.py update_db

API Access

There are a number API calls to return JSON in various forms.

• /api
  • /tickets/(open|closed|both)
    • /[0-9]+ (n days previous to today)
    • /YYYY-MM-DD(/[0-9]+) (n days previous to the date)
    • /YYYY-MM-DD/YYYY-MM-DD (simple date range)
  • /more_info/YYYY-MM-DD (single day)
    • /YYYY-MM-DD (range)
  • /list/YYYY-MM-DD (singe day)
    • /YYYY-MM-DD (range)

More Information

Visit our project page.

Join the official Open311 Dashboard mailing list.

open311dashboard's Issues

Fix City foreign keys.

Traceback -

justin@justin open311dashboard[master]% python manage.py update_db
https://open311.sfgov.org/dev/Open311/v2/requests.xml?jurisdiction_id=sfgov.org&start_date=2011-08-28T00%3A00%3A00Z&end_date=2011-08-29T00%3A00%3A00Z&page=1
Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/Users/justin/dev/me/python/open311dashboard/dashboard/management/commands/update_db.py", line 220, in handle
    requests = process_requests(start, end, page)
  File "/Users/justin/dev/me/python/open311dashboard/dashboard/management/commands/update_db.py", line 154, in process_requests
    insert_data(requests)
  File "/Users/justin/dev/me/python/open311dashboard/dashboard/management/commands/update_db.py", line 143, in insert_data
    r.save()
  File "/Users/justin/dev/me/python/open311dashboard/../open311dashboard/dashboard/models.py", line 57, in save
    super(Request, self).save()
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/db/models/base.py", line 560, in save_base
    transaction.commit_unless_managed(using=using)
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/db/transaction.py", line 124, in commit_unless_managed
    connection.commit_unless_managed()
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/db/backends/__init__.py", line 183, in commit_unless_managed
    self._commit()
  File "/usr/local/Cellar/python/2.7.2/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 200, in _commit
    return self.connection.commit()
django.db.utils.IntegrityError: insert or update on table "dashboard_request" violates foreign key constraint "city_id_refs_id_8bcf731c"
DETAIL:  Key (city_id)=(1) is not present in table "dashboard_city".

Error when running update_db on empty database?

I'm not sure if this is an issue with my setup, but I'm getting this error whenever I run update_db:

File "/Users/joannecheng/Documents/programming/open311dashboard/env/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
django.db.utils.IntegrityError: null value in column "street_id" violates not-null constraint

I dug through update_db and the Request model for a while, and couldn't find where a new instance of the street model was created.

Autocomplete boundaries

Boundaries for autocomplete seem to not work. Typing addresses in the search box returns addresses from everywhere in the world.

Change status types for service requests

Sometimes a city can delete previously uploaded service requests. This makes it difficult to update the status of a service request. We need three types of statuses: open, closed, and unknown. This change needs to be made to the update-open-311-requests Python script.

Autocomplete Display

Autocomplete display kind of wacky. list bullets are appearing in the search result list, and the result list box stretches out to the right edge of the page.

GeoJSON Map Offset

TL;DR: Geometries projected into GeoJSON and hardcoded into the streets/neighborhoods pages are offset south approximately .0001 degree.

Preface

In all development thus far, we have been working solely with San Francisco geographic data including street centerlines and neighborhood data. This data is projected in NAD_1983_StatePlane_California_III_FIPS_0403_Feetwhich is specific to California. With plans to expand to other cities, keeping the data in this format must be reprojected into a more widely used projection. For web mapping, it was determined that the easiest SRID to use was 900913 (spherical mercator). Shapefiles were reprojected using ogr2ogr and then imported into PostGIS using GeoDjango's LayerMapping utility (See: 76d8ad2).

Problem

On the street_detail view, we would like to give uses context by displaying a map highlighting a specific block. Currently this is accomplished by loading the GeoJSON geometry into a context variable and displaying this with the PolyMaps through the po.geoJson().features() method. At lower zoom levels (below about 16), the street looks fine but small. At higher zoom levels, it becomes evident that the line is shifted a few pixels south of where it should be. In some cases, the geoJson feature is not even touching the correct street on the basemap.

The problem appears to be with projecting back to WGS84 from the database. The calculations appear to be off by somewhere around 1/10000 of a degree of latitude.

Attempted Solutions

1. @atogle suggested projecting the shapefiles into WGS84 prior to the 900913 projection and DB import. For mapnik files this appeared to work but the geoJson projections do not change.
2. The degree of difference between WGS84 and EPSG:4269 is smaller than geoJson can process so this does not affect the geoJson projection.

Attribute Error when searching for an address

When searching for 4th and King St., there is an attribute error: "'NoneType' object has no attribute 'days'."

See http://open311.ep.io/street/341/ for an example.

No module named qsstats

I get an error when I try to open the front page:

"Could not import dashboard.views. Error was: No module named qsstats"

Then when I tried to install it...

sudo pip install qsstats
Downloading/unpacking qsstats
Could not find any downloads that satisfy the requirement qsstats
No distributions at all found for qsstats

Street Index Errors

When searching for a particular street e.g., 4th and mission, we get an index error: list index out of range.

See http://open311.ep.io/street/16299/ for an example.

cannot import name ENABLE_GEO

While trying to get the dashboard running locally, I ran into this error during python manage.py syncdb. Looks like ENABLE_GEO is missing from settings.py and settings_local.py.

screenshots?

There's plenty of good instructions on what to install and how to do it.

What's limited is information about why to do this, what services are provided.

311dashboard.com, the homepage listed in the readme, is now blank

Display on Front Page for # of requests

The display on the front page for the number of requests in the past day or so is kind of screwy. I can't seem display them on the same row, in thirds.

max, min in 'streets' collection return wrong results if street is one-sided

If we define the max, min with mongoutils/migrations/streets.py and a street only has one side (right, left as defined in the shapefile) the min will always be zero. We need to do some exception handling to check for this and also need to check if this keeps our currently defined url scheme unique.

issue with "cache_page"

Has the caching functionality been completely removed? I keep getting this error:

Request Method: GET
Request URL: http://localhost:8000/map
Django Version: 1.3
Exception Type: NameError
Exception Value:

name 'CACHE_TIME' is not defined

Exception Location: /.../programming/open311dashboard/dashboard/views.py in , line 258

Commenting out "@cache_page(CACHE_TIME)" on line 257 makes everything work.

update_db script error

I'm getting an ExpatError when I try to update the database.

Traceback...

https://open311.sfgov.org/Open311/?jurisdiction_id=sfgov.org&start_date=2011-09-01T00%3A00%3A00Z&end_date=2011-09-02T00%3A00%3A00Z
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File "/usr/local/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/usr/local/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(_args, *_options.dict)
File "/usr/local/Cellar/python/2.7.1/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
output = self.handle(_args, *_options)
File "/Users/mlevans/open311dashboard/dashboard/management/commands/update_db.py", line 221, in handle
requests = process_requests(start, end, page, city)
File "/Users/mlevans/open311dashboard/dashboard/management/commands/update_db.py", line 149, in process_requests
requests = parse_requests_doc(requests_stream)
File "/Users/mlevans/open311dashboard/dashboard/management/commands/update_db.py", line 84, in parse_requests_doc
except ExpatError:
NameError: global name 'ExpatError' is not defined

Add jurisdiction_id to the records created in couchdb

Currently there aren't any keys on the service requests that indicate where the request originated. The closest value without having to use the address fields looks like the jurisdiction id that is sent as a parameter to the open311 endpoint. This should be carried through the update and parse process then submitted as a key/value on each service request.

As we add more cities this will be essential in querying across them.