open-city/councilmatic

DEPRECATED, replaced by https://github.com/datamade/chi-councilmatic

councilmatic's Issues

Classify legislation in to topics

Logic is in place to handle this:

• MetaData_topics tables
• displaying topic labels for each item
• topics are clickable and searchable
• topic classifier function defined in local_settings.py

Todo:

• finish topic classifier for all common cases

Committee Pages

• Show the throughput of legislation through a committee
• Show the most held-up legislation
• Show 'abnormal legislation' (hiccups in processing)

Links to Media Coverage

QueryError at /search/

Legislation is indexed, but throwing an error when searching:

http://councilmatic.opencityapps.org/search/?q=&sponsors=Pawar%2C+Ameya

Link to PDF attachment on legislation detail

If present, we should link to the PDF on Legistar. Currently showing some blank paperclip.

Link to full ordinance on Amlegal.com

Possibly using grep, link the short description text to the full, existing ordinance that is being modified.

For example, when "Amendment of Chapter 4-236" appears in the short description of the proposed ordinance, link to Chapter 4-236 in the official Municipal Code of Chicago repository on Amlegal.com.

EC2 micro instance running out of memory when scraping

SPIKE: Either limit memory usage or get a bigger instance.

Feb  4 18:51:14 ip-10-245-25-135 kernel: [1474786.933970] Out of memory: Kill process 4713 (python) score 93 or sacrifice child
Feb  4 18:51:14 ip-10-245-25-135 kernel: [1474786.933977] Killed process 4713 (python) total-vm:327172kB, anon-rss:56376kB, file-rss:0kB

Handle unresponsive pages

Related to legistar-scraper 5

Need to handle None response properly.

Search result order

For searches that are do not have keyword, return results in order of recency.

These searches include legislation type, sponsor, status, controlling body,

For example...
http://councilmatic.opencityapps.org/search/?q=&file_types=Order&file_types=Appointment&file_types=Communication&file_types=Executive+Order&file_types=Report&file_types=Ordinance&file_types=Claim&file_types=Oath+of+Office

Fetch list of Councilmembers from Legistar

Currently the council members are hard-coded for Philly. We can take advantage of the People page to dynamically generate a list

http://chicago.legistar.com/People.aspx

This should be fixed first in the legistar-scraper: https://github.com/fgregg/legistar-scrape

Show bookmarked legislation on http://chicagocouncilmatic.org/subscriptions/

On a logged-in user's subscription page, show a list of all the subscribed legislation.

Problem on legislation object view: action and result

There's a small design problem with the "actions" section. The text for the action description ("referred", "passed", etc.") are printed right before the result ("pass", "fail", etc.), which gets pretty confusing:

Actions

April 13, 2011:  City Council Referred
April 26, 2011:  Committee on Buildings (inactive) Recommended to Pass Pass
May 4, 2011:  City Council Passed Pass

From this page.

Would be good to add a little visual separation.

More leg link on front page should link to non-routine legislation

Contact info in footer

We should put contact info in the footer of pages if people find problems or want to make comments.

Legislation not being geocoded properly

All pieces of legislation seem to have the same geo-point.

Sponsors list for legislation improperly parsing on comma

In the data, sponsors are listed as [last name], [first name]. Need to parse this properly to show one full name for a sponsor.

On legislation detail page: http://councilmatic.opencityapps.org/legislation/1282392

Write django-admin command to classify legislation in-place

In addition to classifying legislation as we scrape it, we should have a command to classify legislation in the database. https://docs.djangoproject.com/en/dev/howto/custom-management-commands/

create topic dashboard for sponsor

testing some queries:

phillyleg.models.LegFile.objects.annotate(num_topic=Count('metadata__topics__topic')).filter(sponsors__id=40).order_by('-num_topic')

phillyleg.models.MetaData_Topic.objects.annotate(num_legfiles=Count('references_in_legislation__legfile')).filter(references_in_legislation__legfile__sponsors__id=40).order_by('-num_legfiles')

Identical actions are being recorded as separate events in the history, when councilmatic revisits page

For example http://councilmatic.opencityapps.org/legislation/1322564

This code is not working for some reason: a8e8af8#councilmatic/phillyleg/management/scraper_wrappers/stores/councilmatic_orm.py

Wrong title

http://councilmatic.opencityapps.org/legislation/840085

and

http://chicago.legistar.com/LegislationDetail.aspx?ID=840085&GUID=79E5A3F5-D0E2-42DF-8072-D7EB33011CDE

have different titles. I think this is because we are getting the title from the search results listing not from the detail page.

Setup application_keys

Private keys like Google Analytics are stored in application_keys.py which should be placed in your PYTHONPATH. EC2 server can't seem to locate this file.

How to bookmark is less than clear

Been getting some questions from users on how to bookmark. Apparently the star icon is not cutting it.

Search missing from legislation detail page

Many users will get to councilmatic through google directing them to a specific legislation detail page.

Once they are done, they should be figure out they can search for another piece of legislation.

After sign up - receive "account is inactive' error when trying to subscribe to a list

1. Signed up with email address.
2. Clicked email link to confirm account
3. Searched for 'procurement'
4. Clicked subscribe to this list
5. Directed to login page
6. Signed in, given "account is inactive error"

Remove hard-coded councilmembers

Philly councilmembers are listed on this page:

http://councilmatic.opencityapps.org/search/

This list is hard coded from from migration 0025

Sort of summary of legislation

At the highest level, order should be "City, Ward, Council, Small Claim, Honorifics"

Within a topic, subtopic should also be sorted by number of pieces of leg.

With a subtopic, sub-sub-topics should be sorted by numbers of pieces of leg

Routine and NonRoutine should either be ommitted or visually seperated from rest of topic summaries.

CreateWork schema.org for legislation

http://schema.org/CreativeWork

Handle incremental updates

Part 1

Currently updatelegfiles throws a UNIQUE CONSTRAINT error when trying to save a piece legislation already in the system. Should handle this gracefully and move on to the next.

Part 2

Come up with a smarter way of deciding what legislation to update. Current implementation will index the entire Legistar site, which takes several days to go through. Perhaps use datestamps? Is there a 'last updated' date we can search on?

incorrectly parsing names with suffixes

Legislator names with suffixes like Jr. are not parsed correctly. Example:

"Walter Jr. Burnett"
http://councilmatic.opencityapps.org/legislation/1082697

BeatifulSoup4 dependency in updatelegfiles

updatelegfiles depends on BeatifulSoup4:

$ python manage.py updatelegfiles
Traceback (most recent call last):
File "manage.py", line 11, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 69, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/councilmatic/chi-fork/councilmatic/councilmatic/phillyleg/management/commands/updatelegfiles.py", line 18, in
from phillyleg.management.scraper_wrappers import CouncilmaticDataStoreWrapper
File "/home/councilmatic/chi-fork/councilmatic/councilmatic/phillyleg/management/scraper_wrappers/init.py", line 1, in
from sources.insite_scraper import PhillyLegistarSiteWrapper
File "/home/councilmatic/chi-fork/councilmatic/councilmatic/phillyleg/management/scraper_wrappers/sources/insite_scraper.py", line 7, in
from bs4 import BeautifulSoup
ImportError: No module named bs4

Let the data be downloaded by the user

refactor topic lookup list

currently a lot of redundant code on the home page and councilmember view for this. should be abstracted away in to a partial.

Example searches are not valid

On the front page of councilmatic we suggest that user can search

for "Rahm Emanuel"

the search results do not return all the legislation sponsored by rahm

for "zoning"

that works

for "resolution #R2013-1"

does not return a search result

that contains R2013-2

Launch

Can't bookmark legislation from search page

getting 500 error when I try to bookmark a piece of leg from search view
bookmarking seems to work from detail view

Automatically label topics of legislation

Much of the legislation is routine and boiler plate. We should be able to learn and label legislation that are just "congratulations", "zoning changes", "signage", etc.

Ideally we could find the major routine types, so we could then identify and highlight non routine legislation.

update strategy

I think something like this might work.

• Get the most recent piece of legislation from DB, (end_leg)
• Do an empty search of legislation for all years
• Check scraped item of legislation FULLY matches end_leg. If not, then add to DB, if true terminate

Add aldermen contact info

Add email, phone, address to CouncilMember model and view

Legislation view: better link to ordinance document

One of the confusing parts of legistar is the fact that the pages aren't really an ordinance or legislative documents - they're just an "index" page for a PDF of the actual document.

But this is all obscured by the way they link to the legislative document:

Attachments: 1. Or2011-405.pdf

From this legistar page.

We're using this same approach on the legislation view. Here's a better way to do it:

Description

Issuance of permits for sign(s)/signboard(s) at 4730 W Irving Park Rd.
Read the full <ordinance/appointment/order/whatever> here. (link to the actual document)

From this councilmatic page.

The "Source" language is also confusing:

Source

Order Or2011-405

Not clear what this does until you click on the link. Better to say "See this <ordinance/appointment/etc> on the Clerk's website."

Topic Taxonomy

We should decide what topics to use. Here is what the clerk office uses

• AGREEMENTS
• AIRPORTS
• ALLEY
• APPOINTMENTS
• BONDS & BOND ISSUES
• BRIDGES & VIADUCTS
• BUDGET
• BUDGET & APPROPRIATIONS
• CITY COUNCIL
• CITY DEPARTMENTS/AGENCIES
• CLAIMS
• COMMENDATIONS & DECLARATIONS
• COMMITTEE/PUBLIC HEARINGS
• COOK COUNTY
• DECLARATIONS
• DONATION OF CITY EQUIPMENT
• EMPOWERMENT ZONES
• ENERGY/ENVIRONMENTAL ISSUES
• ENTERPRISE ZONES
• EXECUTIVE ORDERS & PROCLAMATIONS
• FINANCE FUNDS
• FOREIGN AFFAIRS
• HISTORICAL LANDMARKS
• ILLINOIS, STATE OF
• JOURNAL CORRECTIONS
• LAWSUITS/SETTLEMENTS
• MUNICIPAL CODE AMENDMENTS
• OATHS OF OFFICE
• PARKING
• PEDDLING/VENDORS
• PERMITS/LICENSE/FEE EXEMPTIONS
• PROPERTY
• PROXIES
• PUBLIC TRANSPORTATION
• PUBLIC WAY USAGE
• REPORTS
• SIGNS/SIGNBOARDS
• SOCIAL ISSUES & PROGRAMS
• SPECIAL EVENTS
• SPECIAL SERVICE AREAS
• STADIUMS/ARENAS
• STREETS
• TAG DAYS
• TAX INCENTIVES
• TAX INCREMENT FINANCING DISTRICTS
• TAX LEVIES
• TAXES/TAX LEVIES
• TRAFFIC
• TRIBUTES
• UNITED STATES GOVT.
• VETERAN AFFAIRS
• WATER
• ZONING RECLASSIFICATIONS

Should be able to bookmark legislation on legislation detail page

periodic 500 internal servor errors

visiting http://councilmatic.opencityapps.org/legislation/1195619 I got

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.23 (Amazon) Server at councilmatic.opencityapps.org Port 80

when I refresh, I get to the page. I've seen these errors a few times before, after caching was set up.

Recent legislation on homepage should be ordinances or orders

On the home page, the recent legislation should show recent ordinances or orders, not claims, resolutions. A little more likely to be interesting.

Set default location to Chicago

Addresses are currently being geocoded with Philly as the city.

Narrow by Topic is too long

Probably need to structure this better, maybe as a nested list, and possibly prune away topics that are rare..

Can't bookmark legislation from search page

On a search pages, if can't bookmark legislation. Raises a

NameError at /bookmark/

global name 'request' is not defined

opinions are not working

Setup daily jobs

Configure and run daily_jobs to regularly update legislation and send out notifications.

philly sponsors showing up in database tied to Chicago legislation

Not sure how it got in here, but

http://councilmatic.opencityapps.org/legislation/836013

View of legislation based upon geography

Let me see all legislation having to do with stuff happening within X miles from an address.

Subscription as well.