guelo/councilmatic

Oakland City Council Legislative Information

👋 Hi, I’m @guelo

councilmatic's Issues

Scraper stops after a few thousand items

The last time I ran it the scraper stopped after about 3,700 items.

HTTPS

At least login and registration.

Map sponsorships from offices to people

Some sponsors lie "Office of the Mayor" should be mappd to the people that had the tenure when the legislation was created

Request to homepage throwing a 500

Going to http://198.211.101.108:8000/ returns an internal server error

Star subscription doesn't give good feedback

Search for File # returns nothing

Searching for the exact file # of an item doesn't find it. For example
/search/?q=12-0690 doesn't find /legislation/1455551

Action not displayed on some legislation pages

Examples:

http://councilmatic.openoakland.org:8000/legislation/1485785

http://councilmatic.openoakland.org:8000/legislation/1477172

Add resend activation link

There should be a resend activation link on the login page for the "account is inactive" message

Deploy production server

Get off of runserver to a proper deployment

Scrape From field for Sponsor data

GEOSException on members page

See http://councilmatic.openoakland.org:8000/member/38

Put upcoming legislation discussions on home page.

People are more likely to use and discuss legislation that's still being discussed in session than stuff that's already been decided.

Depends on issue 28.

Signup doesn't work

Remove initial asterisk from Controlling Body

The initial "*" in front of the Controlling Body items seems inconsistent, and does not offer any useful information, as it's not explained anywhere on the page.

Site name and domain not automatically set at installation

Clicking the "More" button on the homepage leads to an empty page

The more button leads to the following URL:

http://127.0.0.1:8000/search/?q=&topics=Non-Routine

However, we don't have topics, so this doesn't have any matches. Suggested fix: home button goes to:

http://127.0.0.1:8000/search/

Default sort order of Councilmembers on members page

Some ideas: order by district, alphabetically, amount

Use calendar data to tune scraper.

Rather than just scraping daily or weekly, we could tune the scraper to dig deeper based on the contents of the calendar. Things on the agenda are more likely to change, so we could use that to make the scraper look for more volatile legislation data.

Depends on #28 .

Topics

Get the topics feature working

Scraper crashes if a LegAction has no date

scraper crashed with this exception

http://oakland.legistar.com/View.ashx?M=F&ID=1992583&GUID=F3C6C632-7298-4CC3-8261-A1D02765A29B
INFO 2013-05-14 20:26:02,063 hosted_legistar_scraper 14081 -1218922816 Scraped legfile with key u'1144400'
INFO:phillyleg.management.scraper_wrappers.sources.hosted_legistar_scraper:Scraped legfile with key u'1144400'
DEBUG:phillyleg.management.scraper_wrappers.sources.hosted_legistar_scraper:{'status': u'In Committee', 'minutes_url': None, 'sponsors': [], 'key': u'1144400', 'final_date': '', 'id': u'ced071012', 'title': u'', 'url': u'http://oakland.legistar.com/LegislationDetail.aspx?ID=1144400&GUID=82B5A413-1D8B-450F-98AC-11EED9A47AC9', 'controlling_body': u'Meeting of the Oakland City Council', 'version': '', 'intro_date': datetime.date(2012, 6, 28), 'type': u'Informational Report'} [{'key': u'1144400', 'fulltext': 'FILED OFFICE OF THE CfT 7 C( tP> OAKLAND ^ CITY OF OAKLAND 2mm28 .PMI2:.37 CITY HALL \xe2\x80\xa2 1 FRANK H. OGAWA PLAZA \xe2\x80\xa2 OAKLAND, CALIFORNIA 94612 Office of tine City Administrator ' [phone] July 10, 2012 CITY COUNCIL COMMITTEE ON COMMUNITY AND ECONOMIC DEVELOPMENT Oakland, California Chairperson Bruimer and Members of the Committee: Subject: Scheduling of Committee Meetings In an effort to keep Committees informed of outstanding items to be reviewed, staff will provide you with a list at each meeting. Staff would like your direction in scheduling these items, and any others you may know of, for Council/Committee review. City Council Committee on Community and Economic Development: 2"^ and 4"* Tuesday, 2:00 - 4:00 PM. (Brunner IChairK De La Fuente, Kernighan, Nadel: Staff: As Assigned) Anticipated Date Set: Pending/No Specific Date 1. A Report And: 1) Resolution Amending A) Resolution No. 69687 C.M.S., Passed February 2, 1993, Authorizing The Local Employment Program, And B) Resolution No. 74762 C.M.S., Passed January 29, 1999, Authorizing A Pilot Apprenticeship Program, Both Of Which Require Contractors To Hire Oakland Residents; And 2) An Ordinance Amending Ordinance Number 12388 C.M.S., Adopted December 18, 2001, To Authorize Revision Of The Local Employment And Apprenticeship Programs By Resolution (Legistar 07-1831; Rules Cmte 11/20/08, 12/4/08, 12/18/08; CED Cmte 1/13/09) 2. Conduct A Public Hearing And Upon Conclusion Adopt The Following Pieces Of Legislation: (1) An Ordinance Amending The Oakland Municipal Code Titie 10, Adding Chapter 70, Establishing A Traffic Impact Program (TIP) Pursuant To Califomia Govemment Code Sections 66000 Through 66025 (Mitigation Fee Act) For Areas In Oakland, Where Housing Developments Are Having A Major Impact On Traffic Condhions, Including Procedural Requirements For The Adoption, Imposition, And Adjustment Of Traffic Impact Fees (TIF); And (2) A Resolution Establishing A Traffic Impact Fee (TIF) 1 Item 2 CED Cmte July 10,2012 \x0c\nPursuant To Adoption And Implementation Of The Traffic Impact Program (TIP) For The Areas In Oakland, Where Housing Development Projects Are Having A Major Impact Of Traffic Condifions, Including The Adoption And Imposition Of Traffic Impact Fees (TIF) And Designated Projects; (Legistar TBD; Rules Cmte 1/25/07) For Tracking Purposes 1. (Semi-Annual) Semi-Annual Report, Regarding Status Of The City's Commercial And Residential Loan Programs, Including Job Creation Resulting From Commercial Lending; And, An Informational Report From The City Attomey, Regarding The One Stop Capital Delinquent Loans Assigned To The City Attomey's Office - Status Of Collection Efforts (from 4/11/00 Committee Meeting) (Last Report 4/13/10) 2. (Annual) Annual report on CEDA's Performance Measures (Last Report 4/7/09) 3. (Semi-Annual) A report on the Job Training Performance Standards of City-funded Workforce Development Programs (Last Report 11/29/11) 4. (Annual) Consolidated Annual Performance Report (CAPR) (from 11/16/99 Committee Meeting) (Last report 3/23/10) {Requires a Public Hearing) 5. (Annual) Reports on the Sustainable Community Development Initiative (Last report 2/28/06) 6. (Quarterly) Receive A Progress Report And Possible Action On Implementation Activities To Date Associated With The Oakland Retail Enhancement Strategy (Last report 3/24/09) Respectfully Submitted, Jeanna J. Santana City Administrator Item 2 CED Cmte July 10, 2012 \x0c', 'description': u'View Report.pdf', 'url': u'http://oakland.legistar.com/View.ashx?M=F&ID=1992583&GUID=F3C6C632-7298-4CC3-8261-A1D02765A29B', 'file': u'View Report.pdf'}] [] []
next leg record
http://oakland.legistar.com/View.ashx?M=F&ID=1992546&GUID=C2E95213-F70E-431B-9E6D-93126FF6E70A
Traceback (most recent call last):
File "manage.py", line 11, in
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/init.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(_args, *_options.dict)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
output = self.handle(_args, *_options)
File "/home/councilmatic/clone2/councilmatic/councilmatic/phillyleg/management/commands/updatelegfiles.py", line 91, in handle
self._get_new_files()
File "/home/councilmatic/clone2/councilmatic/councilmatic/phillyleg/management/commands/updatelegfiles.py", line 116, in _get_new_files
import_leg_files(curr_key, source, ds)
File "/home/councilmatic/clone2/councilmatic/councilmatic/phillyleg/management/commands/updatelegfiles.py", line 35, in import_leg_files
source.scrape_legis_file(curr_key, source_obj)
File "/home/councilmatic/clone2/councilmatic/councilmatic/phillyleg/management/scraper_wrappers/sources/hosted_legistar_scraper.py", line
legislation_attrs, legislation_history = self.scraper.expandLegislationSummary(summary)
File "/usr/local/lib/python2.7/dist-packages/legistar/scraper.py", line 198, in expandLegislationSummary
return self.expandSummaryRow(summary, self.parseLegislationDetail)
File "/usr/local/lib/python2.7/dist-packages/legistar/scraper.py", line 221, in expandSummaryRow
return parse_function(soup)
File "/usr/local/lib/python2.7/dist-packages/legistar/scraper.py", line 239, in parseLegislationDetail
fulltext = self._extractPdfText(self.host + a['href'])
File "/usr/local/lib/python2.7/dist-packages/legistar/scraper.py", line 382, in _extractPdfText
doc = slate.PDF(pdf_data)
File "/usr/local/lib/python2.7/dist-packages/slate/slate.py", line 38, in init
self.doc.set_parser(self.parser)
File "/usr/local/lib/python2.7/dist-packages/pdfminer/pdfparser.py", line 333, in set_parser
raise PDFSyntaxError('No /Root object! - Is this really a PDF?')
pdfminer.pdfparser.PDFSyntaxError: No /Root object! - Is this really a PDF?

Understand why pdf seems broken

When I turned on full_text setting option there were bugs in the scraped data, messy searches?

Figure out static data for councilmembers/districts/jobs

update database with councilmemebr jpegs on the filesystem

Search pagination numbers wrap around at a certain width

The pagination arrows and the "of" wrap around to a second line at a certain medium width. It looks fine at narrow and wide widths.

Actions are not saved by the scraper

The scraper is not saving actions because of an exception being thrown by scraper's expandHistorySummary

Save vote data to the DB

Sponsors search doesn't work

For example /search/?sponsors=Dan Kalb

We need to generalize the parts of the code that calls councilmembers "alderman" etc.

City-specific council terminology should not be hardcoded. Ideally we shouldn't need a fork to use the right names for Oakland.

Use Django template hierarchies for this?

Searching with no criteria yields many pages of blank results

Submitting a search without criteria, i.e.
http://councilmatic.yoleg.com:8000/search/?q=

should yield either no results or a list of all legislation. Instead, it returns a set of 10+ blank pages.

Add "Topics" disable configuration option

The topic seem to be taken from the beginning of the Title field for Chicago's councilmatic

Scrape public calendar for upcoming legislation

This can be used for scraper tuning and the front page, to be specified in additional issues.

Web server throwing HTTP 500 DatabaseError after latest upstream merge

There's an extra 'd' at the end of sponsor names in search results

Back out of last merge from chicago

We don't want topics or hardcoded councilmembers.

Prepare pull requests for open-city

Danny Wan listed twice

Separate Oakland CSS and html templates

Come up with optimal scraper schedule

The scraper should ideally only scrape newer and updated items since the last time it scraped. There seems to be some logic for this in the code but it is not used.

Also, feeds and other tasks in job.template.

Use Legistar rss feed to get last updated date.

Don't skip empty actions

The city clerk seems to leave empty actions for a lot of items. Currently these are not saved to the db because it throws an exception looking for an url in scraper.expandHistorySummary which doesn't exist because their is not 'Action Details' URL

Figure out why scripts don't auto create the django site and domain name

Singup and Login don't work

Add online forums

Look into embedding discourse.org or Disqus.

Note that discussion will probably become heated; look for a system with auto moderation tools and plenty of tuning.

It would be helpful if threads or subgroups could be created automatically on the introduction of new legislation, and if all posts for a given piece of legislation would go to the same thread/subgroup.

It would be nice if we could list hot threads on councilmatic.

List legislative bookmarks individually with an unsubscribe buttton.

Because you receive emails for all bookmarks at once under subscriptions, and you should be able to easily remove a bookmark.

Change footer text to say OpenOakland

Change About page content

Line height on legislation header is too tall

When the controlled by line wraps it doesn't look right, the line height is too tall.

Pagination bug on subscriptions page

It shows "Page of ." on /subscriptions

Search box hint text has bad examples

Change Rham Emmanuel and record #