chihacknight/civic-json-worker

Worker script that pulls metadata about opengov projects

civic-json-worker's Issues

Owner contributions to repos might not be counting

Just added a repo for which I am the owner and sole contributor (real community effort, I know but I was testing the add project endpoint, OK?). Even after running the update manually, it didn't update my user totals on the people page. This might mean that the owner counts are not getting added in there...

Create dummy user to backup data

As fun as it is to have my github commit history augmented by a robot committing on my behalf every four hours, I am constantly reminded that this is happening because in one hour out of four, my github user account is over its API limit. (This will also effect https://github.com/open-city/chicago-river-sewage because my account is being used to push the scraped data to github every so often.)

Not only is this slightly annoying for me, it also means that the actual work that this worker is trying to accomplish is almost certainly not completing. There may be a more efficient way to leverage the API that doesn't require that many calls so maybe that's really what we ought to do. In the meantime, making a dummy user would be appreciated.

@derekeder Any thoughts?

error submitting new repo

Tried submitting a few repos, getting an error back

Serve JSON files from EC2 instance instead of S3

Need to consolidate this junk more so that it's easier to maintain

add contributors to project_details.json

will require making another API call to contributors_url

"contributors": [
{
    "login": "hunterowens",
    "avatar_url": "https://2.gravatar.com/avatar/ff16180e6d9715d5ae17526cb83d7cfb?d=https%3A%2F%2Fidenticons.github.com%2Fc3663e10b721e5d713a55aebf8621901.png",
    "html_url": "https://github.com/hunterowens",
    "contributions": 74
  },
  {
    "login": "evz",
    "avatar_url": "https://1.gravatar.com/avatar/1d0c5faee140af87d7d6967bc946ecc6?d=https%3A%2F%2Fidenticons.github.com%2F44e80db9ed8527f429c969e804432b0f.png",
    "html_url": "https://github.com/evz",
    "contributions": 55
  },
  {
    "login": "fishmanadam",
    "avatar_url": "https://2.gravatar.com/avatar/d1529c1f159f49be15cf5fde0366b599?d=https%3A%2F%2Fidenticons.github.com%2Fdb534c76db466f7764d7e745900be88d.png",
    "html_url": "https://github.com/fishmanadam",
    "contributions": 7
  },
  {
    "login": "jtwalsh0",
    "avatar_url": "https://0.gravatar.com/avatar/05e1d117da7a62657f617bd553f4ed83?d=https%3A%2F%2Fidenticons.github.com%2F31d750398f38453e376cf435deb5fea9.png",
    "html_url": "https://github.com/jtwalsh0",
    "contributions": 2
  },
  {
    "login": "matthewgee",
    "avatar_url": "https://1.gravatar.com/avatar/1dbb7a239d59f1effc696f6424050a5a?d=https%3A%2F%2Fidenticons.github.com%2F9bf88927e24ab526ed285da976feecf0.png",
    "html_url": "https://github.com/matthewgee",
    "contributions": 1
}
]

Includes invalid-email-address Fake User in Project Contributors List

Originally reported in the open-gov-hack-night issue tracker under issue 14, searching for "invalid email address" in the People search of open-gov-hack-night will link to a real Github user:

It's an account provided by Github to indicate when there is an invalid author email address for a commit. In this case, it was found on 93 commits of look-at-cook; the email address in question was devnull@localhost.

Since this tracks a publicly curated list of projects, it's out of our control to make sure all future project contributors are also valid. Instead, we should filter out this user account from the list of contributors.

useful endpoints

add a project

$.post( 'http://civic-json-app.herokuapp.com/add-project/', data={'project_url': "https://github.com/my-name/my-project/"}, function(resp){ console.log(resp) });

delete a project

$.post( 'http://civic-json-app.herokuapp.com/delete-project/', data={'project_url': "https://github.com/my-name/my-project/"}, function(resp){ console.log(resp) });

Add participation data to project details

See: chihacknight/chihacknight.org#22

Modify civic-json-worker API to work for any city / group

This civic tech project tracking API is generically useful infrastructure. Instead of having people deploy their own versions of this API, which has proven difficult, we should just modify this service to work for multiple civic hacking groups and cities.

Our API is already up and running. Civic hacking groups would just need to submit their repo urls to the endpoint from their websites. The system would hit the Github API and get additional repo details, just like it's doing for Chicago. Brigades could then request a list of their projects to display on their own project pages.

Here's what would need to happen, as far as I can tell. (@evz, please jump in):

• The add project endpoint would need to require 'city' and 'group' params for url submissions, and store these along with the url.
• Add city and group field to the complete project_details.json file. Project pages could then pull only projects for only their cities / groups.

Make sure duplicate projects are not submitted.

Double check that when a user submits a project, it's not already in the listing. See: chihacknight/chihacknight.org#21

Project list doesn't seem to be updating

75 projects listed here: https://s3.amazonaws.com/civic-json/projects.json

But only 59 show up here: https://s3.amazonaws.com/civic-json/project_details.json

investigate meaning of updated_at

Seems to be showing erroneous dates

Need better delete mechanism

It would seem that there are duplicates still in the machine. See: chihacknight/chihacknight.org#21

XSS error when trying to delete project

Trying to delete deprecated app:

$.post( 'http://civic-json-app.herokuapp.com/delete-project/', data={'project_url': "https://github.com/smartchicago/chicago-health-atlas"}, function(resp){ console.log(resp) });

Getting an XSS error

people.json

Let's create a JSON file that shows a list of people with total number of contributions of the projects we track.

Want to set a projects page up for Oakland, CA

Hey guys, I've cloned this and want to hook it up for Oakland but I'm a bit of a newbie. For the civic worker app, it seems like I need to get an Amazon EC2 account/space where I can store the Flask app and the output JSON as a resource.
Sound like the right approach?

Error when submitting site through form

Entered "https://github.com/smartchicago/wikichicago"

Returned: Error submitting your project! Check that the URL is correct.

I also tried submitting "https://github.com/GovInTrenches/ctahardship" and received the same error.

Updating projects seems to have mixed results

As of right now, hitting /update-projects/ is emptying out the detail listing. I think this may have to do with github responding with a non-200 status code when we attempt to fetch the info about repos. I have a theory (which I will test in the morning) that this might be hitting a rate limit (they tend to be rather restrictive about non-authenticated requests).

A better approach here would be to not dump out the file before rebuilding it (duh). Another thing to do would be to actually authenticate the request so that we aren't rate limited.

Anyways, research is needed

organizations.json

Another JSON file for tracking organizations and total number of repositories (based on what we're tracking).

When a duplicate link is submitted, say so in the response

Dunno why I didn't think of this before. Huh. Related to chihacknight/chihacknight.org#21