18f/api.data.gov

A hosted, shared-service that provides an API key, analytics, and proxy solution for government web services.

api.data.gov

https://api.data.gov is a free API management service for federal agencies. Our aim is to make it easier for agencies to release and manage APIs.

• Program Notes for api.data.gov
• GitHub Repository for API Umbrella, the underlying technology of api.data.gov

Notes

To edit this site, edit the main branch. Changes should take effect within minutes.

The website content for api.data.gov built with Hugo.

All contributions are welcome. To submit a change, fork this repo, commit your changes, and send us a pull request.

Development

The content files to edit are in ./content. You can view your changes as you make them by running the preview server in Docker. Assuming you have Docker installed on your development computer:

git submodule update --init --recursive # Make sure to pull in git submodules

docker compose up

This will start a local web server running at http://localhost:4490/

After you're happy with your changes, commit and submit a pull request.

Deploy

Changes pushed to main should automatically be published to production within a few minutes via the CI GitHub Action.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

api.data.gov's Issues

add google webmaster tools

coordinate an 18f presentation

what are initial meetings to have

• +Noah
• Others?

add theme to terms page

https://api.data.gov/terms/

Ensure that agency can easily embed documentation on their site

Noticing the example of www.regulations.gov/developers, where they link offsite to the api documentation on api.data.gov. We want to ensure that they could easily integrate any of these parts directly into their site.

Flesh out the program's ATO

Draft pricing levels

add loose repo descriptions

Add a v. brief description for these repos:

• https://github.com/NREL/api-umbrella-router/
• https://github.com/NREL/api-umbrella-web
• https://github.com/NREL/api-umbrella-gatekeeper
• https://github.com/NREL/api-umbrella-config

create code samples for widgets, analytics sharing

If an agency is a user of api.data.gov, we want to offer them basic code samples they can use for embedding the following within their website or elsewhere:

• api analytics (see #44 for ideas)
• key registration (maybe not?)
• anything else?

Add an FAQ section or page

weigh api.data.gov usability testing

integrate other documents into GH issues

roadmap
loose notes

Enable minute-by-minute traffic graphs in the admin

The main graph of API traffic in the admin can be switched between hourly/weekly/monthly views. We've actually already implemented a minutely mode, but it's currently hidden for performance reasons (it can be a lot of data to pump back if you happen to select minutely while looking at a couple months worth of data).

This minutely view can be very helpful for diagnosing traffic spikes, and regulations.gov has been making use of this. We should do a couple little things to make this more usable for everyone:

• Expose the minutely mode button, but add logic so that you can only view 24 hours of data when you switch into this mode.
  • Semi-Related: We should probably implement something similar for hourly on a larger time scale (eg, cap the amount of hourly data you can view to a couple months).
• Adjust the time scale on the x-axis of the graph when you switch into minutely mode. Currently the x-axis just repeats the day over and over. It should at least show hour or 30 minute ticks instead.

Add a bit of explanation to the beta ribbon

A good point just made by @makaimc - it'd be good to have a link or a hover over behind the ribbon to give some context, so that developers better understand what they are getting with this (and how reliable it will be). @makaimc - any other thoughts?

Look into reported analytics discrepancies

One of our users reported that they new a client made ~194,000 requests on July 22. However, in our view of the analytics, only 24,652 requests were made. Need to investigate what's up. The account ID that reportedly made the requests is 758f2ff8-434e-4d7e-b1e0-af15d4a94f26.

Action Needed: Move api.data.gov website content to public domain

To align this project with GSA and 18F's source code policy, we'd like to move the content in this repo into the public domain (this is mainly the documentation and static web page content on api.data.gov). Right now this repo is missing a license altogether, so it would also be good to make this explicit.

To previous contributors to this project, @shawnjohnson, @nvembar, @brookeheaton: Can you confirm with a 👍 if you are okay with relinquishing all copyright claims to your prior contributions to the project, both code and text, and moving this project to the terms of CC0 Public Domain Dedication? If you're not okay with this, or have any questions, just let us know. Thanks!

First pass at FAQ/documentation

create an alert box where agencies can announce planned downtime

As we recently saw with the Regulations.gov API, wee need to provide a means for the API owner to post information on api.data.gov about upcoming downtime.

In this case, the regulations.gov team had posted a notice on the API homepage on their website, but users who only browsed api.data.gov wouldn't have known.

My suggestion would be to have a simple alert box built into each documentation page (perhaps above the description or floating on the right), as well as an alert box on the api.data.gov homepage. Have it built with placeholder content, then commented out. Then add info to the about page that tells agencies about it and how to employ it via pull request, and be sure to alert existing and new users about it and the importance that they use it.

@philipashlock @GUI @shawnjohnson

migrate to 18F org

this repo, that is.

clarify where to file issues for the service itself

Should the issue tracker for this repo just be for the project website at api.data.gov or should it also be the issue tracker for the service itself? Or, would the latter be better suited elsewhere, like here?

api key emails getting caught in spam filter

We've had two reports of the api key emails getting caught in a spam filter. I'm not sure whether those are flukes but it's worth keeping an eye on and possibly testing more around.

Should 'edit' links go to master or gh-pages branch

The edit links at the bottom of pages seem to go to the master branch of this repo, but is it that or the gh-pages branch that they'd want to edit in order to suggest an edit to the site?

If gh-pages, would it make sense to possibly remove the master branch?

Potential updates to terms

• Should we build in more aspects of an SLA here?

add an api-umbrella explanation

perhaps a faq or section of /about.

The goal would be to have just 1 sentence explaining what each of the repos are and how they interact.

Sketch out an initial list of asks for outside developers

Enable Dashboard

It would be nice to have a few chart and tables available for the administrative dashboard. Maybe start simple with an activity graph of the last few hours, already filtered to certain API's.

Need POST(/PUT/PATCH) support

FBOpen is rapidly heading in a direction where we need to have a write API. There are other 18F projects (cc @dwcaraway) which will need this capability, too.

Stating the obvious, we will also need a way to control roles and permissions for API keys, since we'll need to limit write access to certain subsets of our users.

Offer basic metrics page

We're beginning to get recurring requests for certain data points, namely:

• Number of agency customers live
• Number of agency customers in development
• Number of API keys created to date
• Number of API keys created, by month
• Number of API calls processed, by month

plan team travel

• api meetup
• fall team-wide meeting
• initial visit
• half-day legal/security events
• an event focused just on api.data.gov?
• what else?

coordinate a digitgov presentation

Shooting to schedule during the August visit

Fix admin login for MyUSA

The MyUSA login for the administrative side of api.data.gov is currently not working. Need to investigate and look into upgrading the omniauth-myusa gem.

HTTPS by default

HTTPS is already configured at https://api.data.gov. So, api.data.gov should make it the recommended and default protocol.

I'm sure the team here already understands the various benefits of HTTPS, and I don't mean to preach to the choir, but here are a few:

• Broader client-side compatibility. Any APIs under api.data.gov wishing to support CORS or JSONP for in-browser operation will be completely blocked on HTTPS websites, as active mixed content.
• Enhanced privacy for apps and users using APIs at api.data.gov -- HTTP headers and query string parameters (among other things) will be encrypted. For APIs accepting particularly sensitive data (like lat/long or zip), this is critical.
• The primary benefit of HTTPS: a stronger guarantee that you're speaking with the real api.data.gov.

Much of the performance hit involved in SSL can be mitigated - @igrigorik's excellent istlsfastyet.com contains many references for doing this.

There are a few ways this issue could be tackled:

• Update the documentation to always use https:// in the examples.
• Update all links to api.data.gov to use https://.
• Begin redirecting all http:// requests to https:// for the main api.data.gov website.
• Begin redirecting all http:// requests to https:// for any contained API.
• Set a HSTS header that tells clients to always use https:// going forward.

sketch out initial marketing campaign

ensure customer outreach to all clients

update theme (and possibly engine) for /contact

http://api.data.gov/contact needs updating theme-wise.

But I also wonder if it's better just to point people to the issue tracker? Up to you, though.

Suggestion: make it clear that api.data.gov is not an API to data.gov

Hi there, I recently stumbled upon api.data.gov and applaud all the awesome initiatives like this one happening right now - thank you!

One confusion I think should be address on this site is that even though it is a subdomain of data.gov, it isnt actually an API of data.gov whose APIs are documented at http://www.data.gov/developers/apis.

Two potential solutions:

1. Make it clear on the front page of api.data.gov what the site is (the list of web services in the left column does this) but also link to the data.gov APIs
2. Choose a different subdomain (probably easier said than done)

confirm style guide

API.Data.gov v. api.data.gov

Update theme to match www.data.gov

Data.gov's redesign is fully launched and has been rolled out to the rest of it's subdomains, so we should do the same for api.data.gov.

@GUI - let me know what would make that easy and we can try to line things up so as to keep this a minimal effort.

This will address the Data.gov issue #390

Swagger docs not working in IE

It was reported that our swagger docs on api.data.gov aren't working in Internet Explorer. Need to verify which versions of IE this is affecting and figure out if this is something we're doing wrong or a bug on Swagger's end.

transition GA to 18F

research pulling in outside api analytics into api.data.gov

Re-enable the HTTP caching layer

The HTTP caching layer of api.data.gov is currently disabled. We need to get it re-enabed, but we need to validate some things first.

The caching layer got disabled when we encountered this bug with OPTIONS requests. We've worked around the issue on our end, and we've also got the underlying issue patched in NodeJS. However, while this was disabled, I discovered that some of our unrelated Varnish caching servers have been experiencing periodic failed requests due to this gzip+chunked responses+streaming issue in Varnish. The issue is very sporadic and seems to affect a very small percentage of requests, but it's still concerning, and I'd like to sort it out before re-enabling caching. I'm not 100% sure this same problem is happening on our api.data.gov servers, so that needs to be verified, but I suspect it might be if Varnish gets re-enabled (since we saw this issue on a few other different Varnish setups where streaming was enabled).

So before re-enabling caching:

• Verify our workaround for the failed OPTIONS requests works for the fbopen team.
• Try to come up with a test case to reproduce the sporadic issues seen when streaming is enabled in Varnish and the backend sends back chunked responses that are gzipped.
• Assuming the same Varnish streaming issue is present in our stack, I believe the issue is somewhat inherit with Varnish 3 and streaming enabled with no known workarounds. In that case, we have a few options:
  • Disable streaming in Varnish: I'm pretty sure this fixes the problem, but then streaming is disabled, which I'd like to avoid (since I believe some users are dependent on it).
  • Upgrade to Varnish 4: Varnish 4 makes streaming the default (it was not in Varnish 3), so I'd assume this issue has been fixed in Varnish 4 (but more testing would be needed).
  • Switch the HTTP cache to Apache Traffic Server. I've started using this on another project and have been impressed. I've tested for the same gzip+streaming issues and haven't seen it. Some of Traffic Server's other features like it's clustering and persistent cache are also interesting. But it's doesn't seem quite as configurable as what Varnish allows with VCL, so we'd need to figure out how to handle things like Surrogate-Control headers.

Agency-specific admin privileges

When an administrator logs into the admin part of api.data.gov, they should only be able to view their own agencies APIs configuration, analytics, etc. They should then also have the ability to publish their own API backend configuration without publishing any other pending configuration changes from other agencies.

This ticket has more of the technical details: NREL/api-umbrella#9

Create expansion plan

Relative date ranges (date math) for analytics queries

In the analytics admin, you can select a range of dates to perform your analytics queries on. The date range picker also includes some common ranges like "last 7 days" or "this month". If I select the last 7 days on 2014-07-21 then I'll end up with a URL like:

/admin/#/stats/users/tz=America%2FDenver&start=2014-07-15&end=2014-07-21&interval=day

However, if I bookmark that URL or share it with other admins, they will always be viewing this 7/15-7/21 time frame. This static view of dates may be desirable in some circumstances, but I think if I've selected the "last 7 days" option, then most of the time I want a URL that will always show the last 7 days, regardless of when I access. This would allow you to bookmark that URL or shared URLs for analytics queries you regularly have to pull (for example, if you have to pull your active users every month).

My general thoughts of how this might work:

• If I select any of the date picker's relative date ranges ("today," "yesterday," "last 7 days," "last 30 days," "this month," or "last month") the URL I'm given should use relative dates so if I visit that URL again, I'm still viewing that date range relative to today.
• If I select "Custom Range" from the date picker and then choose specific dates, then the URL should use static dates so loading that URL will always show the specific dates you picked.
• We should add more relative date ranges for common ranges, like YTD, this fiscal quarter, last fiscal quarter, last fiscal year, and maybe others.
• The URL should support elasticsearch's date math. We should use this format where possible for the relative dates (like "last 30 days"), but some other ranges may require our own custom URL parameters (like if we add fiscal quarter ranges). Basing the feature on this date math feature would allow URL hackers to save their own reports for odd date ranges (eg, I want to always look at the last 3 weeks for some reason), without us necessarily having to them in the UI right now (but it would be nice to figure out a more elegant approach in the UI).

All of this really builds towards being able to save common reports, which has been on the radar for a while: NREL/api-umbrella#16 But perhaps just by making the URLs bookmarkable with relative date ranges, that accomplishes most of that goal.

@shawnjohnson Thanks for the idea to use elasticsearch's date math! I hadn't seen that before, but that seems like it'll definitely make this easier and more flexible to implement. Otherwise, does what I've outlined here seem logical to you? Mainly that the date range picker will always result in relative URLs unless you pick "Custom Range" and explicitly pick dates? And while I think exposing the date math in the URL will give you whatever flexibility you need, I am curious what relative date ranges you commonly run queries for, since I'd like to add those to the date range picker. I commonly have to run reports based on fiscal quarters, so I'll probably be adding those ranges, but if you have other frequent ranges, those would be great to know.

move data.gov to GSA

Any reason for us not to move the data.gov link on the homepage to the GSA page? It's a gsa site and I'm inclined to keep that consistent.

API key accounts

Right now the key signup process is: fill out the signup form, immediately get a key. If you want a new key, you fill out the form again.

There's been periodic discussion over creating more of an "account" concept, so users might have a password to login and view and manage multiple API keys they might have for different projects. This could also be a place for users to view analytics on their own API key usage. We've also discussed using MyUSA for this account login mechanism.

I think we have some parties that may still want the no-realy-signup process for getting api keys, but I think others are interested in more of a required signup process. So maybe make it optional? For example, fill out your name and e-mail to get a key, or signup for an account to manage your keys, view analytics, etc? I'm not exactly sure how all this might work, or what we really want to do, so this would obviously require a bit more discussion.

Allow users to view their own API key usage

We've had several requests from API users who would like to view usage metrics on their own API keys. This might be similar to the analytics we give to admins, but given to users to view on their own API keys.

resolve /community page

http://api.data.gov/community/

I found that page while googling and can't tell how it exists (I don't see that in the gh-pages or master branch). Any thoughts on that? Any other pages like this exist?

If possible, it'd be good to axe that page entirely so it's not left out there like a shell.

consider a news/blogposts section to the site

• http://gcn.com/blogs/pulse/2013/08/agency-apis.aspx
• http://www.programmableweb.com/news/worldwide-government-open-data-infrastructure-takes-shape/2013/08/06