solararbiter/solarforecastarbiter-api

HTTP API and database schema for the Solar Forecast Arbiter

solarforecastarbiter-api's Issues

get obs/forecast data doesn't support csv

Looks like get obs/forecast data currently only supports JSON. It should also support csv.

https://dev-api.solarforecastarbiter.org/#tag/Observations/paths/~1observations~1{obs_id}~1values/get

https://dev-api.solarforecastarbiter.org/#tag/Forecasts/paths/~1forecasts~1{forecast_id}~1values/get

add authentication via JWT

JSON missing/null value parsing

If we're expecting users to serialize missing or null values into JSON as the javascript null object, Flask will parse it into None and pandas into a NaN.
If we need to be more permissive with missing data in JSON, we'll have to find/replace accepted NaN values.

draft of /reports blueprint

handle mysql disconnection gracefully

Don't want a disconnect to send a 500

Update to work with APISpec v1.0.0

While working on #35 I found that APISpec now has a 1.0.0 release. Changelog can be found here: https://apispec.readthedocs.io/en/stable/changelog.html

intro documentation

We should add more intro material that outlines how we expect most people to interact with the API: POSTing and GETing preexisting observations. An important item to clarify is one csv per observation (so one csv per data type) and therefore likely multiple POST/GET operations per site.

create the docs repo and swagger/redoc docs for api

Integration tests

On top of existing Unit tests we need 'integration' tests to ensure the system functions as expected. the requests used to test the dev-storage backend #39 are a good place to start for these.

• storage creates a uuid for new metadata objects, adds 'Provider' field.

Accept file uploads

Accept file uploads with the multipart/form-data content type for forecast and observation endpoints.

draft of /trials blueprint

add flask-talisman for security headers

other characters allowed in names

Let's list other possible characters that might be allowed in names. Some characters that I never want to allow are <>;@&=

dashboard link

"Most users will interact with the API indirectly through actions on the dashboard."

dashboard links to dashboard.solarforecastarbiter.org, which doesn't exist. Can we instead link to @lboeman's new page proposed in SolarArbiter/SolarArbiter.github.io#52 ?

draft of /forecasts blueprint

including response schemas, request schemas, methods, routes

remove irradiance loss factor

as discussed here SolarArbiter/SolarArbiter.github.io#60 (comment)

Handle values updates

Currently, inserting values with the same timestamps throws a 1062 duplicate primary key error. We may need to add ON DUPLICATE KEY UPDATE to the INSERT statements in the store procedures for forecasts, observations and cdf_forecasts.

Deletion 404s

Since mysql fails silently when something doesn't exist, we should check for this and return something. Currently, deleting an object or trying to delete a missing object returns a 200 and trying to delete something you don't have access to throws a 404.

missing values on marshmallow deserialize

Many fields in the schema are not required but have no missingor default values. We should probably set these fields so we know the once place where default values come in. The database storage_interface also won't work as is since the fields are not set at times.

Deletion 403

When user has read permissions but cannot delete an object.

enforce regular data uploads

once the data model is redefined for observations

create test database

Include metadata links in value responses

Add a metadata link to csv header comments (as well as uuid) and _links attribute to json.

host the api on openshift

value type to interval value type

As in SolarArbiter/SolarArbiter.github.io#62

Turn nested site objects into links

Forecasts and Oservations should not include the full nested site object, but rather a _links object with site attribute that links to site metadata.

API doc issues

Need to address most of these before sending out stakeholder feedback request:

"Authorization request must include client_id='c16EJo48lbTCQEhqSztGGlmxxxmZ4zX' and audience='https://api.solarforecastarbiter.org'" can we make it look like this: "Authorization request must include client_id='c16EJo48lbTCQEhqSztGGlmxxxmZ4zX' and audience='https://api.solarforecastarbiter.org'" or similar?

The get site forecasts section says "Get all forecasts associated with site that user has access to". Probably better to say "Get metadata for all forecasts...". Same for get site observations.

get site information or metadata? probably metadata.

observation variables not up to date. Same for forecasts.

observation data example contains only 1 line. I guess it's ok but including a second line would make it more obvious that multiple data points are allowed in one upload.

The API has issue_frequency. The use cases describe this as run_length / issue frequency. run_length_issue_frequency?

Should we bring more/all of the forecast attribute descriptions into the API doc?

Interval length, not duration.

Intervals per submission is no more.

Add something like "to be determined in March/April 2019 to Reports and Trials".

fix values API documentation

Right now, the API docs say that a requests to /observations/id/values returns an array of dicts. The actual behavior returns just the dict as it should. We should update the docs and check them for the other values endpoints

tag api for initial review

0.0.1a?

Decide on Error Handling

We'll want to define how errors are reported in responses so that responses are easily consumable (e.g. json returned to the dashboard for display). Probably worth implementing some custom exceptions and error handlers for Flask as shown here: http://flask.pocoo.org/docs/1.0/patterns/apierrors/

implement data model

consider flask-seasurf for crsf protection

draft of /observations blueprint

including response schemas, request schemas, methods, routes

fix test coverage

this coveralls.io.yml line needs to be changed or the test directory needs to be renamed to tests: https://github.com/SolarArbiter/solarforecastarbiter-api/blob/master/.codecov.yml#L27

I don't care which way we go. We're using tests in the core repo.

determine rbac/authorization

String validation

We should validate things like site name better to avoid storing dangerous SQL injection or xss strings in the DB. Only allow letters digits and a few punctuation marks

add poa_global to allowed variables

use connection pooling for the database

See https://stackoverflow.com/questions/47711689/error-while-using-pymysql-in-flask/48059894#48059894. Right now a connection is created for each request

Set Proper API responses

All endpoints should respond with:

• a 200 code when successful
• 401: unauthorized when a user is not logged in (this may need to be merged with 404 depending on if we require signup for public data.
• 400 when a logged in user has permissions to perform an action but the request is invalid for some other reason. This usually produces a meaningful error message.
• 404 User lacks permissions or the resource does not exist. This is so that users cannot discover resources they do not have access to.

validate modelingparameters schema

Should check what kind of tracking type is set and validate that the required fields for that tracking type are present

draft of /user blueprint

including response schemas, request schemas, methods, routes

add probabilistic forecast specs

As documented here SolarArbiter/SolarArbiter.github.io#61

Forecast Blueprint

Implement /forecasts endpoints in forecasts blueprint. Schema should also be updated with proper validation.

notes for validation:
will need to verify this, duration, and issue freq later
and issue_time. probably easiest to use pandas timedeltas for the former three, and a time function for issue_time

Originally posted by @alorenzo175 in #28

Working Test API from static content in demo module.

• create demo objects to act as static data store.
• Make storage module to load and interact with static data store.
• Update dashboard to call API for data.

issues discovered while working on dashboard:

• Observations lack interval_length field
• site_id stored as UUID when parsed by marshmallow. cast to string for storage.
• parse and store time duration variables in minutes
• don't allow duplicate timestamps in uploads.

Create users on signup

Add users to the database after signup. Perhaps this is a manual process, or there is a flag the admins have to select after the process described in #81

Delete values

Allow users to delete values at specific timestamps/ a list of timestamps

Update to use Marshmallow 3.0.0 on release

Implement permissions administration

For granting/revoking permissions. i.e. Read/write permissions on resources. This will have two aspects: Api endpoints and MYSQL triggers.

ensure users are validated after signup

Should make sure users have validated email and that the organization has accepted the TOU before allowing access to login. Users should already require organization administrator approval since they will not have any default organization permissions.

decide how metrics/reports are generated via api

Do they each have a blueprint? Does one post the uuids of things that need to be compared?