adsabs/adsws

ADS web services

ADSWS - ADS Web Services

About

Core API module for the NASA-ADS, handling:

• authentication
• passing requests to the correct local and/or remote microservices
• rate limiting

Installation

    git clone https://github.com/adsabs/adsws.git
    pip install -r requirements.txt 
    alembic upgrade head
    vim instance/local_config.py # edit edit edit...
    python wsgi.py

Testing

    pip install -r dev-requirements.txt
    py.test adsws

adsws's Issues

@require_oauth hangs in some cases

Steps to reproduce:

• Serve adsws and vis-services
• Set scopes on /word-cloud to ['api:search','api:tvrh']
• request <adsws>/services/vis/word-cloud?q=star; response will timeout

Slackback feedback web end point

Do it.

Tests for discoverer

accounts: more robust parsing of POST data

Currenty the get_post_data method breaks if content-type is not set to application/json; this should be fixed such that form encoded data should work without specifying content-type

use @require_oauth instead of @login_required for /accounts ?

Consider the implications of both authentication paradigms.

travis reports 'build passing' but it is actually failed

that applies to the latest master - i'm seeing the same problems that you saw on travis:

======================================================================================= test session starts ========================================================================================
platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
plugins: cache, pep8, cov
collected 39 items

adsws/modules/classic/testsuite/test_classic_user.py ...xx.
adsws/modules/oauth2server/testsuite/test_provider.py ..........
adsws/tests/test_api.py FF
adsws/tests/test_api_bumblebee.py ..
adsws/tests/test_api_limits.py .
adsws/tests/test_api_solr.py ....
adsws/tests/test_core_clients.py .
adsws/tests/test_factory.py ..
adsws/tests/test_frontend_authentication.py .....
adsws/tests/test_frontend_authentication_classic_fallback.py ..xx..

============================================================================================= FAILURES =============================================================================================
_____________________________________________________________________________________ ApiTestCase.test_headers _____________________________________________________________________________________

self = <adsws.tests.test_api.ApiTestCase testMethod=test_headers>

def test_headers(self):
    current_app.config['CORS_DOMAINS'] = {'http://localhost': 1}

    r = self.client.get('/test', headers=[('Origin', 'http://localhost')])

  self.assertEqual(r.headers.get('Access-Control-Allow-Origin'), 'http://localhost')

E AssertionError: None != 'http://localhost'

adsws/tests/test_api.py:59: AssertionError
--------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------
ERROR:adsws.api:
Request: GET /test
IP: None
Agent: None | None None
Raw Agent:
Oauth2: None

Traceback (most recent call last):
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1470, in full_dispatch_request
self.try_trigger_before_first_request_functions()
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1497, in try_trigger_before_first_request_functions
func()
File "/dvt/workspace/adsws/adsws/modules/oauth2server/views/server.py", line 52, in setup_app
bind_cache_grant(current_app, oauth2, OAuthUserProxy.get_current_user)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/oauth2.py", line 91, in bind_cache_grant
cache = Cache(app, config_prefix)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/cache.py", line 18, in init
self.cache = getattr(self, cache_type)(**kwargs)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/cache.py", line 76, in _redis
return RedisCache(**kwargs)
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/werkzeug/contrib/cache.py", line 486, in init
raise RuntimeError('no redis module found')
RuntimeError: no redis module found
_____________________________________________________________________________________ ApiTestCase.test_options _____________________________________________________________________________________

self = <adsws.tests.test_api.ApiTestCase testMethod=test_options>

def test_options(self):
    r = self.client.options('/gp', headers=[
                    ('Origin', 'http://localhost'),
                    ('Access-Control-Request-Headers', 'accept, x-bb-api-client-version, content-type')])

  self.assertEqual(r.headers.get('Access-Control-Allow-Methods'), 'GET, OPTIONS, POST')

E AssertionError: None != 'GET, OPTIONS, POST'

adsws/tests/test_api.py:69: AssertionError
--------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------
ERROR:adsws.api:
Request: OPTIONS /gp
IP: None
Agent: None | None None
Raw Agent:
Oauth2: None

Traceback (most recent call last):
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1470, in full_dispatch_request
self.try_trigger_before_first_request_functions()
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/flask/app.py", line 1497, in try_trigger_before_first_request_functions
func()
File "/dvt/workspace/adsws/adsws/modules/oauth2server/views/server.py", line 52, in setup_app
bind_cache_grant(current_app, oauth2, OAuthUserProxy.get_current_user)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/oauth2.py", line 91, in bind_cache_grant
cache = Cache(app, config_prefix)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/cache.py", line 18, in init
self.cache = getattr(self, cache_type)(**kwargs)
File "/dvt/workspace/adsws/python/src/flask-oauthlib/flask_oauthlib/contrib/cache.py", line 76, in _redis
return RedisCache(**kwargs)
File "/dvt/workspace/adsws/python/local/lib/python2.7/site-packages/werkzeug/contrib/cache.py", line 486, in init
raise RuntimeError('no redis module found')
RuntimeError: no redis module found

GET /login -> POST /login

More restful, and automatically csrf protected

change email workflow should not allow a user to lock themselves out of the account

In the current implementation, a user can change their email to one to which they do not have access, thus locking themselves out of their account.

The most straightforward fix is to not perform the user.update(email=new_email) procedure until the new email has been verified. In this way, the change email workflow is a no-op until they visit the new email's verification link

/v1/feedback

seems to be a part of the api, but it is not governed/protected by the same rules - in fact, it is a simple application exposed under /v1/feedback. It doesn't have oauth scopes which means ADS team can be freely spammed by anyone who finds the endpoit. Am I reading it correctly?

Check validity of `oauth_token` during `bootstrap_bumblebee()`

Currently db.session.query(Token).first() is performed for users that come with a clientid in their session. If this token is expired, the only way the user can access api functionality is to remove their session to get a new, valid token.

End point for inviting users to join if they do not have an account

Currently, if you try to add a user to a library, it will just return an error. I think it would be great to send them an invite.

The more I think about it, maybe it is better to just have a generic invite end point, which does not have to be related to that of the biblib-service directly.

This directly affects my approach in adsabs/biblib-service/issues/79.

Implement @limit_rate

Should we allow requests with non-url encoded strings?

Noticed this in one of the logs, which is possible to reproduce if one passes in a ":" to directly to the api.

ValueError: Error trying to decode a non urlencoded string. Found invalid characters: set([u':']) in the string: 'q=bibcode:2015arXiv150301104A&fl=title'. Please ensure the request/response body is x-www-form-urlencoded

References adsabs/bumblebee#123

Add e-mail integration to the feedback service

There is still the use case for people wanting to have e-mails received , so a relevant end point can be added in the views.

See the accounts utils https://github.com/adsabs/adsws/blob/master/adsws/accounts/utils.py#L25.

This is semi-related to issue #70.

Implement ProxyView.POST

third-party services that are API clients should forward user's token

References adsabs/vis-services#5

development: hard dependency on redis

It's not ideal that we have a hard dependency on redis. I saw recently that they fleshed out the backend implementations in flask-ratelimiter; check to see if flaskcache/noop or something similar is available

better configuration

the deployment needs to touch alembic.ini and instance/local_config.py

this is sloppy -- also, i should use configobj to parse the config, to give some flexibility

Implement consul WEBSERVICES discovery

Webservices should be able to be defined by something like consul://name_of_service, which instructs the discoverer to query its local consul DNS for that service. Bonus points for re-trying failed connections.

JSON everything, including errors!

In the case of oauth2 401 responses, this will require overwriting/patching the abort() method in flask, I'm afraid.

http://flask.pocoo.org/snippets/97/ does exactly this, but optionally returns the html formatted error based on the http accept header.

Paths still not configured properly (Database)

Provision works after changing SQLITE urls in alembic.ini and instance/local_config.py in the sense that the container is built without problems. Also, running python wsgi.py does not produce any errors. However, curl localhost:6002 produces the following:

root@8582527807ea:/adsws# curl localhost:6002
INFO:werkzeug:127.0.0.1 - - [11/Sep/2014 13:09:49] "GET / HTTP/1.1" 500 -
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>
root@8582527807ea:/adsws# ERROR:werkzeug:Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 177, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 165, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/wsgi.py", line 648, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/adsws/adsws/middleware.py", line 58, in __call__
    return self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1479, in full_dispatch_request
    response = self.process_response(response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1693, in process_response
    self.save_session(ctx.session, response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 837, in save_session
    return self.session_interface.save_session(self, session, response)
  File "/adsws/adsws/ext/session/interface.py", line 150, in save_session
    self.backend.set(sid,
  File "/usr/local/lib/python2.7/dist-packages/flask/helpers.py", line 724, in __get__
    value = self.func(obj)
  File "/adsws/adsws/ext/session/interface.py", line 82, in backend
    if isinstance(storage_string, six.string_types) \
  File "/adsws/adsws/ext/session/backends/sqlalchemy.py", line 51, in __init__
    self.model.__tablename__):
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", line 791, in has_table
    cursor = _pragma_cursor(connection.execute(statement))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1751, in execute
    connection = self.contextual_connect(close_with_result=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1799, in contextual_connect
    self.pool.connect(),
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 641, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1055, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 285, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 411, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 537, in __connect
    connection = self.__pool._creator()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 96, in connect
    connection_invalidated=invalidated
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 90, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 377, in connect
    return self.dbapi.connect(*cargs, **cparams)
OperationalError: (OperationalError) unable to open database file None None

ProxyView: consider passing raw response instead of resp.json()

We are undoubtedly serializing and de-serializing the json responses multiple times (unnecessarily) when passed through the ProxyView.

verification email should recieve `verify_url` in POST, send to that URL

rate-limiter does not get the correct ip in production

Rate limiter needs to check for the existence ofX-forwarded-for, and remote services should pass user client_id and not their own.

(improve) pass uid in header to backend services

E-mail from UID end point

The library microservice will need to present an owner of the library. One option for now is the start of the e-mail address. Given the same limitations to the library database as in issue #58, it would be good to have an end point to retrieve the e-mail address of the user when supplying the accounts user UID.

Sound fine to you? Any better solutions?

For more details see, adsabs/biblib-service/issues/46

Implement a global `/resources` route

flask-discoverer should work out of the box for this, but the question of hiding routes (like the oauth2 stuff, see #28) will still be open

Improve ConsulService.resolve()

The current implementation that heavily relies on dnspython is opaque and exceedingly difficult to test; This should be improved.

cache: set headers on a per-service basis

tests: refactor

Make a central, clearly defined place for tasks related to csrf handling, login, and header parsing in the test context. Currently that functionality is spread over several files. Much of it is redundant or not even used.

Incorporate discoverer in api

Discoverer augments the API with whichever standalone microserves it found. From a code-organization point of view, it makes sense to have that logic be part of the API and not a standalone app.

session stickiness

The request (if possible) should be routed to the same instance, at least for SOLR - that means that the api needs to pass appropriate headers, andthe load balancer at AWS can decide based on that. BTW: from what I see, the sessions are not sticky for AWS (which is good, IMO), but I didn't check solr

better csrf passing

The current implementation passes a csrf token with the token data structure at /bootstrap. This works, but has the side-effect of potentially causing the user to send an expired csrf token (3600s default lifetime) by the time they get to account maintenance activities.

Perhaps a better solution would to use a dedicated endpoint to csrf. That endpoint would be visited intelligently by bumblebee: On first order, every time the user initiates some user account related action. This endpoint would be cheaper than the full access token endpoint.

bootstrap: incorrect ratelimiting

requests that have a client id encoded in their session cookie should not count towards their ratelimits

implement smart-config via flask-consulate

oauth2: touch `client.last_activity_at` inside `@require.oauth`

references #53

UID from e-mail address endpoint

ADS Libraries would need an endpoint that returns the API user ID for a given user from their e-mail address.

ProxyView response codes not correctly passed

Separate CORS logic from @route

These two should be separated, flask-cors provides the CORS functionality.

rate_limit on bootstrapped function-based views

if view.view_class.rate_limit:
    AttributeError: 'function' object has no attribute 'view_class'

does not work if we're bootstrapping function based views

Including extensions should not set routes

Routes like

/v1/services/oauth/authorize
/v1/services/oauth/invalid/
/v1/services/oauth/errors/

should not be created, period.

need new version of solr

i'm not sure which version is right now deployed, but it seems that /qtree is returning xml - instead of json - i checked it in master-next and there it is ok. so we'll need new deployemnt (but the changed don't require reindexing)

tests: use mocking or testing.redis instead of the redis defined in app.config

accounts: last_login_at is not modified when authenticated via session cookie

At the moment, this value is only updated on explicit authentication to /login

Internal server error - it is likely a wrong request - so the error is wrong

$ curl -H 'Authorization: Bearer:xxxxx
UqrovddSZOzjmJ' 'http://api.adslabs.org/v1/search?q=bibcode:2011ApSSP...1..125H
&indent=true'
<html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

let browser cache the response

the api should be returning headers that allow browser to cache response

a good list can be found here, under the heading:

make ajax cacheable

microservices are being generated default tokens when accessed via bumblebee

In the database, client with name='vis' gets an access_token with scope api:seach at bumblebee load.

accounts: verify `email1==email2` in change email workflow

personal token dosn't grant access to api:search

and in general, one cannot specify a scope(s)

accounts: add ability to change email address