lyft/discovery

This service provides a REST interface for querying for the list of hosts that belong to all microservices.

⚠️ This repository has been archived and is no longer accepting contributions ⚠️

Discovery

This service provides a REST interface for querying for the list of hosts that belong to a given service in microservice infrastructure. Host information is written to and read from backend store (DynamoDB by default). This project relies on the following libraries:

• Flask (web-framework)
• Flask-Cache (for caching and reusing results for GET requests)
• Pynamodb (for reading/writing DynamoDB data)

Also check requirements.txt for other dependencies.

Discovery service settings

Discovery service settings are controlled by environment variables.

• HOST_TTL
  • If the last heartbeat was not performed in the last HOST_TTL seconds, discovery service will remove host from backend storage. Default value is 600 (10 minutes).
• CACHE_TTL
  • Flask cache expiration in seconds, discovery calls BACKEND_STORAGE to fill the cache. This cache is used for hosts retrieval by service or service repo. Default value is 30 seconds.
• BACKEND_STORAGE
  • Type of the backend storage used in discovery service. Supported values are: DynamoDB, InMemory, InFile. By default DynamoDB backend is used.
• CACHE_TYPE
  • Supported values 'simple' or 'null'. Default value is 'null' which effectively turn flask caching off.
• APPLICATION_DIR
  • Application directory.
• APPLICATION_ENV
  • Environment discovery service runs in, e.g., development or production. Default value is development.
• DEBUG
  • If debug mode is used. Default value is true.
• LOG_LEVEL
  • Set logging level. Default value is DEBUG.
• PORT
  • Port flask app using. Default value is 8080.
• DYNAMODB_TABLE_HOSTS
  • Used only in case of DynamoDB backend.
• DYNAMODB_URL
  • Used only for development in case of DynamoDB backend running locally.
• DYNAMODB_CREATE_TABLES_IN_APP
  • Used for creating DynamoDB table, useful only in case DynamoDB backend storage used.

API

GET /v1/registration/:service

Returns metadata for the given :service.

• service
  • (required, string) name of the service metadata is queried for.

On successful response, response body will be in the following JSON format:

{
    "env": "...",
    "hosts": [],
    "service": "..."
}

• env
  • (required, string) environment discovery service runs in, e.g., development or production.
• hosts

  • (required, object) list of non expired hosts (hosts that last checked in to discovery service in the last HOST_TTL period). Each host is in the following JSON format:

    {
        "ip_address": "...",
        "last_check_in": "...",
        "port": 9211,
        "revision": "...",
        "service": "...",
        "service_repo_name": "...",
        "tags": {}
    }

  • ip_address

    • (required, string) ip address of the host.

  • last_check_in

    • (required, string) heartbeat timestamp converted to string, last time host registered with discovery service.

  • port

    • (required, integer) port on which the host expects connections, Envoy will connect to this port.

  • revision

    • (required, string) service SHA running on the host.

  • service

    • (required, string) service name.

  • service_repo_name

    • (required, string) service repo, used for selecting hosts based on the service_repo (can be empty if not set).

  • tags

    • (required, object) see tags here.
• service
  • (required, string) service name.

GET /v1/registration/repo/:service_repo_name

Returns list of non expired hosts for :service_repo_name (query based on secondary index, for example, DynamoDB GSI). Format is the same as query based on service.

POST /v1/registration/:service

Registers a host with a service. Response body does not contain any data.

• service
  • (required, string) Service for which operation is performed.

Request params:

• ip
  • (required, string) ip address of the host.
• service_repo_name
  • (optional, string) service repository name, can be used for quick search.
• port
  • (required, integer) port on which the host expects connections, Envoy will connect to this port.
• revision
  • (required, string) SHA of the revision the service is currently running.
• tags
  • (required, object) JSON in the following format.

DELETE /v1/registration/:service/:ip_address

Deletes the host for the given service with ip_address. Returns response code 400 if no service/ip_address entity exists.

• service
  • (required, string) name of the service metadata is queried for.
• ip
  • (required, string) ip address of the host.

POST /v1/loadbalancing/:service/:ip_address

Updates the weight of hosts for load balancing purposes.

• service
  • (required, string) Service name for which weight is updated.
• ip_address
  • (optional, string) IP address of the host for which weight is updated. If not given, all hosts for the given service will have their weights updated.

Request params:

• load_balancing_weight
  • (required, integer) Host weight, an integer between 1 and 100.

Tags JSON

  {
     "az": "...",
     "region": "...",
     "instance_id": "...",
     "canary": false,
     "load_balancing_weight": 1
  }

• az
  • (required, string) AWS availability zone that the host is running in. You can provide arbitrary but the same value for all hosts if zone aware stats/routing is not required. If you use non AWS backend storage currently you have to provide az anyway, you can use some fixed predefined value for all hosts.
• region
  • (required, string) AWS region that the host is running in.
• instance_id
  • (required, string) AWS instance_id of the host.
• canary
  • (optional, boolean) Set to true if host is a canary instance, used by Envoy.
• load_balancing_weight:
  • (optional, integer) Load balancing weight is used by Envoy for weighted routing. Values must be an integer between 1 and 100.

Main Classes

• app/routes/api.py
• defines the HTTP routes for service registration
• app/resources/api.py
• the entry point for service registration GETs and POSTs
• app/services/host.py
• contains the business logic (e.g. caching, input validation, etc) for service registration GETS and POSTs
• app/models/host.py
• The pynamo model for service registration information for a host

Unit Testing

Note currently it's not working on public repository without tweaking (there is an opened issue for this) To run all unit tests, run make test_unit.

discovery's Issues

flake8 should be part of the travis CI that is run, to ensure conformance

you can get around presubmit hooks, this will make it harder for bad commits to sneak in. once travis is up, this shouldn't be hard to add

Blocked by #7

flake8 should be a git pre-submit hook

add gevent to requirements.txt

wsgi.py and manage.py require gevent, so it should be added to requirements.txt. I'd add this, but i'm not sure if you want to pin this to a particular version.

Run discovery with any backend database

I propose the implementation set backends dynamically, as opposed to statically. My use case involves using Apache HBase as the backend data store. With a small change to the BackendSelector class in app/resources/api.py (below), users may bring their own backend. I can support this with a working implementation. What are the maintainers' levels of openness towards this feature?

`class BackendSelector(object):

@staticmethod
def select():
    """
    Select backend storage based on the global settings.
    """

    storage = settings.value.BACKEND_STORAGE

    if storage == 'DynamoDB':
        return query.DynamoQueryBackend()
    elif storage == 'InMemory':
        return query.MemoryQueryBackend()
    elif storage == 'InFile':
        return query.LocalFileQueryBackend()
    elif storage:
        # import the query backend starting from the plugins folder 
        query_location_from_plugins = 'plugins.' + storage + '.app.services.query'
        backend_name = storage + 'QueryBackend'

        try:
            query_module = importlib.import_module(query_location_from_plugins)
        except ImportError:
            raise ImportError("Verify {} is a valid path".format(query_location_from_plugins))

        try:
            query_backend = getattr(query_module, backend_name)()
        except AttributeError:
            raise AttributeError("Verify {} has classname {}".format(query_module, backend_name))

        return query_backend

    else:
        raise ValueError('Unknown backend storage type specified: {}'.format(storage))`

All tests should be runnable locally

With the exception of tests which depend on AWS (or GCP, whatever external thing)

Travis CI should run tests on every commit etc

This is pretty standard practice for open source projects

Blocked by #6

refactor configuration

I'm thinking specifically of the code in

https://github.com/lyft/discovery/blob/master/app/settings.py

This gets back to the modularity mentioned in #11

In a world where people have different backends, we want a clean way to let the required configuration depend on the pieces that they use.

My sense of one possible way to do this is to have a configuration for the backend, and then let the backends register the configuration that they need, and they'll be responsible for pulling it out.

That said, this file is also a bit of a mess and can be made much cleaner ex any of the above.

AWS Lambda hosting of this service

It would be very convenient to. deploy this service as a Lambda rather than as a separate application.

add makefile

The documentation mentions a makefile for make test_unit, but there is no makefile

GCP as a backend

Shouldn't be hard once #4 is in, though there is the longer-term question of testing. I guess ideally the model would be inverted a bit...there'd be the core library, and then you could couple it with the backend of your choice. Those backends would be responsible for testing. Still, I don't know how the python ecosystem deals with that.

This isn't a GCP specific issue... if this is possible, I imagine people will want other backends (especially since the needed functionality is so simple...it's easy to imagine a redis backend, etc etc)

how to use

is there some instruction about how to use?

Allow registering service/ip/port in discovery service

Currently pair of service name and ip needs to be unique, discovery service will overwrite settings when service/ip is non unique.

For example, if you register

service_name = service_1 ip = 127.0.0.1 port = 80 ... other tags, etc
service_name = service_1 ip = 127.0.0.1 port = 8080 ... other tags, etc

with discovery service, there only will be the latest entry saved (first one overwritten).

Delete logic relies on service/ip pair uniqueness as well.

Make statsd optional

It should be possible to run this without the statsd logging (there may be some environment magic to do this? otherwise we can use dependency injection, but it seems like the sort of library that they'd have a hook to turn off)

Doco on how to run this thing?

Hi,

I never used flask. Can we even have just a one liner some where to say how to run this? Flask example says

$ pip install Flask
$ FLASK_APP=hello.py flask run
 * Running on http://localhost:5000/

And I can't find where's the FLASK_APP either.

Thanks. 😆

Add more documentation

• Document request/response format for discovery service
• Document how to run discovery service