lambdaclass/reconnections

We were tired of implementing a reconnection system for our database clients so we created this

reconnections

Reconnections is an Erlang/OTP application that provides a uniform interface for managing connections to external services such as databases.

Rationale

Service drivers tend to implement different connection logic: some don't handle reconnections at all, some reconnect but crash if the service isn't available on startup, etc. Covering all possible scenarios usually requires implementing ad hoc logic on every project. This library attempts to abstract driver specifics and allow to setup reconnection strategies via configuration.

Some of the design goals are:

• Provide a uniform API across different libraries.
• Provide out of the box drivers for the most commonly used Erlang libraries (e.g. epgsql, eredis, etc.).
• Make it easy to add new drivers.
• Start connection attempts when the application starts, without making them a hard depdendency, in other words: don't crash if a connection can't be established on application startup.
• Provide or simplify the implementation of circuit breakers to avoid requesting disconnected services.

The aim is to help implementing systems as those described in Stacking Theory for Systems Design by Jesper L. Andersen and It's About the Guarantees by Fred Hebert.

Development status

This is still in an experimental stage; we are currently studying the behavior of different libraries so we can later flesh out the proper API to manage the connections.

Build

$ rebar3 compile

Notes on library error behaviors

Behaviour of each service when there are no connection.

eRedis

• On start: If the redis server is not available, eredis fails and doesn't try to reconnect.
• Disconnection: If there is a disconnection after the connection was correctly established, eredis keeps on trying to reconnect to the server.

The sleep time between attempts to reconnect can be set with the fifth paremeter ReconnectSleep in the eredis:start_link/5 call.

ePgsql

• On start: If the postgresql server is not available, epgsql fails and doesn't try to reconnect.
• Disconnection: If there is a disconnection after the connection was correctly established, epgsql sends an exit signal to the process that started the connection.

CQerl

• On start: If the cassandra server is not available, cqerl fails and doesn't try to reconnect.
• Disconnection: If there is a disconnection after the connection was correctly established, cqerl fails and doesn't try to reconnect, just log the error. Doesn't send an exit signal neither.

reconnections's Issues

Init has to receive a list of services to start

Remove dependencies from app.src and add them in rebar3 profile dev

Add docker command

docker run --rm -it -v (pwd):/reconnections -v ~/.cache:/root/.cache -w /reconnections erlang rebar3 as test shell

Add test for each driver

The tests will work against services running in docker.

• There needs to be a way (perhaps calling os:cmd) to interface with the running docker to tell it to kill a service.
• There needs to be a way to ask docker or the os whether a service is running
• There needs to be some process randomly killing (and resurrecting) services during the test run
• The test should constantly try to use the service.
• when the service call fails check that both the OS and the driver report the service as being disconnected
• after a disconnection, test that the driver attempts reconnection according to the configured backoff, that reconnection succeeds and that service calls work again.

Remove ASCii separators in modules

%% priv

Generate a test-name with Uid

separate server from driver specific logic

The servers always do the same, driver should only encapsulate whatever changes from driver to driver.

Consider making driver a behavior.

Add concuerror and basho bench

Check fuse_server from jlouis to see if we should use state or an ets to circuit break

send parameters to connect service through init/1

don't use applications:get_env/2 in gen_server

use maps instead of tuples in supervisor

Add tests for start the app with the services down

Rename Server folder to Drivers

General styles

• Enter before same functions with different parameters
• No ASCII separators
• Rename server/ to drivers/
• Use Maps instead of touples in supervisor

List of services to support

• epgsql
• eredis
• cqerl
• riak erlang client
• emysql
• rabbitMq
• brod
• MongoDB Erlang
• neo4j
• sql lite
• Amazon DynamoDB
• CouchDB

Check if it is a better option to use a statem

validate sys.config, before to start app