dfelton/kobens-exchange

Framework for facilitating trades on an exchange.

kobens-exchange's Issues

Database resilience

The current interactions with the database are a current area of brittleness for the application. Abrupt failure will occur under any of the following scenarios, and the application will not self-recover:

• Database server is offline during initial connect
• Database server goes away after initial connect and before the process ends (which, we're aiming for long running processes that never end)
• Data for the table has changed in an unexpected way
  • While this scenario will not cause the application to abruptly end, it would probably result in the application proceeding in unknown ways.

Goal here for resolution of this issue is to at least begin to introduce some resilience into the application in the event of some of these scenarios. Lets try to:

• Have connect operation enter a sleep and retry loop in event of database server not being available.
• Connect logic should be abstracted in such a way that any area of the application can call it should an exception be caught after initial connect, that indicates the database has gone away.
• Have current database calls written in such a way that retries will occur if database connectivity is lost (first call reconnect method, then, once connected, retry initial operation).
• Logging should exist surrounding failures.

Monitor loop gets stuck.

Somewhere within the contents of this do {} while(); loop the application has had a history of getting stuck.

Exact steps to reproduce is currently unknown but the following observations have occurred surrounding it:

• The process will still be running, it has not ended abnormally.
• During timeframe of the issue, CPU usage will be essentially 0%. Whereas a healthy app shows CPU usage bouncing around between 0% and 100% on a regular basis.
  • This has been observed using the kobens-monitor app.
• Killing the application and restarting it, will resume operations as intended. Often times it has been observed that there are updates the monitor should be taking action on, and updates not occur until the process is killed and restarted.

Other notes:

• Exact issue could potentially lie in kobens-gemini if the issue is surrounding one of the curl calls to the Gemini exchange.
• Don't have hard data around this but I believe the problem occurs during times of finicky internet reliability.