mithraiclabs/psyoptions-market-tracker

A server for listening to PsyOptions and Serum, indexing data, and an API to consume the longer term data

How to run locally

1. install direnv (macOS: brew, ubuntu: apt) + hook into shell

2. run direnv allow && docker-compose up --build. it will probably take a few mins of docker doing its thing before everything is ready

3. annoying step that won't be necessary soon... copy the contents of mutation.gql, then cd hasura && hasura console. Once the browser console opens, paste the generated mutation in the graphql editor and run it, it should create all the market and associated asset records.

4. open http://localhost

5. do a buy or sell trade on devnet BTC-USDC June 2021 options

6. hopefully it'll show up in the table on http://localhost within a few seconds

Architecture

Deployment

Edit .envrc and run docker-compose up --build -d

psyoptions-market-tracker's Issues

Add autoscaling to Hasura

This may (or may not) be an indication that the single Hasura task is hitting the soft 1024 file descriptor limit docs

Incorporate market meta package for active PsyOption markets

An adversary could create multiple Seurm markets for a single PsyOption market. To avoid confusion we need to use the market-meta package to determine active or inactive PsyOption markets

Update dashboard to match new markets

decimals sizing and underlying/quote asset swapped around

Remove Serum Vial dependencies

Serum-Vial is overkill for indexing and tracking Serum events and adds unnecessary networking/timing implications.

We should roll our own code to listen to events/updates on necessary Serum market accounts.

• EventQueue

OpenOrders look up fails on first trade

The first time an account places an order the OpenOrders.load call fails with Error: Open orders account not found. The second time a trade is made on that market, the open_order_account gets stored properly

Make configurable for other projects

Currently there is a fair bit of PsyOptions specific logic intertwined into the Listener and Hasura (e.g. the migrations and table set up) containers.

There are many other projects that would likely use this cloud infrastructure to track data for their projects, so we'll need to make those application specific requirements more configurable and flexible.

Improve Timescale instance for "Too many clients already"

The following error was showing up in some Hasura logs

{"type":"http-log","timestamp":"2021-09-09T01:34:32.766+0000","level":"error","detail":{"operation":{"user_vars":{"x-hasura-role":"admin"},"error":{"internal":"FATAL:  sorry, too many clients already
","path":"$","error":"connection error","code":"postgres-error"},"request_id":"d8de1115-fa19-41ff-b5c9-149a78b9b095","response_size":118,"query":{"variables":{"object":{"order_id":"46891623435369680254532","client_id":"9254101784418634862","serum_market_address":"qDvaLApMYkbp9NTgYJaNy7CDsPhXMbDjEr3g3oqEMaw","account":"Abe39jLaH5BSmLryRbAGu1wMfCgPkXnvkQV7JZDuvDxT","slot":95649683,"account_slot":2,"type":"done","side":"sell","timestamp":"2021-09-09T01:33:51.437Z"}},"query":"
      mutation ($object: serum_events_insert_input!) {
        insert_serum_events_one(object: $object) {
            serum_market_address
        }
      }
      
        "}},"http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/graphql","ip":"3.70.45.116","method":"POST","content_encoding":null}}}

This seems to coincide with when a new Hasura task is being spun up and the old one is being shut down (see image below for hasura timing).

And the following for Timescale's last error logs.

This makes sense as the Hasura connection pool would surpass the postgres max_connections if two were running at the same time and they used the Timescale tuning to determine max connections.

Database username, password, etc security

Currently there is nothing secure about the username, password, and database name in the CloudFormation templates. We'll need to store this information in AWS secrets manager and reference the secrets from the respective resources.

Remove hardcoded parameters

Some of the PsyOptions specific code is hardcoded for ease of development. Many of those hardcoded variables should be environment specific.

OpenOrder accounts aren't tracking properly

Calculate Implied Volatility PsyOptions markets

Implied Volatility definition

Subscribe to all Serum markets that belong to PsyOptions option markets

Serum DEX program id and PsyOptions program id on devnet

DEVNET_DEX_PROGRAM_ID=DESVgJVGajEgKGXhb6XmqDHGz3VjdgP7rEVESBgxmroY
DEVNET_PROGRAM_ID=7UkUQY1gqYyumhyoDy73D5JoThqGY9SJw21k6QveesfP

Meta data on May 28, BTC - USDC devnet PsyOptions markets

devnet_beta_market_data.json.zip

Track 24 hour change on Serum markets

Add Authorization to Hasura

Currently the hasura GraphQL endpoint is wide open for anyone to run an insert, update, etc. We should probably put in some sort of authorization (maybe a simple token based authorization) that is only known the to the Listener or other private services that need to actually write to the database.