Getting up and running

[fremantle-industries]

Getting up and running

[chrism2671]

Yes of course! Thanks for all your help! :)

[chrism2671]

That seemed to fix it!

[rupurt]

Howdy @chrism2671,

Can you post a copy of your config?

The dummy keys are just to get the tests passing on CI, you should not need to do anything with them. If the venue you want to stream data allows public access without API credentials you can just set an empty map for accounts when configuring the venue.

I believe OkEx is the only venue out of the adapters I currently have that requires API credentials for streaming data.

[chrism2671]

So I'm just using the example config with no modifications:
https://github.com/fremantle-capital/tai/blob/master/config/dev.exs.example

[chrism2671]

I have a feeling the issue is coming from Confex (a module I'm not familiar with).

I tried replacing the :ex_poloniex config section with absolute paths:

config :ex_poloniex,
  api_key: {:system_file, "/home/chris/code/elixir/tai/secrets/poloniex_api_key"},
  api_secret: {:system_file, "/home/chris/code/elixir/tai/secrets/poloniex_api_secret"}

but that didn't seem to make any different.

I'm running:

MIX_ENV=dev elixir --sname tai -S mix run --no-halt

[chrism2671]

(it does work if I strip out the confex adapter and just stick in the API keys as strings.)

[rupurt]

@chrism2671 have you set the environment variables for POLONIEX_API_KEY & POLONIEX_API_SECRET?

e.g.

export POLONIEX_API_KEY="/home/chris/code/elixir/tai/secrets/poloniex_api_key"
export POLONIEX_API_SECRET="/home/chris/code/elixir/tai/secrets/poloniex_api_secret"
MIX_ENV=dev elixir --sname tai -S mix run --no-halt

[chrism2671]

Ah, I didn't do that, I hadn't realised that's what it was looking for.

I have actually got the thing running now by just pasting in API keys directly; I don't think there's anything wrong with tai as such.

I have a couple of other questions.

I'm trying to write my first advisor. Let's say, for example, I want to do a moving average. I need to maintain that MA value somewhere. I'm trying to store it in %State{}, but without much luck. How should I store advisor state?

[chrism2671]

I think I figured it out (stick in {:ok, whatever_state_you_want}, is that right?)

[rupurt]

@chrism2671 good to hear it's running for you now. This raises a good point though, that the configuration options are not sufficiently documented. I will add some more clarity to the README.

To keep track of data while the advisor is running you can update the store within %Tai.Advisor.State{} like you mentioned by returning the tuple {:ok, new_store} e.g.

def handle_inside_quote(_venue_id, _product_symbol, _market_quote, _changes, state) do
   count = Map.get(state.store, :count, 0)
   new_store = Map.put(state.store, :count, count + 1)
   {:ok, new_store}
end

[chrism2671]

Yep, that's exactly what I wrote.

I have another question. Is it possible to see the existing positions on an exchange? I can see Tai.VenuesAdapters.Bitmex.Positions, but it's unclear to me if there's a general interface for this, and an analogue for OKEx.

Equally, for orders, is there a way to pull this from the exchange, rather than relying on tai's internal state?

[chrism2671]

One more question, is it possible to detect and handle executions?

[rupurt]

I have another question. Is it possible to see the existing positions on an exchange? I can see Tai.VenuesAdapters.Bitmex.Positions, but it's unclear to me if there's a general interface for this, and an analogue for OKEx.

@chrism2671 positions are a very early WIP. There is some starter code but it doesn't do much so I have not documented it anywhere. When I need to deal with positions I currently just track that in the advisors run_store.

Equally, for orders, is there a way to pull this from the exchange, rather than relying on tai's internal state?

There is no way for tai to explicitly pull the order state. You would have to use the wrapper packages directly.

One more question, is it possible to detect and handle executions?

Yes. You can provide an order_updated_callback.

[chrism2671]

OK great. I might try to build that.

Is there an easy way to debug the contents of an advisor? When the advisor crashes, it just quits silently and changes status back to 'unstarted', without actually displaying the error. I'm working with iex -S mix.

[rupurt]

@chrism2671 that would be super sweet!

Take a look at the video in the README for an example of the dev workflow. You can tail the log if you configure the file logger backend.

[chrism2671]

Oh that helps!

So the error is that it's looking for handle_info/2 on my advisor. I've defined it as

def handle_info(_, state), do: {:noreply, state}

but I'm not sure I should be defining it at all?

Is there a way to inspect a running advisor's state? I tried Tai.Advisors.Store.all() but that's not what I was looking for ;).

[rupurt]

@chrism2671 you will need to define handle_info like you mentioned.

To inspect the state of the advisor you can do something like:

name = Tai.Advisor.to_name(group_id, advisor_id)
pid = Process.whereis(name)
state = :sys.get_state(pid)

[chrism2671]

So consider a case where you have limit orders on one instrument, and you want to fire a market order as soon as the limit order is filled. You'd put the market orders in the order_updated of the limit order, but in this case, state is out of scope, so you wouldn't be able to update your position. Am I missing something?

I have to say, I'm really enjoying working with tai!

[chrism2671]

Just as an update to this, I tried doing:
order_updated_callback: &order_updated/3

  def order_updated(_previous_order, _updated_order, state) do
    IO.inspect("Order updated")
    IO.inspect state
    {:ok, state.store}
  end

But couldn't make this work (BadArityError). (My reasoning was looking at line 136 of advisor.ex, callback.(old_order, updated_order, state)).

[rupurt]

That's really cool to hear @chrism2671. Made my day 😄

Apologies for the delayed reply. I was out camping this weekend 🏕

If you want to execute within the context of the advisor to have access to state you can send the advisor process a message e.g.

advisor_name = Tai.Advisor.to_name(state.group_id, state.advisor_id)

%Tai.Trading.OrderSubmissions.BuyLimitGtc{
   # …
   order_updated_callback: &Tai.Advisor.cast_order_updated(advisor_name, &1, &2, callback, {:extra, :args})
}
|> Tai.Trading.Orders.create()

[chrism2671]

I'm glad!

That seemed to work. I sent it through as a closure and state was available!

Just wondering, on PriceLevel, should the price be Decimal?

I'm trying to think about how to implement market orders. I can fire them with ExOkex (which I'm already using for pulling balances), or send limit orders priced to lift orders already on the book. Do you have any suggestions? It seems like price is a required field for OrderStore.

[rupurt]

Just wondering, on PriceLevel, should the price be Decimal?

I've elected to make the conversion to Decimal as lazy as possible for performance reasons. I've taken the approach to try and just pass through the information from the venues as fast as possible in the venue adapters.

I'm trying to think about how to implement market orders. I can fire them with ExOkex (which I'm already using for pulling balances), or send limit orders priced to lift orders already on the book. Do you have any suggestions? It seems like price is a required field for OrderStore.

This is another deliberate choice. I'm avoiding market orders as much as possible due to the unreliability of venues. I don't want tai to be associated with a crazy fat finger trade if possible. tai only supports limit orders at this point so market orders would have to be simulated by chasing the quotes through the order book.

[chrism2671]

I've got things up and going now, and am testing on real orders.

I've got an issue on Bitmex live that appears to be the following when some orders are cancelled.

09:00:25.994 [info] {"data":{"account_id":"main","client_id":"e97a4ad7-975b-4eee-b8de-b409636173eb","close":null,"cumulative_qty":"0","enqueued_at":"2019-09-03T09:00:25.522402Z","error_reason
":"nil","last_received_at":"2019-09-03T09:00:25.891739Z","last_venue_timestamp":"2019-09-03T09:00:25.963Z","leaves_qty":"0","price":"0.01713","product_symbol":"ethu19","product_type":"future"
,"qty":"1","side":"sell","status":"canceled","time_in_force":"gtc","type":"limit","updated_at":"2019-09-03T09:00:25.992482Z","venue_id":"bitmex","venue_order_id":"38cf2237-604b-77e8-ea06-e867
d4394aa1"},"type":"Tai.OrderUpdated"}
09:00:26.041 [info] {"data":{"action":"passive_cancel","client_id":"e97a4ad7-975b-4eee-b8de-b409636173eb","required":["open","partially_filled","filled","expired","pending_amend","amend","ame
nd_error","pending_cancel","cancel_accepted"],"was":"canceled"},"type":"Tai.OrderUpdateInvalidStatus"}
09:00:26.164 [info] {"data":{"account_id":"main","client_id":"ba048b4e-e75c-4e95-8a66-709cb27ad3c5","close":null,"cumulative_qty":"0","enqueued_at":"2019-09-03T09:00:25.529810Z","error_reason
":"nil","last_received_at":"2019-09-03T09:00:26.161613Z","last_venue_timestamp":"2019-09-03T09:00:26.137Z","leaves_qty":"0","price":"0.01713","product_symbol":"ethu19","product_type":"future"
,"qty":"1","side":"sell","status":"canceled","time_in_force":"gtc","type":"limit","updated_at":"2019-09-03T09:00:26.162688Z","venue_id":"bitmex","venue_order_id":"e17b8701-a479-acd3-46fd-6892
486b0f04"},"type":"Tai.OrderUpdated"}
09:00:26.175 [error] Task #PID<0.597.0> started from :advisor_debug_main terminating
** (FunctionClauseError) no function clause matching in Tai.Trading.Orders.Cancel.notify_updated_order/1
    (tai) lib/tai/trading/orders/cancel.ex:84: Tai.Trading.Orders.Cancel.notify_updated_order({:cancel_error, {:error, {:invalid_status, :canceled, :pending_cancel}}})
    (elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Function: #Function<0.29610064/0 in Tai.Trading.Orders.Cancel.cancel/1>
    Args: []

I'm not 100% sure where this is coming from (it seems to be transient). Any thoughts?

[rupurt]

@chrism2671 give v0.0.35 a go. There are a bunch of passive order sequence bug fixes in there.

[rupurt]

@chrism2671 I've added an example ping/pong advisor which should provide you with a better reference to work from https://github.com/fremantle-capital/tai/tree/master/apps/examples/lib/examples/ping_pong

[chrism2671]

Thanks!

I'm just trying to merge in the latest commit. My def init() no longer seems to fire. I did change it def after_start(), but I'm just trying to follow the logic. Should I be using init()?

Before:

  def init(state) do
    IO.inspect "Init"
    store_update = 
      state.store
      |> Map.put_new(:advisor_name, Tai.Advisor.to_name(state.group_id, state.advisor_id))
    state = Map.put(state, :store, store_update)
    {:ok, state, {:continue, :subscribe_to_products}}
  end

After:

  def after_start(state) do
    IO.inspect "Init"
    store_update = 
      state.store
      |> Map.put_new(:advisor_name, Tai.Advisor.to_name(state.group_id, state.advisor_id))
    state = Map.put(state, :store, store_update)
    {:ok, state.store}
  end

[chrism2671]

Just dug a little more into this, confirmed after_state() is running, but doesn't seem to be saving the state.

[rupurt]

@chrism2671 after_start is the new hook you'll want to use. There's an update on latest master that saves the run_store 08f2f19

[chrism2671]

Got it going, and moved everything over to handle_event().

I'm just trying to figure out; is there an easy way to refresh the balance register? It doesn't seem to track the exchanges (bitmex, okex), and I'm wondering if there was a good way to do it via REST. I can see hydrate_products_and_balances(), but not a clear way to update it.

[rupurt]

@chrism2671 sweet. Sorry there has been a decent amount of churn recently in the core API...

There is no way to refresh the balances atm but it's definitely something I'd like to add. I just haven't required it so far. I would also like to refresh the products. But other priorities have been more important 😄

[chrism2671]

What's the right way to log fills? Should I be doing this with order_updated_callback/2 or handle_info/2?

[rupurt]

What's the right way to log fills? Should I be doing this with order_updated_callback/2 or handle_info/2?

order_updated_callback

You'll want to handle the :partially_filled & :filled status updates. e.g.

• https://github.com/fremantle-capital/tai/blob/master/apps/examples/lib/examples/ping_pong/manage_order_update.ex#L44
• https://github.com/fremantle-capital/tai/blob/master/apps/examples/lib/examples/ping_pong/manage_order_update.ex#L60

[chrism2671]

Awesome, that worked really nicely.

I'm getting these back from Bitmex in my dev.log on amend. The actual behaviour on the exchange seems fine.

│16:41:34.615 [warn] {"data":{"msg":{"account":14674,"clOrdID":"gtc-pmjhWzP8RIygWQcK/ari5A==","orderID":"8│
860b9f3-1c9b-487e-433b-04d58a01e0e5","price":0.01942,"symbol":"ETHZ19","timestamp":"2019-10-28T16:41:34.59│
4Z","transactTime":"2019-10-28T16:41:34.594Z"},"received_at":"2019-10-28T16:41:34.615085Z","venue_id":"bit│
mex"},"type":"Tai.StreamMessageUnhandled"}                                                                │
│16:41:36.109 [warn] {"data":{"action":"Elixir.Tai.Trading.OrderStore.Actions.PendAmend","client_id":"c77b│
a810-f7ce-4e68-8595-1727acbf8580","last_received_at":null,"last_venue_timestamp":null,"required":["open","│
partially_filled","amend_error"],"was":"pending_amend"},"type":"Tai.OrderUpdateInvalidStatus"}   

This is while amending limit orders resting on bitmex. Can I safely ignore these? The actual amends seem fine (qty=1).

[chrism2671]

I'll add it seems to get out of sync with the positions on bitmex (i.e. it misses a fill), maybe once every 12 hours (I compare the net position from the orders table with the position from REST every minute).

[chrism2671]

lnav screenshot:

[rupurt]

It looks like you're trying to amend an order that is currently getting amended. You'll need to guard the status of the order before attempting the operation e.g. https://github.com/fremantle-capital/tai/blob/master/apps/examples/lib/examples/ping_pong/manage_quote_change.ex#L25

I'll add it seems to get out of sync with the positions on bitmex (i.e. it misses a fill), maybe once every 12 hours (I compare the net position from the orders table with the position from REST every minute).

Do you think this is related to the multiple amend operations? I've created a story and will keep an eye on it.

[chrism2671]

Sorry, only just noticed this message! I added a bunch more stuff to make sure I was trying to amend an order that wasn't in an amend-able state, and that seems to have gotten rid of the error messages.

It does still get out of sync with Bitmex, this seems to happen after 12-24 hours of operation, and usually after there's been a lot of volatility, Bitmex has been :overloaded. I run a REST sanity check every minute.

As an aside, I sidecar-ed Grafana into it, which produces pretty graphs, and would definitely recommend using it if you don't already.

I have a question. I'm trying to get the Bitmex.Stream.Connection to subscribe to the margin channel on Bitmex. I can't clearly see where auth_channels get declared, and on start_link, it seems like channels = []. Can I declare this in the config file?

I know I'll need write handlers for this message type; I just want to get the data flowing in so I can figure it out.

[rupurt]

Heyo @chrism2671,

That's super dope! 😄 I also use Grafana to do something similar.

Sorry, only just noticed this message! I added a bunch more stuff to make sure I was trying to amend an order that wasn't in an amend-able state, and that seems to have gotten rid of the error messages.

Great to hear

It does still get out of sync with Bitmex, this seems to happen after 12-24 hours of operation, and usually after there's been a lot of volatility, Bitmex has been :overloaded. I run a REST sanity check every minute.

Thanks for investigating. How do you feel about creating a PR that does this check and logs an event? It would be really handy to be able to alert on it.

[rupurt]

I have a question. I'm trying to get the Bitmex.Stream.Connection to subscribe to the margin channel on Bitmex. I can't clearly see where auth_channels get declared, and on start_link, it seems like channels = []. Can I declare this in the config file?

I know I'll need write handlers for this message type; I just want to get the data flowing in so I can figure it out.

Sorry overlooked this...

You do it in the venue configuration e.g.

config :tai,
  venues: %{
    bitmex: [
      enabled: true,
      adapter: Tai.VenueAdapters.Bitmex,
      products: "xbtusd",
      quote_depth: 3,
      channels: [:margin],
      accounts: %{
        main: %{
          api_key: {:system_file, "BITMEX_API_KEY"},
          api_secret: {:system_file, "BITMEX_API_SECRET"}
        }
      }
    ]

You'll also need to add it to the list of optional channels in the BitMEX connection. https://github.com/fremantle-capital/tai/blob/master/apps/tai/lib/tai/venue_adapters/bitmex/stream/connection.ex#L84

[chrism2671]

Sorry for slow response!

Thanks for investigating. How do you feel about creating a PR that does this check and logs an event? It would be really handy to be able to alert on it.

Currently I do this in the advisor- 'tis a bit hacky & looks a bit like this:

  def rest_position(state) do
    case Tai.Venues.Config.parse_adapters[@venue_id] |> Tai.Venue.positions(:main) do
      {:ok, []} ->
        0
      {:ok, positions} -> 
        position = 
          positions
          |> Enum.filter(fn x -> x.product_symbol == state.config.bitmex_leg end)
          |> hd
          |> Map.get(:qty)
          |> Decimal.to_integer
          
        log_metric(state, "bitmex_position_rest", position)

        if state.store[:initial_position] do
          websocket_position = position_from_orders(state, :filled) + state.store.initial_position
          sanity =  websocket_position == position
          if sanity == false do
            Logger.error("Position out of sync websocket: #{websocket_position} rest: #{position}")
            # shutdown(state, "Position out of sync websocket: #{websocket_position} rest: #{position}")
          end
        end
        position
      {:error, error} ->
        Logger.error "Could not get bitmex position (using cached position) #{inspect error}"
        initial_position = if state.store[:initial_position], do: state.store.initial_position, else: 0
        position_from_orders(state, :filled) + initial_position
    end
  end

It probably makes sense to have some kind of sanity checking layer, but I'm not sure the best way to engineer this in (I'm not sure I'm a good enough coder (yet!) to contribute here!)

I do see two options.

• Do a 60 second position reconciliation via REST at the Adapter layer
• Add a method in Advisor that gets run every x seconds that allows someone to write their own sanity check. This is quite nice as it means writing sanity checks is a part of writing an Advisor, and it's probably easy to add.

I think Advisor probably also needs a on_shutdown() thing to clean up on a fail (cancel all orders, flatten all positions); that part might be quite straightforward.

[rupurt]

I like the idea of doing something like this at the adapter layer.

You can cancel all orders in the BitMEX adapter with a timer. Thanks to @yurikoval 😄

bitmex: [
      enabled: true,
      adapter: Tai.VenueAdapters.Bitmex,
      products: "xbtusd",
      quote_depth: 3,
      accounts: %{
        main: %{
          api_key: {:system_file, "BITMEX_API_KEY"},
          api_secret: {:system_file, "BITMEX_API_SECRET"}
        }
      },
      opts: %{
        autocancel: %{ping_interval_ms: 15_000, cancel_after_ms: 60_000}
      }
]

[rupurt]

@chrism2671 can I close this issue as it seems like you're up and running now 😄

I've also added a built with tai section if you're ever interested in publicly releasing something you've built.