betcode-org/flumine

flūmine - Betting trading framework

flumine's Issues

Paper Trade

Allow paper trading

• Create paper trade simulated order stream (streams.add_simulated_order_stream)
• Update client with paper_trade var
• Update simulated execution

Logging Control

Add logging control interface so that the following details can be recorded, i.e. In a csv or an API

• New market
• Data save location i.e. s3 bucket location

Memory Leak when processing multiple files

When processing multiple files memory explodes.

To be investigated.

Allow order stream to be disabled

Often a user will want to only record data therefore it should be possible to disable the order stream from starting (reduces the number of threads)

Ideally this is done on the Client object

client.enable_order_stream: bool = False

OrderPackage retry

On http error retry execution.

• Max retries?
• Delay retry?
• What if final retry still fails?

Exception during add new market

While working with 4339 market types, the following exception happens couple time for 2-3 hours of the script run

Exception in thread MarketStream_output_thread: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/flumine/streams/marketstream.py", line 47, in handle_output block=True, timeout=self.streaming_timeout File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/queue.py", line 178, in get raise Empty _queue.Empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/site-packages/flumine/streams/marketstream.py", line 51, in handle_output market_ids=self.flumine.markets.open_market_ids File "/usr/local/lib/python3.7/site-packages/flumine/markets/markets.py", line 44, in open_market_ids return [m.market_id for m in self.markets.values() if not m.closed] File "/usr/local/lib/python3.7/site-packages/flumine/markets/markets.py", line 40, in markets return {key: value for key, value in self._markets.items()} File "/usr/local/lib/python3.7/site-packages/flumine/markets/markets.py", line 40, in <dictcomp> return {key: value for key, value in self._markets.items()} RuntimeError: dictionary changed size during iteration

Threading - Catch errors

Missing errors from threads, ie. workers

Execution - keep alive

All new execution requests use a new requests.Session, this results in the https round trip.

Simulation improvements

Various improvements / edge cases in the simulation class and middleware:

• Update PIQ based on 'potential' cancellations

• Prevent double counting of liquidity i.e. if you take £x sitting at y should x still be available on the next update (separate middleware? flag?)

• Validate market is open when placing orders (error if not)

• Validate runner is active

• Validate marketVersion when sent in place/replaceRequest

• Validate market is open for SP and not in play when placing market on close orders

• Cancel limit orders on market version updates if not PERSIST

• Non runner adjustment on matched orders and orders in the market

Docker

Create public docker image and tutorial on how to use.

Memory Leak

Flumine seems to leak memory when left running for extended periods, I believe it is due to order data not being collected by the GC but I am not sure why.

Image shows instance_016 leaking, the other instances are just data stream instances (no orders or market data)

Finance - Blotter

Create Blotter and BlotterMarket classes that hold orders.

Add to pypi

Market resettlement

Currently the storage engine will zip and load a market when initially closed however sometimes betfair resettle or provide an update after being closed.

• When closed add market to 'pending to zip/send to storage engine'
• Every hour zip/send any pending markets (don't delete from temp)
• If update received add to pending markets again
• End of day (EOD) involves deleting any loaded markets from /tmp

Order status EXPIRED

Add order status when order cannot be executed, handle place/cancel/update/replace

Strategy Runner Context Improvements

• Limit strategy count per runner, e.g. Strategy can only execute once per selection

• Multi, allow multiple trades to be active on a runner

• Place reset seconds

• Reset seconds

Implement backtesting logic for replace/update orders

• Update

• Replace

Market Close Bug

Often a market will come through the stream which has already closed, this causes an issue with the process_closed_market function as the Market object does not yet exist.

A hacky error catch has been added but this should be removed and a Market object be created in advance of the close function being called.

https://github.com/liampauling/flumine/blob/master/flumine/baseflumine.py#L197

Bug / refactor keep-alive worker

Keep-alive worker currently executes every 1200s, to be refactored:

• Execute every 600s?

• Check if expiry close to expiry based on trading client locale

readme bug

Hi,

Just noticed that the setup in the README mentions

"from flumine.resources import StreamRecorder"

Which doesn't seem to exist.

Trade not completed if no live orders and violation

Trades are not completed if there are no executable orders in a market and the order violates a control

RuntimeError: dictionary changed size during iteration

Happens one time, after working a couple of hours. Can reproduce.

INFO:flumine.order.order:Order status update: Pending INFO:flumine.order.order:Order status update: Pending INFO:flumine.markets.blotter:1 order packages created INFO:flumine.execution.betfairexecution:execute_place INFO:flumine.order.trade:Trade status update: Pending INFO:flumine.execution.baseexecution:Order Place: SUCCESS INFO:flumine.order.order:Order status update: Executable INFO:flumine.order.trade:Trade status update: Live INFO:flumine.order.trade:Trade status update: Pending INFO:flumine.execution.baseexecution:Order Place: SUCCESS INFO:flumine.order.order:Order status update: Executable INFO:flumine.order.trade:Trade status update: Live INFO:flumine.order.order:Order status update: Execution complete INFO:flumine.order.trade:Trade status update: Complete INFO:flumine.order.order:Order status update: Replacing INFO:flumine.markets.blotter:1 order packages created INFO:flumine.execution.betfairexecution:execute_replace INFO:flumine.order.trade:Trade status update: Pending INFO:flumine.execution.baseexecution:Order Cancel: SUCCESS INFO:flumine.order.order:Order status update: Execution complete INFO:flumine.execution.baseexecution:Order Replace: SUCCESS INFO:flumine.order.order:Order status update: Pending INFO:flumine.order.order:Order status update: Executable INFO:flumine.order.trade:Trade status update: Live INFO:flumine.streams.orderstream:Stopped OrderStream 1001 INFO:flumine.streams.marketstream:Stopped MarketStream 2001 INFO:flumine.streams.marketstream:Stopped output_thread (MarketStream 2001) INFO:flumine.execution.baseexecution:Shutting down Execution (SimulatedExecution) INFO:flumine.execution.baseexecution:Shutting down Execution (BetfairExecution) INFO:flumine.streams.orderstream:Stopped output_thread (OrderStream 1001) INFO:flumine.baseflumine:Exiting flumine Traceback (most recent call last): File "./pydogs/greyscout2live.py", line 727, in <module> framework.run() File "/usr/local/lib/python3.7/site-packages/flumine/flumine.py", line 32, in run self._process_current_orders(event) File "/usr/local/lib/python3.7/site-packages/flumine/baseflumine.py", line 183, in _process_current_orders strategy_orders = market.blotter.strategy_orders(strategy) File "/usr/local/lib/python3.7/site-packages/flumine/markets/blotter.py", line 27, in strategy_orders return [order for order in self if order.trade.strategy == strategy] File "/usr/local/lib/python3.7/site-packages/flumine/markets/blotter.py", line 27, in <listcomp> return [order for order in self if order.trade.strategy == strategy] RuntimeError: dictionary changed size during iteration

Add marketbook.publish_time to orders

This will allow backtesting users to identify the location of fill times.

API

Add the following endpoints:

• settings

• stream

• recorder

• storage

Add documentation

Use wiki, add simple tutorials and flumine API docs.

Github Actions

• Use for testing rather than travis?

• Auto deploy to pypi when merged to master

Encode does this well.

Error handling to stream

Add error handling to steam, use retrying to recreate and reconnect to stream, need to use the clk to prevent issues with the data being recorded.

Market filter

Need to be able to filter on markets based on market definition.

Flumine.info

Add flumine.info property which shows current state of all objects:

{
    "markets": {
        "live_event_count": 27,
        "open_market_count": 216,
        "live_market_count": 217,
        "total_market_count": 288,
        "closed_market_count": 494
    },
    "client": {
        "exchange": "BETFAIR",
        "username": "username",
        "betting_client": "<APIClient [username]>",
        "account_balance": 10036.03,
        "exposure": -1.4,
        "commission_rate": 0.02,
        "chargeable_transaction_count": 0,
        "trading_controls": [
            "MAX_ORDER_COUNT"
         ]
    },
    "markets": [
        "Market: 1.170562963, £-12.22, 71 orders, closed: True",
        "Market: 1.170562965, £-2.04, 13 orders, closed: True",
        "Market: 1.170562967, £-2.18, 40 orders, closed: True",
        "Market: 1.170562970, £3.34, 50 orders, closed: True",
        "Market: 1.170553165, £-3.80, 58 orders, closed: True",
        "Market: 1.170553157, £22.51, 28 orders, closed: True",
        "Market: 1.170553167, £8.84, 53 orders, closed: True",
        "Market: 1.170553153, £-4.60, 51 orders, closed: True",
    ],
    },
    "streams": [
        "<OrderStreaming(OrderStreaming, started daemon 140624477026048)>",
        "<OrderStreaming(OrderStreaming, started daemon 140624468633344)>",
        "<MarketStreaming(MarketStreaming, started daemon 140624451847936)>",
    ],
    "logging_controls": [
        "<APIControl(logging_control, started daemon 140624065980160)>"
    ],
    "threads": [
        "<_MainThread(MainThread, started 140475220645696)>",
        "<BackgroundWorker(memory_profiler, started daemon 140475181455104)>",
        "<BackgroundWorker(keep_alive, started daemon 140475092694784)>",
        "<BackgroundWorker(poll_account_balance, started daemon 140475084302080)>",
        "<BackgroundWorker(poll_market_catalogue, started daemon 140475075909376)>",
        "<BackgroundWorker(poll_cleared_orders, started daemon 140475067516672)>",
        "<OrderStream(OrderStream, started daemon 140475059123968)>",
        "<DataStream(DataStream, started daemon 140475050731264)>",
        "<Thread(OrderStream_output_thread, started daemon 140475042338560)>",
        "<MarketStream(MarketStream, started daemon 140474622932736)>",
        "<Thread(MarketStream_output_thread, started daemon 140474614540032)>",
        "<Thread(raven-sentry.BackgroundWorker, started daemon 140474606147328)>"
    ]
}

Add hedging/take-profit/stop loss functionality

Allow each complete order to have an adjustable take-profit price and/or stop loss price so it can be framed within a trading range. Currently possible solution is by using multiple strategies.

Err Handling on strategy.process_orders

Add error handling to strategy.process_orders

RuntimeError: dictionary changed size during iteration

Happens 1/10 times at launch on greyhounds markets especially if morning and a lot of markets

strategy = LiveStrategy( market_filter=streaming_market_filter( event_type_ids=["4339"], market_types=["WIN"], bsp_market=True, turn_in_play_enabled=False, country_codes=["AU", "GB", "US"], ), streaming_timeout=0, max_order_exposure=300, max_selection_exposure=50, )

Following logs, check the end

`-> % python3 ./pydogs/greyscout2live.py
INFO:flumine.baseflumine:Adding trading control ORDER_VALIDATION
INFO:flumine.baseflumine:Adding trading control STRATEGY_EXPOSURE
INFO:flumine.baseflumine:Adding client control MAX_ORDER_COUNT
INFO:flumine.streams.streams:Creating new <class 'flumine.streams.marketstream.MarketStream'> (2000) for strategy LiveStrategy
INFO:flumine.baseflumine:Starting flumine
INFO:flumine.worker:BackgroundWorker keep_alive starting
INFO:flumine.streams.streams:Starting streams..
INFO:flumine.streams.orderstream:Starting OrderStream
INFO:flumine.streams.orderstream:Starting output_thread <Thread(OrderStream_output_thread, initial daemon)>
INFO:flumine.streams.marketstream:Starting MarketStream
INFO:flumine.streams.marketstream:Starting output_thread <Thread(MarketStream_output_thread, initial daemon)>
INFO:betfairlightweight.streaming.listener:Register: marketSubscription 2001
INFO:betfairlightweight.streaming.listener:Register: orderSubscription 1001
INFO:betfairlightweight.streaming.stream:[Stream: 2001]: "MarketStream" created
INFO:betfairlightweight.streaming.stream:[Stream: 1001]: "OrderStream" created
INFO:betfairlightweight.streaming.listener:[Connect: 1001]: connection_id: 205-010720110931-525978
INFO:betfairlightweight.streaming.listener:[Connect: 2001]: connection_id: 205-010720110931-525979
INFO:betfairlightweight.streaming.listener:[Subscription: 1002]: SUCCESS (9 connections available)
INFO:betfairlightweight.streaming.listener:[Subscription: 2002]: SUCCESS (8 connections available)
INFO:betfairlightweight.streaming.listener:[Subscription: 1001]: SUCCESS (9 connections available)
INFO:betfairlightweight.streaming.stream:[OrderStream: 1001] 1.171123192 added, 1 markets in cache
INFO:betfairlightweight.streaming.listener:[Subscription: 2001]: SUCCESS (8 connections available)
INFO:betfairlightweight.streaming.stream:[OrderStream: 1001] 1.171123190 added, 2 markets in cache
INFO:betfairlightweight.streaming.stream:[OrderStream: 1001] 1.171123434 added, 3 markets in cache
INFO:betfairlightweight.streaming.stream:[Stream: 1001]: 3 oc added
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123005 added, 1 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093548 added, 2 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123725 added, 3 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093546 added, 4 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123007 added, 5 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123741 added, 6 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093552 added, 7 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123739 added, 8 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123737 added, 9 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123735 added, 10 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123009 added, 11 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093554 added, 12 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093550 added, 13 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123011 added, 14 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123003 added, 15 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123719 added, 16 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123729 added, 17 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123721 added, 18 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123015 added, 19 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123013 added, 20 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124034 added, 21 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124036 added, 22 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124032 added, 23 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124028 added, 24 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124022 added, 25 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124018 added, 26 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123733 added, 27 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123731 added, 28 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124038 added, 29 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124020 added, 30 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124026 added, 31 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123723 added, 32 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093544 added, 33 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124016 added, 34 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123001 added, 35 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122999 added, 36 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093542 added, 37 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124024 added, 38 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124030 added, 39 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123727 added, 40 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122997 added, 41 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093540 added, 42 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091149 added, 43 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091145 added, 44 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090135 added, 45 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124153 added, 46 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091153 added, 47 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091159 added, 48 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131146 added, 49 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131176 added, 50 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131144 added, 51 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091157 added, 52 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131174 added, 53 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091151 added, 54 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131134 added, 55 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131142 added, 56 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090604 added, 57 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131172 added, 58 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131140 added, 59 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131126 added, 60 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131170 added, 61 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131124 added, 62 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091147 added, 63 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131138 added, 64 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091155 added, 65 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131168 added, 66 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131136 added, 67 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090132 added, 68 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090601 added, 69 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091163 added, 70 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171142243 added, 71 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131154 added, 72 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131120 added, 73 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131128 added, 74 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131130 added, 75 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131162 added, 76 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131132 added, 77 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131164 added, 78 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131166 added, 79 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124163 added, 80 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131152 added, 81 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131122 added, 82 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131160 added, 83 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131158 added, 84 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131156 added, 85 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171131150 added, 86 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124157 added, 87 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124165 added, 88 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124159 added, 89 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124167 added, 90 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124175 added, 91 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091143 added, 92 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124173 added, 93 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124155 added, 94 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090130 added, 95 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091141 added, 96 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124171 added, 97 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124169 added, 98 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124161 added, 99 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090128 added, 100 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171090126 added, 101 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122759 added, 102 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123194 added, 103 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091179 added, 104 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093522 added, 105 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122765 added, 106 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123436 added, 107 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123198 added, 108 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123440 added, 109 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123438 added, 110 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124280 added, 111 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123196 added, 112 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093524 added, 113 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122763 added, 114 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122757 added, 115 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093526 added, 116 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091183 added, 117 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123192 added, 118 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122761 added, 119 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122769 added, 120 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122771 added, 121 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093528 added, 122 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123442 added, 123 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091181 added, 124 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093530 added, 125 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093520 added, 126 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123448 added, 127 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124234 added, 128 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122767 added, 129 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124216 added, 130 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122773 added, 131 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123204 added, 132 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123446 added, 133 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123202 added, 134 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124226 added, 135 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124286 added, 136 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123206 added, 137 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123200 added, 138 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124296 added, 139 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124294 added, 140 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124228 added, 141 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124218 added, 142 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123190 added, 143 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124298 added, 144 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124230 added, 145 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124236 added, 146 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124292 added, 147 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124290 added, 148 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124288 added, 149 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124238 added, 150 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123444 added, 151 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091177 added, 152 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123434 added, 153 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124282 added, 154 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122755 added, 155 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123432 added, 156 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093518 added, 157 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091175 added, 158 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123188 added, 159 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124300 added, 160 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122753 added, 161 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124224 added, 162 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124222 added, 163 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124220 added, 164 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124284 added, 165 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123430 added, 166 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123186 added, 167 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171093516 added, 168 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171091173 added, 169 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171122751 added, 170 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123428 added, 171 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171123184 added, 172 markets in cache
INFO:betfairlightweight.streaming.stream:[MarketStream: 2001] 1.171124232 added, 173 markets in cache
INFO:betfairlightweight.streaming.stream:[Stream: 2001]: 173 mc added
INFO:flumine.baseflumine:Adding: 1.171123005 to markets
INFO:flumine.baseflumine:Adding: 1.171093548 to markets
INFO:flumine.baseflumine:Adding: 1.171123725 to markets
INFO:flumine.baseflumine:Adding: 1.171093546 to markets
INFO:flumine.baseflumine:Adding: 1.171123007 to markets
INFO:flumine.baseflumine:Adding: 1.171123741 to markets
INFO:flumine.baseflumine:Adding: 1.171093552 to markets
INFO:flumine.baseflumine:Adding: 1.171123739 to markets
INFO:flumine.baseflumine:Adding: 1.171123737 to markets
INFO:flumine.baseflumine:Adding: 1.171123735 to markets
INFO:flumine.baseflumine:Adding: 1.171123009 to markets
INFO:flumine.baseflumine:Adding: 1.171093554 to markets
INFO:flumine.baseflumine:Adding: 1.171093550 to markets
INFO:flumine.baseflumine:Adding: 1.171123011 to markets
INFO:flumine.baseflumine:Adding: 1.171123719 to markets
INFO:flumine.baseflumine:Adding: 1.171123729 to markets
INFO:flumine.baseflumine:Adding: 1.171123721 to markets
INFO:flumine.baseflumine:Adding: 1.171123015 to markets
INFO:flumine.baseflumine:Adding: 1.171123013 to markets
INFO:flumine.baseflumine:Adding: 1.171124034 to markets
INFO:flumine.baseflumine:Adding: 1.171124036 to markets
INFO:flumine.baseflumine:Adding: 1.171124032 to markets
INFO:flumine.baseflumine:Adding: 1.171124028 to markets
INFO:flumine.baseflumine:Adding: 1.171124022 to markets
INFO:flumine.baseflumine:Adding: 1.171124018 to markets
INFO:flumine.baseflumine:Adding: 1.171123733 to markets
INFO:flumine.baseflumine:Adding: 1.171123731 to markets
INFO:flumine.baseflumine:Adding: 1.171124038 to markets
INFO:flumine.baseflumine:Adding: 1.171124020 to markets
INFO:flumine.baseflumine:Adding: 1.171124026 to markets
INFO:flumine.baseflumine:Adding: 1.171123723 to markets
INFO:flumine.baseflumine:Adding: 1.171124016 to markets
INFO:flumine.baseflumine:Adding: 1.171124024 to markets
INFO:flumine.baseflumine:Adding: 1.171124030 to markets
INFO:flumine.baseflumine:Adding: 1.171123727 to markets
INFO:flumine.baseflumine:Adding: 1.171091149 to markets
INFO:flumine.baseflumine:Adding: 1.171091145 to markets
INFO:flumine.baseflumine:Adding: 1.171090135 to markets
INFO:flumine.baseflumine:Adding: 1.171124153 to markets
INFO:flumine.baseflumine:Adding: 1.171091153 to markets
INFO:flumine.baseflumine:Adding: 1.171091159 to markets
INFO:flumine.baseflumine:Adding: 1.171131146 to markets
INFO:flumine.baseflumine:Adding: 1.171131176 to markets
INFO:flumine.baseflumine:Adding: 1.171131144 to markets
INFO:flumine.baseflumine:Adding: 1.171091157 to markets
INFO:flumine.baseflumine:Adding: 1.171131174 to markets
INFO:flumine.baseflumine:Adding: 1.171091151 to markets
INFO:flumine.baseflumine:Adding: 1.171131134 to markets
INFO:flumine.baseflumine:Adding: 1.171131142 to markets
INFO:flumine.baseflumine:Adding: 1.171090604 to markets
INFO:flumine.baseflumine:Adding: 1.171131172 to markets
INFO:flumine.baseflumine:Adding: 1.171131140 to markets
INFO:flumine.baseflumine:Adding: 1.171131126 to markets
INFO:flumine.baseflumine:Adding: 1.171131170 to markets
INFO:flumine.baseflumine:Adding: 1.171131124 to markets
INFO:flumine.baseflumine:Adding: 1.171091147 to markets
INFO:flumine.baseflumine:Adding: 1.171131138 to markets
INFO:flumine.baseflumine:Adding: 1.171091155 to markets
INFO:flumine.baseflumine:Adding: 1.171131168 to markets
INFO:flumine.baseflumine:Adding: 1.171131136 to markets
INFO:flumine.baseflumine:Adding: 1.171090601 to markets
INFO:flumine.baseflumine:Adding: 1.171091163 to markets
INFO:flumine.baseflumine:Adding: 1.171142243 to markets
INFO:flumine.baseflumine:Adding: 1.171131154 to markets
INFO:flumine.baseflumine:Adding: 1.171131120 to markets
INFO:flumine.baseflumine:Adding: 1.171131128 to markets
INFO:flumine.baseflumine:Adding: 1.171131130 to markets
INFO:flumine.baseflumine:Adding: 1.171131162 to markets
INFO:flumine.baseflumine:Adding: 1.171131132 to markets
INFO:flumine.baseflumine:Adding: 1.171131164 to markets
INFO:flumine.baseflumine:Adding: 1.171131166 to markets
INFO:flumine.baseflumine:Adding: 1.171124163 to markets
INFO:flumine.baseflumine:Adding: 1.171131152 to markets
INFO:flumine.baseflumine:Adding: 1.171131122 to markets
INFO:flumine.baseflumine:Adding: 1.171131160 to markets
INFO:flumine.baseflumine:Adding: 1.171131158 to markets
INFO:flumine.baseflumine:Adding: 1.171131156 to markets
INFO:flumine.baseflumine:Adding: 1.171131150 to markets
INFO:flumine.baseflumine:Adding: 1.171124157 to markets
INFO:flumine.baseflumine:Adding: 1.171124165 to markets
INFO:flumine.baseflumine:Adding: 1.171124159 to markets
INFO:flumine.baseflumine:Adding: 1.171124167 to markets
INFO:flumine.baseflumine:Adding: 1.171124175 to markets
INFO:flumine.baseflumine:Adding: 1.171124173 to markets
INFO:flumine.baseflumine:Adding: 1.171124155 to markets
INFO:flumine.baseflumine:Adding: 1.171124171 to markets
INFO:flumine.baseflumine:Adding: 1.171124169 to markets
INFO:flumine.baseflumine:Adding: 1.171124161 to markets
INFO:flumine.baseflumine:Adding: 1.171122759 to markets
INFO:flumine.baseflumine:Adding: 1.171123194 to markets
INFO:flumine.baseflumine:Adding: 1.171091179 to markets
INFO:flumine.baseflumine:Adding: 1.171093522 to markets
INFO:flumine.baseflumine:Adding: 1.171122765 to markets
INFO:flumine.baseflumine:Adding: 1.171123436 to markets
INFO:flumine.baseflumine:Adding: 1.171123198 to markets
INFO:flumine.baseflumine:Adding: 1.171123440 to markets
INFO:flumine.baseflumine:Adding: 1.171123438 to markets
INFO:flumine.baseflumine:Adding: 1.171124280 to markets
INFO:flumine.baseflumine:Adding: 1.171123196 to markets
INFO:flumine.baseflumine:Adding: 1.171093524 to markets
INFO:flumine.baseflumine:Adding: 1.171122763 to markets
INFO:flumine.baseflumine:Adding: 1.171093526 to markets
INFO:flumine.baseflumine:Adding: 1.171091183 to markets
INFO:flumine.baseflumine:Adding: 1.171122761 to markets
INFO:flumine.baseflumine:Adding: 1.171122769 to markets
INFO:flumine.baseflumine:Adding: 1.171122771 to markets
INFO:flumine.baseflumine:Adding: 1.171093528 to markets
INFO:flumine.baseflumine:Adding: 1.171123442 to markets
INFO:flumine.baseflumine:Adding: 1.171091181 to markets
INFO:flumine.baseflumine:Adding: 1.171093530 to markets
INFO:flumine.baseflumine:Adding: 1.171123448 to markets
INFO:flumine.baseflumine:Adding: 1.171124234 to markets
INFO:flumine.baseflumine:Adding: 1.171122767 to markets
INFO:flumine.baseflumine:Adding: 1.171124216 to markets
INFO:flumine.baseflumine:Adding: 1.171122773 to markets
INFO:flumine.baseflumine:Adding: 1.171123204 to markets
INFO:flumine.baseflumine:Adding: 1.171123446 to markets
INFO:flumine.baseflumine:Adding: 1.171123202 to markets
INFO:flumine.baseflumine:Adding: 1.171124226 to markets
INFO:flumine.baseflumine:Adding: 1.171124286 to markets
INFO:flumine.baseflumine:Adding: 1.171123206 to markets
INFO:flumine.baseflumine:Adding: 1.171123200 to markets
INFO:flumine.baseflumine:Adding: 1.171124296 to markets
INFO:flumine.baseflumine:Adding: 1.171124294 to markets
INFO:flumine.baseflumine:Adding: 1.171124228 to markets
INFO:flumine.baseflumine:Adding: 1.171124218 to markets
INFO:flumine.baseflumine:Adding: 1.171124298 to markets
INFO:flumine.baseflumine:Adding: 1.171124230 to markets
INFO:flumine.baseflumine:Adding: 1.171124236 to markets
INFO:flumine.baseflumine:Adding: 1.171124292 to markets
INFO:flumine.baseflumine:Adding: 1.171124290 to markets
INFO:flumine.baseflumine:Adding: 1.171124288 to markets
INFO:flumine.baseflumine:Adding: 1.171124238 to markets
INFO:flumine.baseflumine:Adding: 1.171123444 to markets
INFO:flumine.baseflumine:Adding: 1.171124282 to markets
INFO:flumine.baseflumine:Adding: 1.171124300 to markets
INFO:flumine.baseflumine:Adding: 1.171124224 to markets
INFO:flumine.baseflumine:Adding: 1.171124222 to markets
INFO:flumine.baseflumine:Adding: 1.171124220 to markets
INFO:flumine.baseflumine:Adding: 1.171124284 to markets
INFO:flumine.baseflumine:Adding: 1.171124232 to markets
WARNING:flumine.baseflumine:Market 1.171123003 not present when closing
WARNING:flumine.baseflumine:Market 1.171093544 not present when closing
WARNING:flumine.baseflumine:Market 1.171123001 not present when closing
WARNING:flumine.baseflumine:Market 1.171122999 not present when closing
WARNING:flumine.baseflumine:Market 1.171093542 not present when closing
WARNING:flumine.baseflumine:Market 1.171122997 not present when closing
WARNING:flumine.baseflumine:Market 1.171093540 not present when closing
WARNING:flumine.baseflumine:Market 1.171090132 not present when closing
WARNING:flumine.baseflumine:Market 1.171091143 not present when closing
WARNING:flumine.baseflumine:Market 1.171090130 not present when closing
WARNING:flumine.baseflumine:Market 1.171091141 not present when closing
WARNING:flumine.baseflumine:Market 1.171090128 not present when closing
WARNING:flumine.baseflumine:Market 1.171090126 not present when closing
WARNING:flumine.baseflumine:Market 1.171122757 not present when closing
WARNING:flumine.baseflumine:Market 1.171123192 not present when closing
WARNING:flumine.baseflumine:Market 1.171093520 not present when closing
WARNING:flumine.baseflumine:Market 1.171123190 not present when closing
WARNING:flumine.baseflumine:Market 1.171091177 not present when closing
WARNING:flumine.baseflumine:Market 1.171123434 not present when closing
WARNING:flumine.baseflumine:Market 1.171122755 not present when closing
WARNING:flumine.baseflumine:Market 1.171123432 not present when closing
WARNING:flumine.baseflumine:Market 1.171093518 not present when closing
WARNING:flumine.baseflumine:Market 1.171091175 not present when closing
WARNING:flumine.baseflumine:Market 1.171123188 not present when closing
WARNING:flumine.baseflumine:Market 1.171122753 not present when closing
WARNING:flumine.baseflumine:Market 1.171123430 not present when closing
WARNING:flumine.baseflumine:Market 1.171123186 not present when closing
WARNING:flumine.baseflumine:Market 1.171093516 not present when closing
WARNING:flumine.baseflumine:Market 1.171091173 not present when closing
WARNING:flumine.baseflumine:Market 1.171122751 not present when closing
WARNING:flumine.baseflumine:Market 1.171123428 not present when closing
WARNING:flumine.baseflumine:Market 1.171123184 not present when closing
Exception in thread MarketStream_output_thread:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flumine/streams/marketstream.py", line 47, in handle_output
block=True, timeout=self.streaming_timeout
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/queue.py", line 178, in get
raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/flumine/streams/marketstream.py", line 51, in handle_output
market_ids=self.flumine.markets.open_market_ids
File "/usr/local/lib/python3.7/site-packages/flumine/markets/markets.py", line 47, in open_market_ids
return [m.market_id for m in self if not m.closed]
File "/usr/local/lib/python3.7/site-packages/flumine/markets/markets.py", line 47, in
return [m.market_id for m in self if not m.closed]
RuntimeError: dictionary changed size during iteration

INFO:flumine.worker:BackgroundWorker poll_account_balance starting
INFO:flumine.worker:BackgroundWorker poll_cleared_orders starting
INFO:flumine.worker:BackgroundWorker poll_market_catalogue starting
INFO:flumine.baseflumine:Updated marketCatalogue for 1.171093546
INFO:flumine.baseflumine:Updated marketCatalogue for 1.171123436
INFO:flumine.baseflumine:Updated marketCatalogue for 1.171122759
`

Multiple Storage Engines

Currently limited to one.

• Update to list

• Refactor recorder / storage engine to be simple when initialising?

Dead head simulated profit calculations may not be correct

In the case where there is a dead head, simulated profit is calculated normally.

market_id	selection_id	side	average_price_matched	size_matched	simulated_profit	runner_status
1.169171164	16742042	LAY	2.28                    2               -2.56	                WINNER
1.169171164	22160756	LAY	5.2                     2               -8.40	                WINNER

Based on the following link I was expecting:
https://support.betfair.com/app/answers/detail/a_id/403/c/483/

market_id	selection_id	side	average_price_matched	size_matched	simulated_profit	runner_status
1.169171164	16742042	LAY	2.28                    2               -0.28	                WINNER
1.169171164	22160756	LAY	5.2                     2               -3.20	                WINNER

Trades have orders

Create interfaces for Trades and Orders.

Execution

Create classes for order execution

BaseExecution

place_order
cancel_order
replace_order
update_order

BetfairExecution

SimulatedExecution

Exchange based execution should use a thread pool with a requests.session pool to reduce latency. Simulated will be single threaded but place/cancel latency will need to be introduced.

Order Stream

Add and integrate order stream and relevant functions.

flumine._process_current_orders()

strategy.process_current_orders()

Non runner adjustments

Adjust simulated trades / orders for non runners

Order.cleared_order

Add cleared_order object to Order object, _process_cleared_orders currently does nothing.

Add max_latency

StreamListener now takes max_latency as a function.

Add Frontend

Use /api to create dynamic frontend similiar to portainer.io with the following:

• dashboard (stream status)
• streams (start / stop / update)
• storage (view / update)
• settings (view / update)

Strategy Exposure Improvements

In the current code, if a user has a matched back for £2@1000, and then tries to lay £2@500, they might receive
a warning...

"Potential selection exposure ({0}) is greater than strategy.max_selection_exposure ({1})"

This should not be the case, as the lay of £500 will be reducing the overall risk.

Other potential problems...

• the current code does not appear to consider unmatched exposure.
• multiple orders, in the same package, could cause exposure to go over the limit, even if they don't individually.

=== Ideas for improvement ===

Refactor blotter.selection_exposure(...) to be called "matched_selection_exposure" to return "matched_profit_if_win" and "matched_profit_if_lose".
Add method blotter.unmatched_selection_exposure to return potential_unmatched_profit_if_win, potential_unmatched_profit_if_lose.

In StrategyExposure._validate, instead of performing the "per selection" validation in the "for order in order_package" loop,
we could iterate over the orders in the order package, and calculate the aggregated profit_if_win and profit_if_lose
values per order.lookup

So, for each "lookup", there will be 3 sources of risk (i.e. profit_if_win [PIW], profit_if_lose [PIL]):

• matched bets
• unmatched bets
• order package

When considering our exposure for unmatched bets and the order package, we only care about potential losses.

So...

exposure_on_win = matched_piw + min(unmatched_piw, 0) + min(order_package_piw, 0)
exposure_on_loss = matched_pil + min(unmathced_pil, 0) + min(order_package_pil, 0)

And the actual_exposure is min(exposure_on_win, exposure_on_loss)

Multi clients

Currently (to keep things simple) flumine is limited to a single client instance however there are times when having access to multiple clients is beneficial.

• Different account per strategy
• Paper trade concurrently with live strategies
• Allows the development of alternative exchanges (matchbook/smarkets/betdaq) or bookmakers

Changes:

• Migrate flumine.client to flumine.clients

• Handle clients in OrderPackage

Strategy error handling

Currently an error in a user created strategy will break everything as it runs in the main thread, error handling needs to be added to:

def call_check_market(strategy_check_market: Callable, market...)
    try:
        strategy_check_market(market...

def call_process_market_book(strategy_process_market_book: Callable, market...)
    try:
        strategy_process_market_book(market...

And then add to baseflumine.

Disable strategy if errors?

Latency warning

Issue raised on slack where the main handler queue was not being processed fast enough resulting on orders/decisions being based on stale data and then the orders themselves being delayed.

• Priority queue?

• Logging warning when data is stale (>1s?)

Middleware

Django type middleware allowing:

process_market_book(market, market_book)
process_order_book(market, order_book)

Typical use case would be the ability to store data in market.context or market.history for example historic data or moving averages.

This would be called before strategies are called or orders processed.

Storage

Make flumine platform independent with the following example strategies:

• local host including docker volumes

• AWS s3

• Google cloud storage

• Digital Ocean Object Storage

• Azure storage

• Dropbox

Client - Transaction Count

Betfair default is 1000 transactions before charge however not all users want to limit to this:

• allow override of limit
• allow removal of transaction count client control

Multi market processing per strategy

Handle:

• Multi markets with pending order per single market update (place orders in PLACE/OU2.5 market based on WIN/MATCH_ODDS update)

• Access another market from within strategy (as above)

place_markets = market.event.get("PLACE")  # [<Market>, ..]

event_markets = markets.event  # {"PLACE": [<Market>], "WIN": [<Market>], "EACH_WAY": [<Market>]}

Enforcement of strategy exposure limits may be incorrect

If we use the following scenario:

• Set max_selection_exposure as 2.
• Back selection A at odds of 5, with a stake of £2.
• This order goes through as it is within the limits.
• Then, lay the selection at odds of 1.5 with a stake of £2. Liability here is £1.
• StrategyExposure then does the following computation:

exposure = (order.order_type.price - 1) * order.order_type.size 
#correct (computed as 1)

current_selection_exposure = market.blotter.selection_exposure( strategy, lookup=order.lookup)
#current_selection_exposure is computed as -2 due to the original back bet.

if (current_selection_exposure - exposure) < -strategy.max_selection_exposure:
    self._on_error(order)
#This is computed as If -2 -1 < -2: self._on_error(order)

In this case, the lay would have been rejected, despite being risk-reducing. Perhaps it would make it easier if "back" exposure was tracked as a positive value, and lay exposure tracked as a negative value (as one would do if they were buying/selling).