kismuz/btgym

Scalable, event-driven, deep-learning-friendly backtesting library

...Minimizing the mean square error on future experience.  - Richard S. Sutton

BTGym

Scalable event-driven RL-friendly backtesting library. Build on top of Backtrader with OpenAI Gym environment API.

Backtrader is open-source algorithmic trading library:
GitHub: http://github.com/mementum/backtrader
Documentation and community:
http://www.backtrader.com/

OpenAI Gym is..., well, everyone knows Gym:
GitHub: http://github.com/openai/gym
Documentation and community:
https://gym.openai.com/

---

Outline

General purpose of this project is to provide gym-integrated framework for running reinforcement learning experiments in [close to] real world algorithmic trading environments.

DISCLAIMER:
Code presented here is research/development grade.
Can be unstable, buggy, poor performing and is subject to change.

Note that this package is neither out-of-the-box-moneymaker, nor it provides ready-to-converge RL solutions.
Think of it as framework for setting experiments with complex non-stationary stochastic environments.

As a research project BTGym in its current stage can hardly deliver easy end-user experience in as sense that
setting meaninfull  experiments will require some practical programming experience as well as general knowledge
of reinforcement learning theory.

---

News and update notes

---

Contents

• Installation
• Quickstart
• Description
  • Problem setting
  • Data sampling approaches
• Documentation and community
• Known bugs and limitations
• Roadmap
• Update news

---

Installation

It is highly recommended to run BTGym in designated virtual environment.

Clone or copy btgym repository to local disk, cd to it and run: pip install -e . to install package and all dependencies:

git clone https://github.com/Kismuz/btgym.git

cd btgym

pip install -e .

To update to latest version::

cd btgym

git pull

pip install --upgrade -e .

Notes:

1. BTGym requres Matplotlib version 2.0.2, downgrade your installation if you have version 2.1:

   pip install matplotlib==2.0.2

2. LSOF utility should be installed to your OS, which can not be the default case for some Linux distributives, see: https://en.wikipedia.org/wiki/Lsof

---

Quickstart

Making gym environment with all parmeters set to defaults is as simple as:

from btgym import BTgymEnv

MyEnvironment = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',)

Adding more controls may look like:

from gym import spaces
from btgym import BTgymEnv

MyEnvironment = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
                         episode_duration={'days': 2, 'hours': 23, 'minutes': 55},
                         drawdown_call=50,
                         state_shape=dict(raw=spaces.Box(low=0,high=1,shape=(30,4))),
                         port=5555,
                         verbose=1,
                         )

See more options at Documentation: Quickstart >>

and how-to's in Examples directory >>.

---

General description

Problem setting

• Discrete actions setup: consider setup with one riskless asset acting as broker account cash and K (by default - one) risky assets. For every risky asset there exists track of historic price records referred as data-line. Apart from assets data lines there [optionally] exists number of exogenous data lines holding some information and statistics, e.g. economic indexes, encoded news, macroeconomic indicators, weather forecasts etc. which are considered relevant to decision-making. It is supposed for this setup that:

  1. there is no interest rates for any asset;
  2. broker actions are fixed-size market orders (buy, sell, close); short selling is permitted;
  3. transaction costs are modelled via broker commission;
  4. 'market liquidity' and 'capital impact' assumptions are met;
  5. time indexes match for all data lines provided;

• The problem is modelled as discrete-time finite-horizon partially observable Markov decision process for equity/currency trading:

  • for every asset traded agent action space is discrete (0: hold [do nothing], 1:buy, 2: sell, 3:close [position]);
  • environment is episodic: maximum episode duration and episode termination conditions are set;
  • for every timestep of the episode agent is given environment state observation as tensor of last m time-embedded preprocessed values for every data-line included and emits actions according some stochastic policy.
  • agent's goal is to maximize expected cumulative capital by learning optimal policy;

• Continuous actions setup[BETA]: this setup closely relates to continuous portfolio optimisation problem definition; it differs from setup above in:

  1. base broker actions are real numbers: a[i] in [0,1], 0<=i<=K, SUM{a[i]} = 1 for K risky assets added; each action is a market target order to adjust portfolio to get share a[i]*100% for i-th asset;
  2. entire single-step broker action is dictionary of form: {cash_name: a[0], asset_name_1: a[1], ..., asset_name_K: a[K]};
  3. short selling is not permitted;

• For RL it implies having continuous action space as K+1 dim vector.

Data selection options for backtest agent training:

Notice: data shaping approach is under development, expect some changes. [7.01.18]

• random sampling: historic price change dataset is divided to training, cross-validation and testing subsets. Since agent actions do not influence market, it is possible to randomly sample continuous subset of training data for every episode. [Seems to be] most data-efficient method. Cross-validation and testing performed later as usual on most "recent" data;
• sequential sampling: full dataset is feeded sequentially as if agent is performing real-time trading, episode by episode. Most reality-like, least data-efficient, natural non-stationarity remedy.
• sliding time-window sampling: mixture of above, episde is sampled randomly from comparatively short time period, sliding from furthest to most recent training data. Should be less prone to overfitting than random sampling.

---

Documentation and Community

• Read Docs and API Reference.
• Browse Development Wiki.
• Review opened and closed Issues.
• Go to BTGym Slack channel. If you are new - use this invite link to join.

---

Known bugs and limitations:

• requres Matplotlib version 2.0.2;
• matplotlib backend warning: appears when importing pyplot and using %matplotlib inline magic before btgym import. It's recommended to import btacktrader and btgym first to ensure proper backend choice;
• not tested with Python < 3.5;
• doesn't seem to work correctly under Windows; partially done
• by default, is configured to accept Forex 1 min. data from www.HistData.com;
• only random data sampling is implemented;
• no built-in dataset splitting to training/cv/testing subsets; done
• only one equity/currency pair can be traded done
• no 'skip-frames' implementation within environment; done
• no plotting features, except if using pycharm integration observer. Not sure if it is suited for intraday strategies. [partially] done
• making new environment kills all processes using specified network port. Watch out your jupyter kernels. fixed

---

TODO's and Road Map:

• refine logic for parameters applying priority (engine vs strategy vs kwargs vs defaults);
• API reference;
• examples;
• frame-skipping feature;
• dataset tr/cv/t approach;
• state rendering;
• proper rendering for entire episode;
• tensorboard integration;
• multiply agents asynchronous operation feature (e.g for A3C):
• dedicated data server;
• multi-modal observation space shape;
• A3C implementation for BTgym;
• UNREAL implementation for BTgym;
• PPO implementation for BTgym;
• RL^2 / MAML / DARLA adaptations - IN PROGRESS;
• learning from demonstrations; - partially done
• risk-sensitive agents implementation;
• sequential and sliding time-window sampling;
• multiply instruments trading;
• docker image; - CPU version, Signalprime contribution,
• TF serving model serialisation functionality;

News and updates:

• 10.01.2019:

  • docker CPU version is now available, contributed by Signalprime, (https://github.com/signalprime), see btgym/docker/README.md for details;

• 9.02.2019:

  • Introduction to analytic data model notebook added to model_based_stat_arb examples folder.

• 25.01.2019: updates:

  • lstm_policy class now requires both internal and external observation sub-spaces to be present and allows both be one-level nested sub-spaces itself (was only true for external); all declared sub-spaces got encoded by separate convolution encoders;
  • policy deterministic action option is implemented for discrete action spaces and can be utilised by syncro_runner; by default it is enabled for test episodes;
  • data_feed classes now accept pd.dataframes as historic data dource via dataframe kwarg (was: .csv files only);

• 18.01.2019: updates:

  • data model classes are under active development to power model-based framework:
    • common statistics incremental estimator classes has been added (mean, variance, covariance, linear regression etc.);
    • incremental Singular Spectrum Analysis class implemented;
    • for a pair of asset prices, two-factor state-space model is proposed
  • new data_feed iterator classes has been added to provide training framework with synthetic data generated by model mentioned above;
  • strategy_gen_6 data handling and pre-processing has been redesigned:
    • market data SSA decomposition;
    • data model state as additional input to policy
    • variance-based normalisation for broker statistics

• 11.12.2018: updates and fixes:

  • training Launcher class got convenience features to save and reload model parameters, see https://github.com/Kismuz/btgym/blob/master/examples/unreal_stacked_lstm_strat_4_11.ipynb for details
  • combined model-based/model-free aproach package in early development stage is added to btgym.reserach

• 17.11.2018: updates and fixes:

  • minor fixes to base data provider class episode sampling
  • update to btgym.datafeed.synthetic subpackage: new stochastic processes generators added etc.
  • new btgym.research.startegy_gen_5 subpackage: efficient parameter-free signal preprocessing implemented, other minor improvements

• 30.10.2018: updates and fixes:

  • fixed numpy random state issue causing replicating of seeds among workers on POSIX os
  • new synthetic datafeed generators - added simple Ornshtein-Uhlenbeck process data generating classes; see btgym/datafeed/synthetic/ou.py and btgym/research/ou_params_space_eval for details;

• 14.10.2018: update:

  • base reward function redesign -> noticeable algorithms performance gain;

• 20.07.2018: major update to package:

  • enchancements to agent architecture:

    • casual convolution state encoder with attention for LSTM agent;
    • dropout regularization added for conv. and LSTM layers;

  • base strategy update: new convention for naming get_state methods, see BaseStrategy class for details;

  • multiply datafeeds and assets trading implemented in two flavors:

    • discrete actions space via MultiDiscreteEnv class;
    • continious actions space via PortfolioEnv which is closely related to contionious portfolio optimisation problem setup;

      • description and docs:

        • MultiDataFeed: https://kismuz.github.io/btgym/btgym.datafeed.html#btgym.datafeed.multi.BTgymMultiData
        • ActionSpace: https://kismuz.github.io/btgym/btgym.html#btgym.spaces.ActionDictSpace
        • MultiDiscreteEnv: https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.multidiscrete.MultiDiscreteEnv
        • PortfolioEnv: https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.portfolio.PortfolioEnv

      • examples:

        • MultiDiscreteEnv: https://github.com/Kismuz/btgym/blob/master/examples/multi_discrete_setup_intro.ipynb
        • PortfolioEnv: https://github.com/Kismuz/btgym/blob/master/examples/portfolio_setup_BETA.ipynb
    • Notes on multi-asset setup:
      • adding these features forced substantial package redesign; expect bugs, some backward incompatibility, broken examples etc - please report;
      • current algorithms and agents architectures are ok with multiply data lines but seem not to cope well with multi-asset setup. It is especially evident in case of continuous actions, where agents completely fail to converge on train data;
      • current reward function design seems inappropriate; need to reshape;
      • continuous space in beta and still needs some improvement, esp. for broker order execution logic as well as action sampling routine for continuous A3C (which is Dirichlet process by now);
      • multi-discrete space is more consistent but severely limited in number of portfolio assets (but not data-lines) due to exponential rise of action space cardinality; the option is to as use many datalines as desired while limiting portfolio to 1 - 4 assets;
      • no Guided Policy available for multi-asset setup yet - in progress;
      • all but episode rendering modes are temporally disabled;
      • whole thing is shamelessly resource-hungry;

• 17.02.18: First results on applying guided policy search ideas (GPS) to btgym setup can be seen here.

  • tensorboard summaries are updated with additional renderings: actions distribution, value function and LSTM_state; presented in the same notebook.

• 6.02.18: Common update to all a3c agents architectures:

  • all dense layers are now Noisy-Net ones, see: Noisy Networks for Exploration paper by Fortunato at al.;

  • note that entropy regularization is still here, kept in ~0.01 to ensure proper exploration;

  • policy output distribution is 'centered' using layer normalisation technique;

    • all of the above results in about 2x training speedup in terms of train iterations;

• 20.01.18: Project Wiki pages added;

• 12.01.18: Minor fixes to logging, enabled BTgymDataset train/test data split. AAC framework train/test cycle enabled via episode_train_test_cycle kwarg.

• 7.01.18: Update:

  • Major data pipe redesign. Domain -> Trial -> Episode sampling routine implemented. For motivation and formal definitions refer to Section 1.Data of this DRAFT, API Documentation and Intro example. Changes should be backward compatible. In brief, it is necessry framework for upcoming meta-learning algorithms.
  • logging changes: now relying in python logbook module. Should eliminate errors under Windows.
  • Stacked_LSTM_Policy agent implemented. Based on NAV_A3C from DeepMind paper with some minor mods. Basic usage Example is here. Still in research code area and need further tuning; yet faster than simple LSTM agent, able to converge on 6-month 1m dataset.

• 5.12.17: Inner btgym comm. fixes >> speedup ~5%.

• 02.12.17: Basic sliding time-window train/test framework implemented via BTgymSequentialTrial() class. UPD: replaced by BTgymSequentialDataDomain class.

• 29.11.17: Basic meta-learning RL^2 functionality implemented.

  • See Trial_Iterator Class and RL^2 policy for description.
  • Effectiveness is not tested yet, examples are to follow.

• 24.11.17: A3C/UNREAL finally adapted to work with BTGym environments.

  • Examples with synthetic simple data(sine wawe) and historic financial data added, see examples directory;
  • Results on potential-based functions reward shaping in /research/DevStartegy_4_6;
  • Work on Sequential/random Trials Data iterators (kind of sliding time-window) in progress, start approaching the toughest part: non-stationarity battle is ahead.

• 14.11.17: BaseAAC framework refraction; added per worker batch-training option and LSTM time_flatten option; Atari examples updated; see Documentation for details.

• 30.10.17: Major update, some backward incompatibility:

  • BTGym now can be thougt as two-part package: one is environment itself and the other one is RL algoritms tuned for solving algo-trading tasks. Some basic work on shaping of later is done. Three advantage actor-critic style algorithms are implemented: A3C itself, it's UNREAL extension and PPO. Core logic of these seems to be implemented correctly but further extensive BTGym-tuning is ahead. For now one can check atari tests.
  • Finally, basic documentation and API reference is now available.

• 27.09.17: A3C test_4.2 added:

  • some progress on estimator architecture search, state and reward shaping;

• 22.09.17: A3C test_4 added:

  • passing train convergence test on small (1 month) dataset of EURUSD 1-minute bar data;

• 20.09.17: A3C optimised sine-wave test added here.

  • This notebook presents some basic ideas on state presentation, reward shaping, model architecture and hyperparameters choice. With those tweaks sine-wave sanity test is converging faster and with greater stability.

• 31.08.17: Basic implementation of A3C algorithm is done and moved inside BTgym package.

  • algorithm logic consistency tests are passed;
  • still work in early stage, experiments with obs. state features and policy estimator architecture ahead;
  • check out examples/a3c directory.

• 23.08.17: filename arg in environment/dataset specification now can be list of csv files.

  • handy for bigger dataset creation;
  • data from all files are concatenated and sampled uniformly;
  • no record duplication and format consistency checks preformed.

• 21.08.17: UPDATE: BTgym is now using multi-modal observation space.

  • space used is simple extension of gym: DictSpace(gym.Space) - dictionary (not nested yet) of core gym spaces.
  • defined in btgym/spaces.py.
  • raw_state is default Box space of OHLC prices. Subclass BTgymStrategy and override get_state() method to compute alll parts of env. observation.
  • rendering can now be performed for avery entry in observation dictionary as long as it is Box ranked <=3 and same key is passed in reneder_modes kwarg of environment. 'Agent' mode renamed to 'state'. See updated examples.

• 07.08.17: BTgym is now optimized for asynchronous operation with multiply environment instances.

  • dedicated data_server is used for dataset management;
  • improved overall internal network connection stability and error handling;
  • see example async_btgym_workers.ipynb in examples directory.

• 15.07.17: UPDATE, BACKWARD INCOMPATIBILITY: now state observation can be tensor of any rank.

  • Consequently, dim. ordering convention has changed to ensure compatibility with existing tf models: time embedding is first dimension from now on, e.g. state with shape (30, 20, 4) is 30x steps time embedded with 20 features and 4 'channels'. For the sake of 2d visualisation only one 'cannel' can be rendered, can be chosen by setting env. kwarg render_agent_channel=0;
  • examples are updated;
  • better now than later.

• 11.07.17: Rendering battle continues: improved stability while low in memory, added environment kwarg render_enabled=True; when set to False - all renderings are disabled. Can help with performance.

• 5.07.17: Tensorboard monitoring wrapper added; pyplot memory leak fixed.

• 30.06.17: EXAMPLES updated with 'Setting up: full throttle' how-to.

• 29.06.17: UPGRADE: be sure to run pip install --upgrade -e .

  • major rendering rebuild: updated with modes: human, agent, episode; render process now performed by server and returned to environment as rgb numpy array. Pictures can be shown either via matplolib or as pillow.Image(preferred).
  • 'Rendering HowTo' added, 'Basic Settings' example updated.
  • internal changes: env. state divided on raw_state - price data, and state - featurized representation. get_raw_state() method added to strategy.
  • new packages requirements: matplotlib and pillow.

• 25.06.17: Basic rendering implemented.

• 23.06.17: alpha 0.0.4: added skip-frame feature, redefined parameters inheritance logic, refined overall stability;

• 17.06.17: first working alpha v0.0.2.

btgym's Issues

state_shape with only one candle (1,4)

If you think in algo-trader terms, the normal is taking an action every time that you get a new candle, so the Agent should do the same. But btgym isn't prepared for state_shape=(1, 4)
There is a reason for that?

Reward function

I would like to know how is computed the reward function ? Is it based on the open/low price ?
Also, what is the meaning of the action "close" ?

FileNotFoundError

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Manjaro Linux Anaconda 5.1 Python 3.6
Jupyter Notebook

Files or part of package has been run:

atari_a3c

use python *.py to start A3C trainning, i get CreateSession still waiting for response from worker warning, and the master node never response.

hi Andrew,

I got a new issue when try to run A3C algorithm in example a3c_random_on_synth_or_real_data_4_6.ipynb, I run the file in a terminal with cmd: python xxx.py, but I got the following error,have you encountered this problem before?

INFO:Env:Environment is ready.
WARNING:worker_8:AAC_8: learn_rate: 0.000100, entropy_beta: 0.038476
INFO:Env:Server started, pinging tcp://127.0.0.1:5009 ...
DEBUG:Env:Server Control mode: received <{'ctrl': 'ping!'}>
DEBUG:Env:Server sent: {'ctrl': 'send control keys: <_reset>, <_getstat>, <_render>, <_stop>.'}
DEBUG:Env:Server seems ready with response: <{'ctrl': 'send control keys: <_reset>, <_getstat>, <_render>, <_stop>.'}>
INFO:Env:Environment is ready.
WARNING:worker_9:AAC_9: learn_rate: 0.000100, entropy_beta: 0.010318
2017-12-29 02:14:33.860902: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:34.660841: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.352340: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.357352: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.362327: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.363592: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.368915: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.372948: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.377060: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0
2017-12-29 02:14:39.388348: I tensorflow/core/distributed_runtime/master.cc:221] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0

Is it able or nessesary to dockerize this project?

IMO, dockerizing is absolutely helpful to reduce environment issues, and also helpful to distributed computing.

can btgym support different stocks data with the same date time range?

hi Andrew,

Can this framework support use different stocks with the same date time range as input? can you give me an example?

I mean the input for the framework is:

input_file1:
20170901 133000;15345;15355;15340;15335;868
20170901 133100;15335;15340;15330;15330;346
20170901 133200;15330;15330;15305;15300;3088
20170901 133300;15305;15320;15320;15300;1290
20170901 133400;15315;15320;15305;15305;1154
20170901 133500;15310;15315;15315;15300;2882
20170901 133600;15310;15315;15315;15305;434
20170901 133700;15310;15315;15310;15305;506
20170901 133800;15310;15315;15310;15305;906
20170901 133900;15315;15315;15315;15310;308
20170901 134000;15310;15315;15315;15310;180
20170901 134100;15315;15315;15305;15305;418

input_file2:
20170901 133000;4003;4006;4005;4003;2816
20170901 133100;4005;4006;4004;4003;1500
20170901 133200;4004;4004;4001;4000;3306
20170901 133300;4000;4002;4002;3999;2476
20170901 133400;4001;4002;4001;4001;1036
20170901 133500;4001;4001;4000;4000;1758
20170901 133600;4000;4005;4004;3999;5274
20170901 133700;4004;4005;4005;4003;1322
20170901 133800;4004;4005;4005;4004;802
20170901 133900;4004;4006;4006;4004;3030
20170901 134000;4006;4007;4006;4005;2144
20170901 134100;4006;4007;4006;4005;1068
20170901 134200;4006;4006;4005;4004;354
20170901 134300;4005;4005;4004;4004;690
20170901 134400;4004;4005;4004;4004;594
20170901 134500;4005;4006;4005;4004;1904
20170901 134600;4005;4007;4006;4005;1962
20170901 134700;4006;4008;4008;4006;2412
20170901 134800;4008;4010;4009;4008;2226
20170901 134900;4010;4010;4009;4009;1240
20170901 135000;4009;4009;4008;4008;334
20170901 135100;4008;4008;4007;4007;750

Thank you!

Problem to run the code

I have a problem to run the code, after the first epoch, I call the function env.reset() to start a new one but I have the following error :

WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
WARN: Environment '<class 'btgym.envs.backtrader.BTgymEnv'>' has deprecated methods. Compatibility code invoked.
learning-rules.py:320: RuntimeWarning: overflow encountered in multiply
self.states = self.states+ learning_rateestimated_valuetraces
learning-rules.py:320: RuntimeWarning: invalid value encountered in multiply
self.states = self.states+ learning_rateestimated_valuetraces
learning-rules.py:320: RuntimeWarning: invalid value encountered in add
self.states = self.states+ learning_rateestimated_valuetraces
Episode 1 finished
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/lines.py:44: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(style) and is_hashable(style):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/style/core.py:92: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if cbook.is_string_like(style) or hasattr(style, 'keys'):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/style/core.py:99: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if not cbook.is_string_like(style):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/font_manager.py:971: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(family):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/font_manager.py:697: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
if is_string_like(family):
/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/text.py:218: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
elif is_string_like(fontproperties):
Process BTgymServer-2:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 453, in run
episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1127, in run
runstrat = self.runstrategies(iterstrat)
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1295, in runstrategies
self._runnext(runstrats)
File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1626, in _runnext
strat._next()
File "/usr/local/lib/python3.5/dist-packages/backtrader/strategy.py", line 328, in _next
self._next_analyzers(minperstatus)
File "/usr/local/lib/python3.5/dist-packages/backtrader/strategy.py", line 362, in _next_analyzers
analyzer._next()
File "/usr/local/lib/python3.5/dist-packages/backtrader/analyzer.py", line 188, in _next
self.next()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 158, in next
self.early_stop()
File "/home/nicolas/Documents/trading/btgym/btgym/server.py", line 80, in early_stop
self.render.render(self.render_at_stop, step_to_render=self.step_to_render, send_img=False)
File "/home/nicolas/Documents/trading/btgym/btgym/rendering/renderer.py", line 284, in render
xlabel=self.render_xlabel,
File "/home/nicolas/Documents/trading/btgym/btgym/rendering/renderer.py", line 312, in draw_plot
self.plt.title(title)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 1465, in title
return gca().set_title(s, *args, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 950, in gca
return gcf().gca(**kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/figure.py", line 1369, in gca
return self.add_subplot(1, 1, 1, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/figure.py", line 1021, in add_subplot
a = subplot_class_factory(projection_class)(self, *args, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 73, in init
self._axes_class.init(self, fig, self.figbox, **kwargs)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 529, in init
self._init_axis()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 622, in _init_axis
self.xaxis = maxis.XAxis(self)
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 676, in init
self.cla()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 760, in cla
self.reset_ticks()
File "/home/nicolas/.local/lib/python3.5/site-packages/matplotlib/axis.py", line 771, in reset_ticks
cbook.popall(self.majorTicks)
AttributeError: module 'matplotlib.cbook' has no attribute 'popall'
Traceback (most recent call last):
File "learning-rules.py", line 335, in
q.learning(env)
File "learning-rules.py", line 304, in learning
current_state = env.reset()
File "/home/nicolas/Documents/trading/btgym/btgym/envs/backtrader.py", line 595, in _reset
if self._force_control_mode():
File "/home/nicolas/Documents/trading/btgym/btgym/envs/backtrader.py", line 509, in _force_control_mode
self.server_response = self.socket.recv_pyobj()
File "/home/nicolas/.local/lib/python3.5/site-packages/zmq/sugar/socket.py", line 491, in recv_pyobj
msg = self.recv(flags)
File "zmq/backend/cython/socket.pyx", line 693, in zmq.backend.cython.socket.Socket.recv
File "zmq/backend/cython/socket.pyx", line 727, in zmq.backend.cython.socket.Socket.recv
File "zmq/backend/cython/socket.pyx", line 150, in zmq.backend.cython.socket._recv_copy
File "zmq/backend/cython/socket.pyx", line 145, in zmq.backend.cython.socket._recv_copy
File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
zmq.error.Again: Resource temporarily unavailable

Modify Dev_Strat to add technical indicators

@Kismuz First of all, thank you so much for creating this awesome gym. It is probably the best gym for trading that I could find on the web at the moment.

As I have read all the threads from other issues and also run the A3C Random notebook that you provided, the reward did not converge over time. I saw you have mentioned that as of now, you have not found any good solution that could lead to the convergence. I have spent sometime with the package, especially with the notebooks examples, and just thought maybe if we provided more information to the agent like technical indicators or the like, it would help improve the performance. I have tried to modify the Dev_Strat_4_6 by adding two simple moving average (50, 100) but keep getting the State observation shape/range mismatch, which I believe is due to the NAN problem since there is no data for the second indicator for the first 100 records. I could fix the problem by set the skip_frame to 100 but this is not a very data-efficient solution. Could you please help instruct me how to tackle this problem? Thank you in advance.

RL example strategy

Hi, thanks for another new great gym environment!

Its not actual issue, more like question:
I am curious about actual RL trained strategies examples (not necessary profitable), because I didn't find any and OpenAI didn't pay much attention to financial gyms. It will be much easier to learn on existed examples how tune architecture or parameters. I would be very grateful if you point me out useful links if you know some. Thanks

The effect of using a GPU does not seem to be good.

Very great project, thank you for your pay.
In the example of unreal_stacked_lstm_strat_4_11, is it faster to use GPU computing than CPU?
The effect of using a GPU does not seem to be good.

Question Performance

Hello first of all thank you for your contribution.
I would like to know what is your opinion concerning using A3C to predict stock market and also for a certain equity what kind of features should be taken into account? High , low, open close ?

Thank you
João Salvado

Get better signal strengths by tuning scaling hyperparameters

Hi @Kismuz
I need your help to clarify a few questions:

1. Currently, Unreal Stacked LSTM Strat 4_11 has better performance on actual FOREX prices than A3C Random, so just out of curiosity, I tried to run both of them with the sine wave data (test_sine_1min_period256_delta0002.csv). It turns out that A3C eventually converges while Unreal Stacked LSTM is nowhere near the convergence. Could you provide me some insights on that?

LSTM results:

A3C results:

2. I had an overflow issue with the tanh(x_sma) while feeding my own data into the gym (SPY, Dow Jones, Russel 2000, and so on). After changing the T2 value from 2000 to 1, the issue went away but I am not sure if that was a proper fix. Could you help me shed some lights on that?

3. I also got data consistency issue when passing equity data traded from 8:30am to 15:15pm instead of 24/7 like FOREX data that you used as examples. What should I do to fix this?

4. I kept getting the "[WinError 2] The system cannot find the file specified" when trying to run the A3C example in Windows.
   

5. When I tried to run the A3C with the new version of tensorflow supporting GPU in Windows, I I got the error below also:
   

Really appreciate your effort to build this awesome gym and great documentation.

Thank you in advance.

BTgymEnv with state_shape parameter, incorrect type

When using the parameter state_shape to create BTgymEnv, it will throw AttributeError: 'tuple' object has no attribute 'keys'

Running environment:

python 3, macosx

from btgym import BTgymEnv

MyEnvironment = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
episode_duration={'days': 2, 'hours': 23, 'minutes': 55},
drawdown_call=50,
state_shape=(4,20),
port=5555,
verbose=1,
)

Files or part of package has been run:

backtrader.py

Actual behaviour:

[2018-04-19 01:04:56.697574] INFO: BTgymAPIshell_0: ...done.
Traceback (most recent call last):
[2018-04-19 01:04:56.698795] INFO: BTgymAPIshell_0: Base Cerebro class used. Base Strategy class used.
File "/btgym/tests/quickstart.py", line 8, in
verbose=1,
File "/btgym/btgym/envs/backtrader.py", line 353, in init
if 'raw_state' in self.params['strategy']['state_shape'].keys():
AttributeError: 'tuple' object has no attribute 'keys'

AttributeError: 'tuple' object has no attribute 'keys'

When I ran the following example code：

from btgym import BTgymEnv
MyEnvironment = BTgymEnv(filename='/home/Code/btgym-master/examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
episode_duration={'days': 2, 'hours': 23, 'minutes': 55},
drawdown_call=50,
state_shape=(4,20),
port=5555,
verbose=1,
)

[@-pc btgym-master]$ cd /home/Code/btgym-master ; env "PYTHONIOENCODING=UTF-8" "PYTHONUNBUFFERED=1" python /home/.vscode/extensions/ms-python.python-2018.3.1/pythonFiles/PythonTools/visualstudio_py_launcher.py /home/Code/btgym-master 33543 34806ad9-833a-4524-8cd6-18ca4aa74f14 RedirectOutput,RedirectOutput /home/Code/btgym-master/examples/test.py
[2018-04-02 03:29:58.845135] INFO: BTgymAPIshell_0: Base Dataset class used.
[2018-04-02 03:29:58.845766] INFO: BTgymAPIshell_0: Connecting data_server...
[2018-04-02 03:29:58.936939] INFO: BTgymDataServer_0: PID: 6136
[2018-04-02 03:29:59.969214] INFO: SimpleDataSet_0: Loaded 372678 records from </home//Code/btgym-master/examples/data/DAT_ASCII_EURUSD_M1_2016.csv>.
[2018-04-02 03:30:00.145812] INFO: SimpleDataSet_0: Data summary:
open high low close volume
count 372678.000000 372678.000000 372678.000000 372678.000000 372678.0
mean 1.107109 1.107198 1.107019 1.107108 0.0
std 0.024843 0.024840 0.024847 0.024844 0.0
min 1.035250 1.035470 1.035220 1.035220 0.0
25% 1.092140 1.092230 1.092040 1.092140 0.0
50% 1.113530 1.113610 1.113450 1.113530 0.0
75% 1.124710 1.124780 1.124630 1.124710 0.0
max 1.161440 1.161600 1.160770 1.161450 0.0
[2018-04-02 03:30:00.152136] INFO: BTgymAPIshell_0: ...done.
[2018-04-02 03:30:00.154973] INFO: BTgymAPIshell_0: Base Cerebro class used. Base Strategy class used.
Traceback (most recent call last):
File "/home/liguodong/Code/btgym-master/examples/test.py", line 28, in
verbose=1,
File "/home/liguodong/Code/btgym-master/btgym/envs/backtrader.py", line 349, in init
if 'raw_state' in self.params['strategy']['state_shape'].keys():
AttributeError: 'tuple' object has no attribute 'keys'

When is the Actual REAL-TIME trading tests going to happen ?

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Files or part of package has been run:

Expected behaviour:

Actual behaviour:

Steps to reproduce:

running setting_up_environment_full raise an error

Hi, it is so nice that you wrote these two setting up notebook, it helps a lot, but when I was running the example/setting_up_environment_full，I got these error messages, could you help me out?

env.reset()
take_some_steps(env, 100)
render_all_modes(env)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
~/projects/btgym/btgym/envs/backtrader.py in _reset(self, state_only)
    606             try:
--> 607                 assert self.observation_space.contains(self.env_response[0])
    608 

AssertionError: 

During handling of the above exception, another exception occurred:

AssertionError                            Traceback (most recent call last)
<ipython-input-10-af63753eff47> in <module>()
----> 1 env.reset()
      2 take_some_steps(env, 100)
      3 render_all_modes(env)

~/.pyenv/versions/miniconda3-4.3.11/lib/python3.6/site-packages/gym/core.py in reset(self)
    102             space.
    103         """
--> 104         return self._reset()
    105 
    106     def render(self, mode='human', close=False):

~/projects/btgym/btgym/envs/backtrader.py in _reset(self, state_only)
    632                 self.log.error(msg)
    633                 self._stop_server()
--> 634                 raise AssertionError(msg)
    635 
    636             if state_only:

AssertionError: 
State observation shape/range mismatch!
Space set by env: 

price_gradients:
   Box(30, 4), low: 0.0, high: 1.0

raw_state:
   Box(30, 4), low: 1.03522, high: 1.1616

Space returned by server: 

raw_state:
   array of shape: (4, 4), low: 1.11363, high: 1.11372

price_gradients:
   array of shape: (4, 4), low: 0.3965167501352529, high: 0.6726070170682124

Full response:
{'raw_state': array([[ 1.11372,  1.11372,  1.1137 ,  1.11371],
       [ 1.1137 ,  1.1137 ,  1.11369,  1.11369],
       [ 1.11369,  1.11371,  1.11363,  1.11371],
       [ 1.1137 ,  1.11372,  1.11369,  1.11372]]), 'price_gradients': array([[ 0.44028635,  0.44028635,  0.47003595,  0.44028635],
       [ 0.45512111,  0.4850045 ,  0.39651675,  0.5       ],
       [ 0.5       ,  0.52996405,  0.5       ,  0.54487889],
       [ 0.52996405,  0.52996405,  0.67260702,  0.52996405]])}
Reward: 0.0
Done: False
Info:
{'step': 5, 'time': datetime.datetime(2016, 7, 4, 0, 7), 'action': 'hold', 'broker_message': '-', 'broker_cash': 100.0, 'broker_value': 100.0, 'drawdown': 0.0, 'max_drawdown': 0.0}

Hint: Wrong Strategy.get_state() parameters?

Problem with observation_space.shape

Hi,

Finally got around to trying a real "deep learning" implementation against btgym and I've run up against a problem. I really don't know enough about openai gym to understand what the problem is fully but it seems to me based on the implementation I am trying and observing other implementations that observation_space.shape returns inconsistent values.

I tried modifying this basic A3C implementation: https://github.com/jaara/AI-blog/blob/master/CartPole-A3C.py

I've attached my attempt below but when I run it I get the error:

ValueError: Error when checking : expected input_1 to have 2 dimensions, but got array with shape (1, 4, 30)

When I print observation_space.shape it seems to change all the time which is expected but I believe it should always follow the same format (again I'm a bit of a newbie with this stuff so I could be wrong).

Are you able to take a look? I'll keep digging and try another implementation - I'm thinking a DQN implementation. The problem with most of the implementations for DQN or A3C is that they rely on the state being implemented as a screenshot so an array of rows and columns sometimes represented as RGB or often just 0 or 1.

Keep up the good work BTW.

# OpenGym CartPole-v0 with A3C on GPU
# -----------------------------------
#
# A3C implementation with GPU optimizer threads.
# 
# Made as part of blog series Let's make an A3C, available at
# https://jaromiru.com/2017/02/16/lets-make-an-a3c-theory/
#
# author: Jaromir Janisch, 2017

import numpy as np
import tensorflow as tf

import gym, time, random, threading
from btgym import BTgymEnv


from keras.models import *
from keras.layers import *
from keras import backend as K

#-- constants
ENV = 'backtrader-v46'

RUN_TIME = 30
THREADS = 8
OPTIMIZERS = 2
THREAD_DELAY = 0.001

GAMMA = 0.99

N_STEP_RETURN = 8
GAMMA_N = GAMMA ** N_STEP_RETURN

EPS_START = 0.4
EPS_STOP  = .15
EPS_STEPS = 75000

MIN_BATCH = 32
LEARNING_RATE = 5e-3

LOSS_V = .5			# v loss coefficient
LOSS_ENTROPY = .01 	# entropy coefficient

#---------
class Brain:
	train_queue = [ [], [], [], [], [] ]	# s, a, r, s', s' terminal mask
	lock_queue = threading.Lock()

	def __init__(self):
		self.session = tf.Session()
		K.set_session(self.session)
		K.manual_variable_initialization(True)

		self.model = self._build_model()
		self.graph = self._build_graph(self.model)

		self.session.run(tf.global_variables_initializer())
		self.default_graph = tf.get_default_graph()

		self.default_graph.finalize()	# avoid modifications

	def _build_model(self):

		l_input = Input( batch_shape=(None, NUM_STATE) )
		l_dense = Dense(16, activation='relu')(l_input)

		out_actions = Dense(NUM_ACTIONS, activation='softmax')(l_dense)
		out_value   = Dense(1, activation='linear')(l_dense)

		model = Model(inputs=[l_input], outputs=[out_actions, out_value])
		model._make_predict_function()	# have to initialize before threading

		return model

	def _build_graph(self, model):
		s_t = tf.placeholder(tf.float32, shape=(None, NUM_STATE))
		a_t = tf.placeholder(tf.float32, shape=(None, NUM_ACTIONS))
		r_t = tf.placeholder(tf.float32, shape=(None, 1)) # not immediate, but discounted n step reward
		
		p, v = model(s_t)

		log_prob = tf.log( tf.reduce_sum(p * a_t, axis=1, keep_dims=True) + 1e-10)
		advantage = r_t - v

		loss_policy = - log_prob * tf.stop_gradient(advantage)									# maximize policy
		loss_value  = LOSS_V * tf.square(advantage)												# minimize value error
		entropy = LOSS_ENTROPY * tf.reduce_sum(p * tf.log(p + 1e-10), axis=1, keep_dims=True)	# maximize entropy (regularization)

		loss_total = tf.reduce_mean(loss_policy + loss_value + entropy)

		optimizer = tf.train.RMSPropOptimizer(LEARNING_RATE, decay=.99)
		minimize = optimizer.minimize(loss_total)

		return s_t, a_t, r_t, minimize

	def optimize(self):
		if len(self.train_queue[0]) < MIN_BATCH:
			time.sleep(0)	# yield
			return

		with self.lock_queue:
			if len(self.train_queue[0]) < MIN_BATCH:	# more thread could have passed without lock
				return 									# we can't yield inside lock

			s, a, r, s_, s_mask = self.train_queue
			self.train_queue = [ [], [], [], [], [] ]

		s = np.vstack(s)
		a = np.vstack(a)
		r = np.vstack(r)
		s_ = np.vstack(s_)
		s_mask = np.vstack(s_mask)

		if len(s) > 5*MIN_BATCH: print("Optimizer alert! Minimizing batch of %d" % len(s))

		v = self.predict_v(s_)
		r = r + GAMMA_N * v * s_mask	# set v to 0 where s_ is terminal state
		
		s_t, a_t, r_t, minimize = self.graph
		self.session.run(minimize, feed_dict={s_t: s, a_t: a, r_t: r})

	def train_push(self, s, a, r, s_):
		with self.lock_queue:
			self.train_queue[0].append(s)
			self.train_queue[1].append(a)
			self.train_queue[2].append(r)

			if s_ is None:
				self.train_queue[3].append(NONE_STATE)
				self.train_queue[4].append(0.)
			else:	
				self.train_queue[3].append(s_)
				self.train_queue[4].append(1.)

	def predict(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)
			return p, v

	def predict_p(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)		
			return p

	def predict_v(self, s):
		with self.default_graph.as_default():
			p, v = self.model.predict(s)		
			return v

#---------
frames = 0
class Agent:
	def __init__(self, eps_start, eps_end, eps_steps):
		self.eps_start = eps_start
		self.eps_end   = eps_end
		self.eps_steps = eps_steps

		self.memory = []	# used for n_step return
		self.R = 0.

	def getEpsilon(self):
		if(frames >= self.eps_steps):
			return self.eps_end
		else:
			return self.eps_start + frames * (self.eps_end - self.eps_start) / self.eps_steps	# linearly interpolate

	def act(self, s):
		eps = self.getEpsilon()			
		global frames; frames = frames + 1

		if random.random() < eps:
			return random.randint(0, NUM_ACTIONS-1)

		else:
			s = np.array([s])
			p = brain.predict_p(s)[0]

			# a = np.argmax(p)
			a = np.random.choice(NUM_ACTIONS, p=p)

			return a
	
	def train(self, s, a, r, s_):
		def get_sample(memory, n):
			s, a, _, _  = memory[0]
			_, _, _, s_ = memory[n-1]

			return s, a, self.R, s_

		a_cats = np.zeros(NUM_ACTIONS)	# turn action into one-hot representation
		a_cats[a] = 1 

		self.memory.append( (s, a_cats, r, s_) )

		self.R = ( self.R + r * GAMMA_N ) / GAMMA

		if s_ is None:
			while len(self.memory) > 0:
				n = len(self.memory)
				s, a, r, s_ = get_sample(self.memory, n)
				brain.train_push(s, a, r, s_)

				self.R = ( self.R - self.memory[0][2] ) / GAMMA
				self.memory.pop(0)		

			self.R = 0

		if len(self.memory) >= N_STEP_RETURN:
			s, a, r, s_ = get_sample(self.memory, N_STEP_RETURN)
			brain.train_push(s, a, r, s_)

			self.R = self.R - self.memory[0][2]
			self.memory.pop(0)	
	
	# possible edge case - if an episode ends in <N steps, the computation is incorrect
		
#---------
class Environment(threading.Thread):
	stop_signal = False

	def __init__(self, instancenumber=0, render=False, eps_start=EPS_START, eps_end=EPS_STOP, eps_steps=EPS_STEPS):
		threading.Thread.__init__(self)

		self.render = render

		#-- create btgym environment
		self.env = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',
		                  start_weekdays=[0, 1, 2],
		                  episode_len_days=1,
		                  episode_len_hours=23,
		                  episode_len_minutes=55,
		                  start_00=True,
		                  start_cash=100,
		                  broker_commission=0.002,
		                  fixed_stake=10,
		                  drawdown_call=30,
		                  state_shape=(4,30),
		                  state_low=None,
		                  state_high=None,
		                  port=5555 + instancenumber,
		                  verbose=1,)

		self.agent = Agent(eps_start, eps_end, eps_steps)

	def runEpisode(self):
		s = self.env.reset()
		print(s)

		R = 0
		while True:         
			time.sleep(THREAD_DELAY) # yield 

			if self.render: self.env.render()

			a = self.agent.act(s)
			s_, r, done, info = self.env.step(a)

			if done: # terminal state
				s_ = None

			self.agent.train(s, a, r, s_)

			s = s_
			R += r

			if done or self.stop_signal:
				break

		print("Total R:", R)

	def run(self):
		while not self.stop_signal:
			self.runEpisode()

	def stop(self):
		self.stop_signal = True

#---------
class Optimizer(threading.Thread):
	stop_signal = False

	def __init__(self):
		threading.Thread.__init__(self)

	def run(self):
		while not self.stop_signal:
			brain.optimize()

	def stop(self):
		self.stop_signal = True




#-- main
env_test = Environment(render=True, instancenumber=0, eps_start=0., eps_end=0.)
print(env_test.env.observation_space.shape[0])
NUM_STATE = env_test.env.observation_space.shape[0]
NUM_ACTIONS = env_test.env.action_space.n
NONE_STATE = np.zeros(NUM_STATE)

brain = Brain()	# brain is global in A3C

envs = [Environment(instancenumber=i) for i in range(THREADS)]
opts = [Optimizer() for i in range(OPTIMIZERS)]

for o in opts:
	o.start()

for e in envs:
	e.start()

time.sleep(RUN_TIME)

for e in envs:
	e.stop()
for e in envs:
	e.join()

for o in opts:
	o.stop()
for o in opts:
	o.join()

print("Training finished")
env_test.run()

_

Other Timeframes

I know a current limitation is accept Forex 1 min (only Forex?), but my datasets are with bigger timeframes.

This is the stacktrace when timeframe is changed:

Process BTgymServer-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()

  File "/home/adrian/btgym/btgym/server.py", line 405, in run
    episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1142, in run
    runstrat = self.runstrategies(iterstrat)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1327, in runstrategies
    self.stop_writers(runstrats)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/cerebro.py", line 1352, in stop_writers
    datainfos['Data%d' % i] = data.getwriterinfo()

  File "/usr/local/lib/python3.5/dist-packages/backtrader/dataseries.py", line 101, in getwriterinfo
    info['Timeframe'] = TimeFrame.TName(self._timeframe)

  File "/usr/local/lib/python3.5/dist-packages/backtrader/dataseries.py", line 57, in TName
    return cls.Names[tframe]
IndexError: list index out of range

Any idea?

zmq.error.Again: Resource temporarily unavailable

While runing the examples, I'm getting error like below

INFO:tensorflow:Starting queue runners. WARNING:worker_1:worker_1: started training at step: 120 Exception in thread Thread-4: Traceback (most recent call last): File "/home/humin/anaconda3/envs/tensorflow/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/home/humin/btgym/btgym/algorithms/runner.py", line 75, in run self._run() File "/home/humin/btgym/btgym/algorithms/runner.py", line 96, in _run self.queue.put(next(rollout_provider), timeout=600.0) File "/home/humin/btgym/btgym/algorithms/runner.py", line 238, in env_runner episode_stat = env.get_stat() # get episode statistic File "/home/humin/btgym/btgym/envs/backtrader.py", line 680, in get_stat if self._force_control_mode(): File "/home/humin/btgym/btgym/envs/backtrader.py", line 508, in _force_control_mode self.server_response = self.socket.recv_pyobj() File "/home/humin/anaconda3/envs/tensorflow/lib/python3.5/site-packages/zmq/sugar/socket.py", line 491, in recv_pyobj msg = self.recv(flags) File "zmq/backend/cython/socket.pyx", line 693, in zmq.backend.cython.socket.Socket.recv File "zmq/backend/cython/socket.pyx", line 727, in zmq.backend.cython.socket.Socket.recv File "zmq/backend/cython/socket.pyx", line 150, in zmq.backend.cython.socket._recv_copy File "zmq/backend/cython/socket.pyx", line 145, in zmq.backend.cython.socket._recv_copy File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc zmq.error.Again: Resource temporarily unavailable
I tried to reduce the number of workers but it looks irrelevant. This error is not interrupting the training process. Is this normal or should I be concerned about it?
My environment: Ubuntu 16.04. Python 3.5

Can I use this gym environment to get a portfolio solution for n assets?

I found that the default action space is ['hold', 'close', 'buy', 'sell']. so how can I invest n assets with different portfolio weights?

Reward always 0

Hello @Kismuz
There is something I am experiencing with the environment where as at each step when taking an action on the environment the reward is always 0. I am not sure this is a bug or I am just missing something

Here is my initialization code

env = BTgymEnv(
            filename='./btgym/examples/data/DAT_ASCII_EURUSD_M1_2010.csv',
            start_weekdays=[0, 1],
            episode_duration={'days': 2, 'hours': 0, 'minutes': 0},
            strategy=MyStrategy,
            start_00=True,
            start_cash=self.capital,
            broker_commission=0.002,
            fixed_stake=10,
            drawdown_call=30,
            state_shape={'raw_state': spaces.Box(low=1, high=2, shape=self.input_shape),'indicator_states': spaces.Box(low=-1, high=100, shape=self.input_shape)},
            port=5002,
            data_port=4800,
            verbose=1,)

Here is how it is used

done = False
    while not done:
        action = agent.act(state)
        next_state, reward, done, info = env.step(action) #The reward here is always 0 regardless of the action I take
        agent.remember(state, action, reward, next_state, done)
        state = next_state

Expected behaviour:

I expect the reward to be varying in that it should be negative or positive or zero occationally.

Actual behaviour:

The reward is always 0 even when I put all the actions within one episode to be a single one.

Please help me out here

Thanks in advance.

News and updates: Slack channel added

vvv Scroll down for latest updates vvv

• 6.02.18: Pushed common update to all a3c agents architectures:
  • all dense layers are now Noisy-Net ones,
    see: Noisy Networks for Exploration paper by Fortunato at al.;
    • note that entropy regularization is still here, lowered to ~0.01 to ensure proper exploration;
      update your entropy_beta custom settings, if done any;
  • policy output distribution is 'centered' using layer normalisation technique;
  • added per-channel weights scaling to obs state before passing to tanh, see #35 and https://github.com/Kismuz/btgym/blob/master/examples/state_signal_scaling.ipynb

All of the above results in about 2x training speedup in terms of train iterations: stacked LSTM agents takes ~20M steps to converge on 6 month real data set (was: 40).

Sell action when there isn't any holding assets

First of all, thank you for this project, I really believe it has a great potential and also let me tell that I'm nothing but a newbie so excuse my question if it is a dumb one.

When I configure and start running, during some epocs it keeps giving SELL actions although there is not any holding assets. 2 questions:

• Why is the reward decreases when it sells even though there isn't anything to sell
• Can we change that in a way that the action list is based on current holdings so it doesn't try to buy when there isn't any money left or sell when there isn't anything to sell

Thanks again

Making predictions

Hi Kizmuz, First of all congratulations for this really great job!

OK, I have trained my workers, and here you are some questions:

Do you have some code to test predictions with their model?
Do you have some code to extract weights from the checkpoint file and deploy the model? If not, could you provide with some guideline instructions to do that?
How do you plan to implement the production test?

Thank you in advance.

Errors running examples in Jupyter notebook

Hi again,

I'm still having problems running the basic examples. I noticed you're running python 3 judging on the leftover pyc files. I'm trying with 2.7.11, latest dependencies. It could be a Python 3 vs 2.7 thing but I can't see why, to me it looks like you're not including backtrader when referencing it (but I can see it is included). Somehow it works for you but not me. Example error below:

from btgym import BTgymEnv
​

Simpliest trading environment,

using year-long dataset of one minute bars for EUR/USD currency pair:

​

MyEnvironment = BTgymEnv(filename='./data/DAT_ASCII_EURUSD_M1_2016.csv',)
​

Print parameters and their default values:

​
for params_name, params_dict in MyEnvironment.params.items():
print('\nParameters [{}] subset: '.format(params_name))
for key, value in params_dict.items():
print('{} : {}'.format(key,value))

---

AttributeError Traceback (most recent call last)
in ()
5
6
----> 7 MyEnvironment = BTgymEnv(filename='./data/DAT_ASCII_EURUSD_M1_2016.csv',)
8
9 # Print parameters and their default values:

btgym/envs/backtrader.pyc in init(self, **kwargs)
214 # if no bt.Cerebro() custom subclass has been passed,
215 # get base class Cerebro(), using kwargs on top of defaults:
--> 216 self.engine = bt.Cerebro()
217 self.params['engine'].update(self.kwargs['engine'])
218 msg = 'Base Cerebro engine used.'

AttributeError: 'module' object has no attribute 'Cerebro'

I then try my basic example from my previous issue I get the following error:

Traceback (most recent call last):
File "test.py", line 27, in
verbose=1)
File "btgym/envs/backtrader.py", line 261, in init
if self.engine.strats[0][0][2]['state_low'] is None or
KeyError: 'state_low'

Any clues? I'm pulling my hair out as I can initialise bt separately in Jupyter and I get the same errors regardless of jupyter or command line.

Problem when run tensorboard_monitor.ipynb

I run tensorborad_monitor.ipynb and found this problem below: Please help

UnsupportedMode Traceback (most recent call last)
in ()
79 images_monitor.write(
80 feed_dict={'human': env.render('human')[None,:],
---> 81 'agent': env.render('agent')[None,:],
82 },
83 global_step=global_step + last_i['step'],

~/.local/lib/python3.5/site-packages/gym/core.py in render(self, mode, close)
147 raise error.UnsupportedMode('{} does not support rendering (requested mode: {})'.format(self, mode))
148 elif mode not in modes:
--> 149 raise error.UnsupportedMode('Unsupported rendering mode: {}. (Supported modes for {}: {})'.format(mode, self, modes))
150 return self._render(mode=mode, close=close)
151

UnsupportedMode: Unsupported rendering mode: agent. (Supported modes for : ['human', 'episode'])

BTgymStrategy not defined in Quickstart-Maximum environment flexibility

In the documentation (https://kismuz.github.io/btgym/intro.html#quickstart), Maximum environment flexibility section, MyCerebro.addstrategy(BTgymStrategy,....

However, BTgymStrategy is not defined here. Do we need to define it first?

how to config the code run with GPU?

hi Andrew,
Thanks for your great work! I wonder if I can run this code on my GPU env. I've tried modify the code 'cpu:0' to 'GPU:0' but got error saying that resource are not available.

Could you pls tell me what's the correct operation?

Tom

setting_up_environment_basic fails on python 3.6, Windows 7

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-914694f81f03> in <module>()
      3 
      4 
----> 5 MyEnvironment = BTgymEnv(filename='./data/DAT_ASCII_EURUSD_M1_2016.csv',)
      6 
      7 # Print environment configuration:

c:\users\e\github\btgym-master\btgym\envs\backtrader.py in __init__(self, *args, **kwargs)
    246         # Connect/Start data server (and get dataset statistic):
    247         self.log.info('Connecting data_server...')
--> 248         self._start_data_server()
    249         self.log.info('...done.')
    250         # ENGINE preparation:

c:\users\e\github\btgym-master\btgym\envs\backtrader.py in _start_data_server(self)
    736             )
    737             self.data_server.daemon = False
--> 738             self.data_server.start()
    739             # Wait for server to startup
    740             time.sleep(1)

C:\ProgramData\Anaconda3\lib\multiprocessing\process.py in start(self)
    103                'daemonic processes are not allowed to have children'
    104         _cleanup()
--> 105         self._popen = self._Popen(self)
    106         self._sentinel = self._popen.sentinel
    107         _children.add(self)

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    221     @staticmethod
    222     def _Popen(process_obj):
--> 223         return _default_context.get_context().Process._Popen(process_obj)
    224 
    225 class DefaultContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    320         def _Popen(process_obj):
    321             from .popen_spawn_win32 import Popen
--> 322             return Popen(process_obj)
    323 
    324     class SpawnContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
     63             try:
     64                 reduction.dump(prep_data, to_child)
---> 65                 reduction.dump(process_obj, to_child)
     66             finally:
     67                 set_spawning_popen(None)

C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)
     61 
     62 #

TypeError: can't pickle _thread.RLock objects

When running UNREAL example

Hello,

When I run the UNREAL example I got the following output.

/home/jsalvado/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
</home/jsalvado/tmp/test_gym_unreal> already exists. Override[y/n]? y
WARNING:Launcher:Files in </home/jsalvado/tmp/test_gym_unreal> purged.
2017-11-27 16:52:54.666375: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:54.670453114 18319 ev_epoll1_linux.c:1051] grpc epoll fd: 7
2017-11-27 16:52:54.671044: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:54.671596938 18320 ev_epoll1_linux.c:1051] grpc epoll fd: 8
2017-11-27 16:52:54.676864: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:12230}
2017-11-27 16:52:54.676891: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> 127.0.0.1:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:54.677761: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:54.677801: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12231, 1 -> 127.0.0.1:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:54.677844: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12230
2017-11-27 16:52:54.679672: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12231
Press Ctrl-C or [Kernel]->[Interrupt] to stop training and close launcher.
2017-11-27 16:52:59.683070: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:59.683829609 18359 ev_epoll1_linux.c:1051] grpc epoll fd: 9
2017-11-27 16:52:59.686654: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1127 16:52:59.687214727 18360 ev_epoll1_linux.c:1051] grpc epoll fd: 10
2017-11-27 16:52:59.689904: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:59.689941: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> localhost:12232, 2 -> 127.0.0.1:12233}
2017-11-27 16:52:59.690832: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12232
2017-11-27 16:52:59.693367: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> 127.0.0.1:12230}
2017-11-27 16:52:59.693405: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> 127.0.0.1:12231, 1 -> 127.0.0.1:12232, 2 -> localhost:12233}
2017-11-27 16:52:59.694368: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:12233
2017-11-27 16:53:04.660194: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session f67f491b8dcaa755 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:0/cpu:0" inter_op_parallelism_threads: 2
2017-11-27 16:53:09.148756: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session dd584888bb6349a4 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:1/cpu:0" inter_op_parallelism_threads: 2
2017-11-27 16:53:09.294430: I tensorflow/core/distributed_runtime/master_session.cc:1004] Start master session 785d00122230e0e1 with config: intra_op_parallelism_threads: 1 device_filters: "/job:ps" device_filters: "/job:worker/task:2/cpu:0" inter_op_parallelism_threads: 2
WARNING:worker_1:worker_1: started training at step: 0
WARNING:worker_2:worker_2: started training at step: 0
WARNING:worker_0:worker_0: started training at step: 0
WARNING:Env:Data_master reset() called prior to reset_data() with [possibly inconsistent] defaults.
WARNING:Env:Dataset not ready, waiting time left: 298 sec.
WARNING:Env:Dataset not ready, waiting time left: 298 sec.

Do you know what can be done to make it work?
Thank you very much.

João Salvado

Is there a roadmap for using cryptocurrency trading data?

error about queue.Empty

Thanks for great work :)

I have an issue while running examples - a3c_random_on_synth_or_real_data ...

I got several <INFO:tensorflow:Error reported to Coordinator: <class 'queue.Empty'>> messages and then stopped.

Is there anyway I can fix it ?? Thank you so much.
Kim.

---

[2018-01-11 20:50:20,439] Error reported to Coordinator: <class 'queue.Empty'>,
Process Worker-6:
Traceback (most recent call last):
File "/home/joowonkim/anaconda3/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/worker.py", line 241, in run
trainer.process(sess)
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 747, in process
data = self.get_data()
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 594, in get_data
data_streams = [get_it() for get_it in self.data_getter]
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/aac.py", line 594, in
data_streams = [get_it() for get_it in self.data_getter]
File "/home/joowonkim/바탕화면/git/btgym/btgym/algorithms/rollout.py", line 33, in pull_rollout_from_queue
return queue.get(timeout=600.0)
File "/home/joowonkim/anaconda3/lib/python3.5/queue.py", line 172, in get
raise Empty
queue.Empty

INFO:tensorflow:global/global_step/sec: 0

[2018-01-11 20:51:38,860] global/global_step/sec: 0

INFO:tensorflow:Error reported to Coordinator: <class 'queue.Empty'>,

[2018-01-11 20:51:48,678] Error reported to Coordinator: <class 'queue.Empty'>,

---

and stopped

Please help

Dear Sir

Could you please help me fix this problem ?

I tried running a3c_sinewave but when i run the last cell (launcher.run())

I found this problem.
[2017-09-04 19:24:45,266] Data file <../examples/data/test_sine_1min_period256_delta0002.csv> not specified / not found.
Process BTgymDataFeedServer-2:1:
Traceback (most recent call last):
File "/home/balldekdee/btgym/btgym/datafeed.py", line 127, in read_csv
assert filename and os.path.isfile(filename)
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/balldekdee/btgym/btgym/datafeed.py", line 142, in read_csv
assert 'episode_dataset' in self.filename
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/balldekdee/btgym/btgym/dataserver.py", line 73, in run
self.dataset.read_csv()
File "/home/balldekdee/btgym/btgym/datafeed.py", line 149, in read_csv
raise FileNotFoundError(msg)
FileNotFoundError: Data file <../examples/data/test_sine_1min_period256_delta0002.csv> not specified / not found.
[2017-09-04 19:24:50,508] Press Ctrl-C to stop training and close launcher.

BTgymEnv init with error 'Dict' object has no attribute 'shape'

Andrew, first thank you for your work! In the last commit initialization env crashes with an error. What could be the problem? Thanks in advance.

MyCerebro = bt.Cerebro()

MyCerebro.addstrategy(
    DevStrat_4_6,
    drawdown_call=5, # max % to loose, in percent of initial cash
    target_call=10,  # max % to win, same
    skip_frame=10,
)

MyDataset = BTgymDataset(
    #filename='.data/DAT_ASCII_EURUSD_M1_201703.csv',
    #filename='./data/DAT_ASCII_EURUSD_M1_201704.csv',
    filename='./data/test_sine_1min_period256_delta0002.csv',
    start_weekdays={0, 1, 2, 3},
    episode_duration={'days': 0, 'hours': 23, 'minutes': 55},
    start_00=False,
    time_gap={'hours': 6},
)

kwargs=dict(
         dataset=MyDataset,
         engine=MyCerebro,
         render_modes=['episode', 'human','external'],
         render_state_as_image=True,
         render_ylabel='OHL_diff.',
         render_size_episode=(12,8),
         render_size_human=(9, 4),
         render_size_state=(11, 3),
         render_dpi=75,
         port=5000,
         data_port=4999,
         connect_timeout=60,
         verbose=0,  # better be 0
     )
env = BTgymEnv(**kwargs)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-2abf37cd704a> in <module>()
----> 1 env = BTgymEnv(**kwargs)

/var/www/btgym/btgym/envs/backtrader.py in __init__(self, *args, **kwargs)
    334 
    335         # Set observation space shape from engine/strategy parameters:
--> 336         self.observation_space = DictSpace(self.params['strategy']['state_shape'])
    337 
    338         self.log.debug('Obs. shape: {}'.format(self.observation_space.spaces))

/var/www/btgym/btgym/spaces.py in __init__(self, spaces)
     37         """
     38         super(DictSpace, self).__init__(spaces)
---> 39         self.shape = self._get_shape()
     40 
     41     def _get_shape(self):

/var/www/btgym/btgym/spaces.py in _get_shape(self)
     40 
     41     def _get_shape(self):
---> 42         return OrderedDict([(k, space.shape) for k, space in self.spaces.items()])
     43 
     44 

/var/www/btgym/btgym/spaces.py in <listcomp>(.0)
     40 
     41     def _get_shape(self):
---> 42         return OrderedDict([(k, space.shape) for k, space in self.spaces.items()])
     43 
     44 

AttributeError: 'Dict' object has no attribute 'shape'

Paper Trading testing on IB using Backtrader

Is it possible to do Testing on Interactive Brokers (IB) as is ? Just to test with actual trading environment without using actual funds. Backtrader uses IB or OANDA for trading and testing

Testing Environment on New Data (Cryptocurrency), getting InvalidIndexError

Hi, I want to thank you for all that you've been doing for supporting backtrader with a custom OpenAI Gym Environment, it really has helped me a lot.

I'm having difficulties adapting the environment to a new dataset, which is for Bitcoin-USD currency.

MyEnvironment = BTgymEnv(filename='btc_usd.csv')

Here I have added the first 10 lines of my btc_usd.csv, for which I've maintained the same formatting as found in your example csv. I've added volume values to each row as well:

20170911 194700;4209.48;4209.98;4209.48;4209.98;3.0283
20170911 194600;4204.65;4209.49;4204.64;4204.64;1.4119397600000003
20170911 194500;4204.64;4204.65;4204.64;4204.65;7.74892999
20170911 194400;4215.01;4215.01;4203.04;4204.64;11.687842009999995
20170911 194300;4215;4215.01;4215;4215.01;1.6720000000000002
20170911 194200;4215.01;4215.01;4215.01;4215.01;6.270305769999999
20170911 194100;4215.01;4215.01;4215.01;4215.01;0.25924983
20170911 194000;4215.01;4215.01;4215.01;4215.01;1.8750531499999998
20170911 193900;4215.01;4215.01;4215.01;4215.01;0.0867
20170911 193800;4215.21;4215.21;4215;4215.01;5.09584227

The error I'm getting is Pandas related:

Traceback (most recent call last):
  File "/Users/yazkhoury/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/Users/yazkhoury/Desktop/ml-trading/btgym/btgym/dataserver.py", line 89, in run
    episode_dataset = self.dataset.sample_random()
  File "/Users/yazkhoury/Desktop/ml-trading/btgym/btgym/datafeed.py", line 253, in sample_random
    first_row = self.data.index.get_loc(adj_timedate, method='nearest')
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/tseries/index.py", line 1411, in get_loc
    return Index.get_loc(self, key, method, tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2138, in get_loc
    indexer = self.get_indexer([key], method=method, tolerance=tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2262, in get_indexer
    tolerance=tolerance)
  File "/Users/yazkhoury/anaconda3/lib/python3.6/site-packages/pandas/indexes/base.py", line 2271, in get_indexer
    raise InvalidIndexError('Reindexing only valid with uniquely'
pandas.indexes.base.InvalidIndexError: Reindexing only valid with uniquely valued Index objects
[2017-09-14 11:00:03,575] Data_server unreachable with status: <receive_failed>.
Data_server unreachable with status: <receive_failed>.

I've attached a WeTransfer url to the CSV file here: https://we.tl/jWPBGGwyzT

I'm just curious as to what the problem could be in Pandas end? I verified that the indexes aren't duplicated for the date-time, and I've tested out the environment on one of your example datasets, and it works perfectly.

I think it might be a formatting issue in my CSV but I'm really not sure. I've saved my CSV file as:
datetime, open, high, low, close, volume for each row.

Using btfeeds and saving/loading trained models

Hi @Kismuz,

Great project! I would like to know if its possible to use the normal btfeeds for backtesting and live feeds within the gym environment?

Is it possible to save and load trained models?

Also is it possible to modify the order size on the fly eg.

            self.buyvalue = round(((self.broker.getvalue() / self.data0.close[0]) * 0.9), 1)
            self.order = self.buy(self.data,
                                  exectype=bt.Order.Limit, 
                                  size=self.buyvalue,
                                  price=self.data0.close[0],
                                  valid=datetime.now() + timedelta(minutes=10))

Thank you in advance,

Kind regards

Can I use the trained agent to generate Actions

I have trained the model. Can I use the trained agent to generate Actions in the Running agent section of setting_up_environment_basic?

Modifying the datasource

I have some csv files that I would like to use as datasource of the format

timestamp,open,high,low,close,volume

What would I have to modify to support this?

Tiny typo

btgym/btgym/envs/backtrader.py

Line 662 in d5c3236

raise ValueError('Unexpected render mode {}'.dormat(mode))

dormat(mode) > format(mode)

raise ValueError('Unexpected render mode {}'.format(mode))

No running server found

Hi,

Great work.

I was looking for examples of market trading environments for OpenAI for an idea I had around machine learning for risk management (I don't think you can teach a computer to trade on random walk data better than you can teach a human - it's like locking both in a box and hoping they can make money) but what I am hoping you can do is teach them risk management/position sizing based on prevailing market conditions which has much more useful applications.

Hooking OpenAI into Backtrader turns out to be more than I could hope for.

Just one issue I ran into which might be worth noting for others or updating your documentation...

When running your examples they all (or at least all the ones I tried) log "No running server found." then exit.

The issue is that you haven't got an agent running. It might be worth demonstrating a basic agent. Eg:

env = ..........
env.reset()
for _ in range(1000):
env.render()
env.step(env.action_space.sample()) # take a random action

ValueError: Shape must be rank 4 but is rank 3 for 'global/MaxPool' (op: 'MaxPool') with input shapes: [1,28,1].

Running environment:

OS: Ubuntu 16.0.4
tensorflow v1.7.0
Nvidia Driver Version: 390.48
Cuda : v9.1.85

Files or part of package has been run:

guided_a3c.py

Actual behaviour:

Traceback (most recent call last):
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 671, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be rank 4 but is rank 3 for 'global/MaxPool' (op: 'MaxPool') with input shapes: [1,28,1].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/aac.py", line 405, in init
self.network = self._make_policy('global')
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/aac.py", line 807, in _make_policy
network = self.policy_class(**self.policy_kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/research/gps/policy.py", line 31, in init
**kwargs
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/policy/stacked_lstm.py", line 460, in init
super(AacStackedRL2Policy, self).init(**kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/policy/stacked_lstm.py", line 370, in init
pixel_change_2d_estimator(ob_space['external'], **kwargs)
File "/home/barry/Documents/Projects/Testing/btgym/btgym/algorithms/nn/networks.py", line 249, in pixel_change_2d_estimator
'SAME'
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 1769, in max_pool
name=name)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1605, in _max_pool
data_format=data_format, name=name)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2508, in create_op
set_shapes_for_outputs(ret)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1873, in set_shapes_for_outputs
shapes = shape_func(op)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1823, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
debug_python_shape_fn, require_shape_fn)
File "/home/barry/anaconda3/envs/A3C_LSTM/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Shape must be rank 4 but is rank 3 for 'global/MaxPool' (op: 'MaxPool') with input shapes: [1,28,1].

Kernel died with basic example

Hi, nice work with btgym, I was working in a similar project (https://github.com/AdrianP-/gym_trading) and I wanted to integrate RL with a zipline/pyalgotrader/etc, but you already did it! Please send me a email :)

However I tried your examples and my kernel always die in MyEnvironment = gym.make('backtrader-v46'). Commenting this line everything works well :D. Why is necessary?

inherit BTgymDataset to use binary feed

Hi,
when I try to derivate BTgymDataset in order to use my own datasource in this way: https://www.backtrader.com/blog/posts/2015-08-11-datafeed-development/datafeed-development.html

I get btgym complaining about the NoneType being not iterable. It is trying to read a csv file even though I did override the necessary methods such as load.

Is there a use case I am missing so that I can use a non csv source of data in the same way backtrader does in the example linked above?
Thx.

AttributeError: 'FigureCanvasAgg' object has no attribute 'renderer' / ValueError: Axis limits cannot be NaN or Inf

Process DrawCerebro-2:1:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/git/btgym/btgym/rendering/plotter.py", line 75, in run
    figfilename='_tmp_btgym_render.png',
  File "/home/btgym/lib/python3.5/site-packages/backtrader/cerebro.py", line 991, in plot
    start=start, end=end, use=use)
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 220, in plot
    self.plotdata(data, self.dplotsover[data])
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 634, in plotdata
    data, opens, highs, lows, closes, volumes, vollabel)
  File "/home/btgym/lib/python3.5/site-packages/backtrader/plot/plot.py", line 580, in plotvolume
    ax.set_ylim(0, volylim, auto=True)
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 3226, in set_ylim
    top = self._validate_converted_limits(top, self.convert_yunits)
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 2836, in _validate_converted_limits
    raise ValueError("Axis limits cannot be NaN or Inf")
ValueError: Axis limits cannot be NaN or Inf

let self.volume = False then,

Process DrawCerebro-2:1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/git/btgym/btgym/rendering/plotter.py", line 78, in run
    rgb_string = fig.canvas.tostring_rgb()
  File "/home/btgym/lib/python3.5/site-packages/matplotlib/backends/backend_agg.py", line 460, in tostring_rgb
    return self.renderer.tostring_rgb()
AttributeError: 'FigureCanvasAgg' object has no attribute 'renderer'

Issue while running code

I have two issues while running the code, first one is when I try to run a basic environment. After performing the first action, I receive the following error :
ValueError: too many values to unpack (expected 4)
and when I print the output of env.step(action) :
"No key received:Server Control mode: received <{'action': 'hold'}>"

from btgym import BTgymEnv from policy_gradient import PolicyGradient import numpy as np env = BTgymEnv(filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',) done = False while not done: action = env.action_space.sample() obs, reward, done, info = env.step(action) print(reward)

Second issue is related to the rewards, the are equal to 0 whatever the actions I choose.

Allowing Long only positions (no short selling)

Hi,

great work!!
One question: I was trying to test a long only agent but it seems setting the portfolio_action, i.e.:
MyCerebro.addstrategy(MyStrategy, drawdown_call=10, skip_frame=1,state_shape=(10,4),portfolio_actions=('hold', 'buy','close'))
does not work and the agent continues with short selling.

Could you provide an example explicitly disallowing short selling?

Thanks!

Typo

Installing with `python3 setup.py develop --user` breaks pip

Use this template when reporting bugs, errors or unexpected behaviour.
Override for general questions, feature requests, proposals etc.

Running environment:

Ubuntu 16.10 LTS
Python 3.5/3.6

Files or part of package has been run:

setup.py

Expected behaviour:

Everything is expected to work as normal.

Actual behaviour:

After running python3 setup.py develop --user, pip3 breaks with import errors.

Seems to be related to this one pypa/setuptools#885

Steps to reproduce: