miroblog/deep_rl_trader

Trading Environment(OpenAI Gym) + DDQN (Keras-RL)

deep_rl_trader's Issues

Traceback: DQN expects a model that has one dimension for each action, in this case 3.

Making all the changes to the PIP code to run, this is not the first time this error has occurred:

Traceback (most recent call last):
File "y:/python_udemy/deep_rl_trader-master/deep_rl_trader/ddqn_rl_trader.py", line 80, in
main()
File "y:/python_udemy/deep_rl_trader-master/deep_rl_trader/ddqn_rl_trader.py", line 59, in main
processor=NormalizerProcessor())
File "C:\Users\danilo.martins\Anaconda3\lib\site-packages\rl\agents\dqn.py", line 111, in init
raise ValueError('Model output "{}" has invalid shape. DQN expects a model that has one dimension for each action, in this case {}.'.format(model.output, self.nb_actions))
ValueError: Model output "Tensor("dense_2/BiasAdd:0", shape=(?, 3), dtype=float32)" has invalid shape. DQN expects a model that has one dimension for each action, in this case 3.

What am I doing wrong, you know?
That's what I just tried to give the RUN to see how it would work here.

Cannot Run this

Using TensorFlow backend.
2020-04-26 16:18:38.050381: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-04-26 16:18:38.050737: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core._multiarray_umath failed to import
ImportError: numpy.core.umath failed to import
2020-04-26 16:18:38.671400: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr

1.Is this because I dont have a GPU?
2.My IDE is Pycharm, should use Anaconda to match the environment? (Cause the pycharm keeps telling me some requirements couldnt be installed, including "Anaconda client ==1.6.0 , bitarray == 0.8.1 ......... "

I think there is a look-ahead bias

Hi there, nice work.
However I think there is a look-ahead bias.
Every timestep, you get state and this state includes the current closeprice.
Then with step method you calculate profit as:

self.exit_price = self.closingPrice
self.reward += ((self.entry_price - self.exit_price)/self.exit_price + 1)*(1-self.fee)**2 - 1 # calculate reward

In this case you are using the same information that you already used to predict the next action.
What do you think about it?

Training Data / Validation Data Overlap?

I noticed in the /data folder, the training data in /train includes all data for validation data in /test. There's no validation split in the model so I assume validation datapoints also have a chance to be trained by the model.
Doesn't that lead to overfit and exaggerated model performance?

How to use this model in real trade env?

The project implements train and test function. If it is used in actual trading envirionment, how to use this model to predict action when every K bar is close? The DQNAgent has not predict function, use model.predict() or some function else?

pip install error

Could not find a version that satisfies the requirement anaconda-client==1.6.0

error when running model.fit

Thank you so much for this great project.

When i try to run ddqn_rl_trader.py on windows (my computer has no GPU, so i use LSTM instead of CuDNNLSTM), i get the following errors:

2019-01-17 17:06:16.101245: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary
start episode ... XBTUSD_5m_70000_train.csv at 0
Traceback (most recent call last):
File "ddqn_rl_trader.py", line 81, in
main()
File "ddqn_rl_trader.py", line 65, in main
dqn.fit(env, nb_steps=5500, nb_max_episode_steps=10000, visualize=False, verbose=0)
File "C:\Python36\lib\site-packages\rl\core.py", line 182, in fit
if not np.isreal(value):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

May i ask what change i can make to this problem?

Thanks a lot

run on google colab "error"

Hi, thank you for this implementation of reinforcement learning.

I created a google colab file. I succeded to run most of the code but I got an error at the very last part.
This is the error im getting :
`Training for 5500 steps ...
start episode ... XBTUSD_5m_70000_train.csv at 0

---

ValueError Traceback (most recent call last)

in ()
1 while True:
2 # train
----> 3 dqn.fit(env, nb_steps=5500, nb_max_episode_steps=10000, visualize=False, verbose=2)
4 try:
5 # validate

/usr/local/lib/python3.6/dist-packages/rl/core.py in fit(self, env, nb_steps, action_repetition, callbacks, verbose, visualize, nb_max_start_steps, start_step_policy, log_interval, nb_max_episode_steps)
180 observation, r, done, info = self.processor.process_step(observation, r, done, info)
181 for key, value in info.items():
--> 182 if not np.isreal(value):
183 continue
184 if key not in accumulated_info:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()`

How can I fix this?

I shared the colab file so it can mabe help other people to.
https://colab.research.google.com/drive/1DyURfsL9091Hx8IsEKwPGFxVl_aUEspp

Thank you for your help,
greg