vivekpa/aialpha

Use unsupervised and supervised learning to predict stocks

aialpha's Issues

Run fails on a clean freshly downloaded project

Hi,

First, thanks for posting this code.

The project has been downloaded as a zip file. At the beginning of the run it exits with an error about a missing dollar_bars.csv
Searching dollar_bars.csv finds 3 references. 2 in run.py/run_full.py sources, that immediately read it via some function, and fail. And 1 in bar_sample.py. But running bar_sample.py fails, since price_vol.csv is missing.

Searching price_vol.csv finds 3 read references from bar_sample.py itself, 1 read reference in test.py, and 1 write reference in test.py, BUT it relies on reading the file first. So no actual write references that create this file.

Hence, a dead end.
Is there a flow I am missing?
When you download it yourself to a clean directory (without the data already there), does it work as is at your end?

Again, thanks for your contribution.

FileNotFoundError

I'm having some issues, getting these errors:

C:\Windows\System32> C:\Users\kanzl\Downloads\AIAlpha-master\AIAlpha-master\run.py install
Using TensorFlow backend.
Creating tick bars...
Reading data in batches of 20000000
Traceback (most recent call last):
File "C:\Users\kanzl\Downloads\AIAlpha-master\AIAlpha-master\run.py", line 14, in
base.batch_run()
File "C:\Users\kanzl\Downloads\AIAlpha-master\AIAlpha-master\data_processor\base_bars.py", line 23, in batch_run
for batch in pd.read_csv(self.file_path, chunksize=self.batch_size, index_col=0):
File "C:\Users\kanzl\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\kanzl\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\Users\kanzl\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\parsers.py", line 880, in init
self._make_engine(self.engine)
File "C:\Users\kanzl\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\kanzl\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\parsers.py", line 1891, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 374, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File sample_data/raw_data/price_vol.csv does not exist: 'sample_data/raw_data/price_vol.csv'

If you could help I would appreciate

Does Wavelet leak future price information into your input data?

I noticed that most software uses Moving Average to smooth data, and the simple moving average obviously has a lag, so I am wondering if Wavelet leaks future price information into training input and test input data.

Dockerize Application:Trying to follow quickstart guide to run the application quickly, having problems

Tried following the quickstart guide to run the application quickly but running into many software dependency issues. Have you considered dockerizing this application?

I have a background in this area, would be interested in contributing to the project to add that capability if you are open to incorporating this as a feature.

Let me know.

processed_data/price_bars/dollar_bars.csv missing

I downloaded the project from Git on 7/31/2019.
run.py has df = preprocess.make_features(file_path=f"sample_data/processed_data/price_bars/dollar_bars.csv", window=20,
csv_path="sample_data/processed_data/autoencoder_data", save_csv=True)
C:_AI\AIAlpha-master\sample_data\processed_data\price_bars only has test

can you help me?

pywt has no attribute dwt

hi , in preprocess.py i have this following error :
pywt module has no attribute dwt ()

LSTM predict

Why not consider Volume as input?

When I interviewed several professional traders, none of them would ignore the Volume information before making any decision. When they refer to a market signal, they will define it in both Price and Volume.

And if one is a high-frequent trader, Level 2 information is mandatory too.

So maybe you can add those to your model, hopefully that will increase the accuracy.

let's connect for further improvment

"I am currently working on using Reinforcement Learning to make a trading agent that will learn the trading strategy to maximise the portfolio."
this line will drag me here. your article is really awesome sir. same time i am also working on reinforcement learning trading. can we connect...! for discussion and exchange ideas?

Yahoo data

I saw in your medium article you said you were using
stock_data = pdr.get_data_yahoo(self.ticker, self.start, self.end)
for data acquisition
yet in your sample data you had a time col which yahoo doesn't provide
is there a way can I use this for daily prices?
Thank you

统计特征问题

您好，您在文中提到‘The sampling that we are used to seeing is time sampled (we get bars every 1min), but this is known to exhibit non stationarities and the returns are not normally distributed. So, as explained in Advances in Financial Machine Learning, we are going to sample it according to the number of ticks, or the amount of volume or the amount of dollars traded’，这段是否有文献支撑呢？我尝试了按number of ticks, or the amount of volume or the amount of dollars traded这几种方式进行sampling并画出统计图，但是并没有呈现出任何统计特征出来。

R library problem

Exploratory package dont work at the moment

Give plz full file of model

Maybe at google disk or mega?

Seems not working neural network

Model.py creates stock price:
stock_price = np.exp(np.reshape(prediction, (1,)))*stock_data_test[i]

Stock price (from testing list) always multiplied with same constant coming from prediction. That's why predicted stock graph always matches original values. Neural network predictor returning always same value for any time value.

Please fix the code.

Please read through the comments in this project

Thanks for putting this code/blog together. Unfortunately your approach suffers from the same problems as other projects that use deep learning to forecast stock prices. You can review the comments here:

https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction

I did find your discussion of wavelets and autoencoders worthwhile, nice work!

Missing directories and symbolic links

With a clean git clone of this, run.py errors out quickly because of path errors. It should be addressed internally but with this at the top of run.py, it will run

import os
aialpha_home=os.getcwd()
datadir = aialpha_home + '/data/processed_data'
os.makedirs(datadir, exist_ok=True)
os.chdir(datadir)
sampledata = aialpha_home + '/sample_data/'
if not os.path.exists('sample_data') :
os.symlink(sampledata, 'sample_data')

autoencodedata = aialpha_home + '/sample_data/processed_data/autoencoder_data'
if not os.path.exists('rf_data'):
os.symlink(autoencodedata, 'rf_data')

os.chdir(aialpha_home)

Evaluate the return prediction, not the price prediction

I only went through the readme. The aim stated there is to predict stock returns, so why evaluate the algorithm using stock prices?

Predicting prices with great accuracy is very easy: predict that tomorrow's price will be today's price. The prediction will be correct up to a few percents at most. But that's useless. The analysis in the readme file doesn't make it obvious that the model does better.

I would suggest computing the f1 score or the return-weighted accuracy of a classifier predicting the sign of the returns.

download() missing 1 required positional argument: 'tickers'

Traceback (most recent call last):
File "run.py", line 1, in
from get_data import GetData
File "/root/DeepMarket/Numerical/AIAlpha/get_data.py", line 3, in
fix.pdr_override()
TypeError: download() missing 1 required positional argument: 'tickers'

Unsupported operand type(s) for /: 'str' and 'float'

Hi,

I cloned the repo, installed the required packages in a new conda environment and runned run.py without any changes to the code.

Getting an error in data_processing line 18; "unsupported operand type(s) for /: 'str' and 'float'"

python version 3.6.9
conda 4.7.11
windows 10 professional

requirements.txt file is missing

In the Readme, it is mentioned that one needs to do pip install -r requirements.txt, but the file is missing in the repository.

Want to modify for my project

How to create encoded_return_train_data ?

You are using encoded_return_train_data in model_20_encoded.py
How can I create encoded_return_train_data from an input file?

Some comments

Hi, Thanks for the very nice work! I am reading through your code and I am having problems understanding some of the steps taken. I'd suggest adding more comments in your code. Nonetheless, your explanations in the README are very useful!

In preprocessing.py: What do you mean with features split? Also, the sequence of steps in make_wavelet_train is not clear: why you used part of the original signal (macd = np.mean(x[5:] - np.mean(x))? And why you created the indicators.csv set?

You called your autoencoder a stacked autoencoder, while your approach does not construct the autoencoder model in a stacked manner: you do not train your layers in turns, but all 3 at once. Reference: http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders

New complementary tool

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.

I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.

The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction).
For this I have used big data tools like pandas python, stock market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.

With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).

I think it could be useful to you, to improve, I would like to give it to you, and if you are interested in improving and collaborating I am also willing, and if not I would like to file it in the drawer.

Probably how long it will take us to run run.py

@VivekPa Can this excemple run without Discrete graphics card?

you are prepocessing test dataset before implementing the model

This means that you assume to know the future!
This would never work.
Regards.

If I had today and tomorrow's average I wouldn't need a fancy network to predict prices

I think ill use a moving window the size of several half lives.

What's the state of the project?

Hi, I really like some ideas of the project but when I look into the code, everything is deactivated and only reforest with minimax scaler is used.
Is the rest working that badly?

In your medium post, you present graphs with price and predicted price. Is there also an evaluation of a trading strategy with an equity curve?

Thank you very much for publishing your progress!

Best

CSV not found error?

Hi Champs:

I'm in the last miles to get it done.
May you guide me how to solve the CSV not found error?
Thanks.

BR Rio

rio@ubuntu:/opt/tensorflow/tensorflow/models/research$ python3 AIAlpha/run.py
Using TensorFlow backend.
Traceback (most recent call last):
File "AIAlpha/run.py", line 8, in
dataset, average, std = nnmodel(500, 0.01, 0.01)
File "/opt/tensorflow/tensorflow/models/research/AIAlpha/model_20_encoded.py", line 13, in nnmodel
train_data = np.array(pd.read_csv("60_return_forex/encoded_return_train_data.csv", index_col=0))
File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 702, in parser_f
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 429, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 895, in init
self._make_engine(self.engine)
File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1122, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1853, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 387, in pandas._libs.parsers.TextReader.cinit
File "pandas/_libs/parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'60_return_forex/encoded_return_train_data.csv' does not exist: b'60_return_forex/encoded_return_train_data.csv'
rio@ubuntu:/opt/tensorflow/tensorflow/models/research$ find . -name "*data.csv"
./AIAlpha/60_return_forex/encoded_return_train_data.csv
./AIAlpha/60_return_forex/return_train_data.csv
./AIAlpha/60_return_forex/return_test_data.csv
./AIAlpha/60_return_forex/encoded_return_test_data.csv
./AIAlpha/features/autoencoded_corrected_data.csv
./AIAlpha/features/autoencoded_data.csv
./AIAlpha/features/autoencoded_train_data.csv
./AIAlpha/features/autoencoded_test_data.csv
./AIAlpha/preprocessing/test_data.csv
./AIAlpha/data_folder/train_data.csv
./AIAlpha/data_folder/training_data.csv
rio@ubuntu:/opt/tensorflow/tensorflow/models/research$