timestocome/test-stock-prediction-algorithms

Use deep learning, genetic programming and other methods to predict stock and market movements

StockPredictions

Use classic tricks, neural networks, deep learning, genetic programming and other methods to predict stock and market movements.

Both successful and unsuccessful experiments will be posted. This section is things that are currently being explored. Completed projects will be wrapped up and moved to another repository to keep things simple.

The main goal of this project is to learn more about time series analysis and prediction. The stock market just happens to have lots of complicated time series and available data

The first evolving neural net does the best job of predicting daily changes. It's impressive. That'll be my first go to tool

The NASDAQ Evolved Network is a good simple example that should be easy to apply to any index

Data sources:

http://finance.yahoo.com/

https://fred.stlouisfed.org/

https://stooq.com

Data and the cleaning programs:

https://github.com/timestocome/StockMarketData

Recommended Reading:

http://www.e-m-h.org/Fama70.pdf Efficient Market Hypothesis

http://faculty.chicagobooth.edu/workshops/finance/pdf/Shleiferbff.pdf Bubbles for FAMA

http://www.unofficialgoogledatascience.com/2017/04/our-quest-for-robust-time-series.html How Google does series predictions

http://www.econ.ucla.edu/workingpapers/wp239.pdf Let's Take the Con Out of Economics

https://www.manning.com/books/machine-learning-with-tensorflow Meap Machine Learning with TensorFlow

https://www.amazon.com/gp/product/B01AFXZ2F4/ Everybody Lies, Big Data, New Data, and What the Internet can tell us about who we really are

https://www.amazon.com/gp/product/B06XDWV2Z2 The Money Formula: Dodgy Finance, Pseudo Science, and How Mathematicians Took Over the Markets

https://blog.twitter.com/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series Finding anomalies in time series

https://www.wired.com/2009/02/wp-quant/ Wired: The Formula that Killed Wall St

http://onlinelibrary.wiley.com/doi/10.1111/j.1467-6419.2007.00519.x/abstract What do we know about the profitability of technical analysis

https://eng.uber.com/neural-networks/ Engineering extreme event forecasting at Uber with RNNs

http://lib.ugent.be/fulltxt/RUG01/001/315/567/RUG01-001315567_2010_0001_AC.pdf An empirical analysis of algorithmic trading on financial markets

http://www.radio.goldseek.com/bachelier-thesis-theory-of-speculation-en.pdf The Theory of Speculation, L. Bachelier

http://dl.acm.org/citation.cfm?id=1541882 Anomaly Detection: A Survey 2009 ACM

http://www.mrao.cam.ac.uk/~mph/Technical_Analysis.pdf Technical Analysis

https://is.muni.cz/th/422802/fi_b/bakalarka_final.pdf Prediction of Financial Markets Using Deep Learning ( see: https://github.com/timestocome/FullyConnectedForwardFeedNets for an example fully connected deep learning network )

http://www.doc.ic.ac.uk/teaching/distinguished-projects/2015/j.cumming.pdf An Investigation into the Use of Reinforcement Learning Techniques within the Algorithmic Trading Domain

On my reading list:

http://socserv.mcmaster.ca/racine/ECO0301.pdf Nonparametric Econometrics: A Primer

http://natureofcode.com/ The Nature of Code

http://www.penguinrandomhouse.com/books/314049/scale-by-geoffrey-west/9781594205583/ Scale: The universal laws of growth...

https://en.wikipedia.org/wiki/The_Drunkard%27s_Walk The Drunkard's Walk

Useful Websites:

http://www.nber.org/ The National Bureau of Economic Research

https://fred.stlouisfed.org/ FRED, Federal Reserve Bank of St Louis

http://www.zerohedge.com/ ZeroHedge, mostly noise, occasionally something useful appears

Cool tools:

https://facebookincubator.github.io/prophet/docs/quick_start.html Facebook Prophet - Python and R time series prediction library

https://research.google.com/pubs/pub41854.html Inferring causal impact using bayesian structural time series models ( Google has an R package http://google.github.io/CausalImpact/ to go with this paper )

https://gbeced.github.io/pyalgotrade/ Python Algorithmic Trading Library

http://pybrain.org/ PyBrain Machine Learning Library

https://github.com/CodeReclaimers/neat-python Python NEAT Library for evolving neural networks

Podcasts:

http://www.podcastchart.com/podcasts/berkshire-hathaway-2017-annual-shareholders-meeting/episodes/berkshire-hathaway-vice-chairman-charlie-munger-speaks-with-yahoo-finance-editor-in-chief-andy-serwer 2017 Berkshire Hathaway Shareholder's Meeting

test-stock-prediction-algorithms's Issues

Reason behind working on this project

I really liked the way you are working on this project, by not just using the Deep Learning models but also adding traditional technical analysis to it.
Can I know why you are working on this? An open-Source Consultation, or a Research Project, an assignment, or just out of interest?

...

.

jupyter / ipython notebook - a new test with PredictMarketWithBayesTheorem

any plans to use the jupyter notebook?
https://jupyter.org/
graphs e.g. pyplot etc are somewhat easier with that, but that it would probably consume significantly more memory and resources

i did a new test of PredictMarketWithBayesTheorem in my repository
PredictMarketWithBayesTheorem.ipynb

the main things i changed are
normally log returns is computed as log(today's price) - log(yesterday's price) = log ( today's price / yesterday's price ). hence this formula is used instead
and for the categories i used a histogram to define them, scroll below for the histogram

this time round there is a difference, what is most interesting is the HL and HG predictions

why are there more README commits in this project?

Hello @timestocome I have started researching upon stock market prediction and came across your project.
Thank You for your work, but I don't understand why you just update the README all the time :P ?

Maybe there is another way out

LSTM may not be suitable for predicting trend in a short stock market period.

I am attempting to classify the shape of a stock(last N days OHLCV data, 5 days for instance) into M kinds of classification (7 output classes).

Such as drop -8% of stock price as a class, drop [-8%, -4%) as another class, etc.

As you can see, the training accuracy is high, and definitely overfitting:
Accuracy for training classes: (7 classes)
[ 82.609 84.146 82.596 87.204 84.164 77.381 80. ]

Where as the predict accuracy is low:
Accuracy for predict following 20 days
[ nan 0. 20. 57.143 50. 0. nan]

For now, I am wondering if change the classification model to DBN could produce a more reasonable data outcome. Hope it helps.

Below is part of the out fragment:

Epoch 1996/2000
81581/81581 [==============================] - 58s - loss: 0.0157 - acc: 0.9954
Epoch 1997/2000
81581/81581 [==============================] - 58s - loss: 0.0139 - acc: 0.9957
Epoch 1998/2000
1650/81581 [..............................] - ETA: 57s - loss: 0.0173 - acc: 0.9958
81581/81581 [==============================] - 58s - loss: 0.0143 - acc: 0.9954
Epoch 1999/2000
81581/81581 [==============================] - 58s - loss: 0.0153 - acc: 0.9954
Epoch 2000/2000
5050/81581 [>.............................] - ETA: 55s - loss: 0.0176 - acc: 0.9943
81581/81581 [==============================] - 58s - loss: 0.0158 - acc: 0.9951
save LSTM model...
############## validation on test data ##############
scaled data mse: 0.130540770636
load LSTM model...
############## validation on train data ##############
scaled data mse: 0.0391746699673
############## validation on valid data ##############
scaled data mse: 0.176731004083
############## validation on lately data ##############
scaled data mse: nan

---------- AMD ----------

classification counter:
[23, 82, 339, 422, 341, 84, 30]
classification possibility:
[ 1.741 6.207 25.662 31.945 25.814 6.359 2.271]
classification train predict:
[ 82.609 84.146 82.596 87.204 84.164 77.381 80. ]
classification valid predict:
[ nan 0. 20. 57.143 50. 0. nan]

---

                   close     volume      predict_profit  a_+1_d  p_+1_d      

Date
2017-03-15 13.98 54885200 -2.360515 -1.0 -2.0
2017-03-16 13.65 44129100 -1.172161 -1.0 -2.0
2017-03-17 13.49 218636000 6.745738 2.0 1.0
2017-03-20 14.40 90863900 -4.027778 -2.0 0.0
2017-03-21 13.82 72191500 2.026049 1.0 1.0
2017-03-22 14.10 61089400 -2.198582 -1.0 -1.0
2017-03-23 13.79 44144100 -0.652647 0.0 0.0
2017-03-24 13.70 49903700 0.000000 0.0 0.0
2017-03-27 13.70 42537800 -0.072993 0.0 2.0
2017-03-28 13.69 37005800 0.146092 0.0 0.0
2017-03-29 13.71 37777200 2.479942 1.0 -1.0
2017-03-30 14.05 43814100 3.558719 1.0 -1.0
2017-03-31 14.55 84362600 0.618557 0.0 0.0
2017-04-03 14.64 48299200 -3.278689 -1.0 1.0
2017-04-04 14.16 58217200 0.070621 0.0 -2.0
2017-04-05 14.17 58384000 -6.351447 -2.0 2.0
2017-04-06 13.27 139038000 1.883949 1.0 1.0
2017-04-07 13.52 70297900 -3.106509 -1.0 1.0
2017-04-10 13.10 46924500 0.000000 0.0 1.0
2017-04-11 13.10 59786900 -2.595420 -1.0 0.0
2017-04-12 12.76 37087100 NaN NaN 0.0

Wanna contribute on this project

Can I contribute on this project by adding some reading list and useful website???
And later also suggest some ways to predict stock with genetic algorithm?