Standardization of Test Datased uses future values (conceptual error)

[JekyllQuant]

Standardization of Test Datased uses future values (conceptual error)

[JekyllQuant]

Yes, your assumption is totally correct but in your code you're not doing it, your doing this:
timeseries = np.array(data[i:i+train+predict])
if scale: timeseries = preprocessing.scale(timeseries)
x_i = timeseries[:-1]
y_i = timeseries[-1]

i.e. you take the full "timeseries" sliding window including X_test and Y_Test and you standardize it using the preprocessing.scale() function.

Instead of writing timeseries = np.array(data[i:i+train+predict]) you should have written timeseries = np.array(data[i:i+train]), take the mean and std from this data, and use it to standardize x_i and y_i

[Rachnog]

@JekyllQuant seems like you have right, just looking on the picture you can see that our predictions are too late. Does it mean that regression approach doesn't work at all? How do you think?

[Rachnog]

Hi, @JekyllQuant
Thank you for your interest, let's try to understand what's the problem.

This because Y_test, at test time, isn't known, so you cannot calculate the mean and the std used in the standardization process to normalize the full sliding window sample.

Let's say we already have some trained neural network for regression. We have some live data from stock market, close prices of last 20 days. They are definitely KNOWN.
x_i = timeseries[:-1]
We want to forecast 21st day price,
y_i = timeseries[-1],
but network outputs the scaled result and we want to restore it. My hypothesis is that I can just take mean and std from already KNOWN 20 days and assume that it's good enough estimate to restore the next day price. I do not use anything from the future, just statistical information from the past trying to restore future.

Correct me, if I am wrong please in my thoughts or implementation.

[JekyllQuant]

Sincere apologies for the very late reply Alex, I completely missed your post.

Does it mean that regression approach doesn't work at all? How do you think?

I cannot say much, but I wouldn't throw everything away. Maybe I would check if the information the NN is giving you can be useful.
For sure now it doesn't look quite good, aesthetically speaking.

[creotiv]

Guys dont forget that you cant normalize data against different mean and std. it will give not correct results. So if you calculate mean and std for you first window, you should use then for all your data, including predicted data.

[cn3c3p]

Hi Alex. Thanks for giving out your code, it's a very good example.
But, I still can't run volatility.py, because I CAN'T find "feature_extractor" module, Would you mind tell me where from it ?
thanks!

[Rachnog]

@cn3c3p added needed files, but I think they even weren't used in the code :)

[markudevelop]

@Rachnog which is the most effective algo after this fix? RNN?

Thanks

[mistborn17]

Consider the mean as M and inputs as x1,x2,.....xn. And we are trying to predict x_{n+1}.

After t

[tomspencer84]

Hi, did anyone ever manage to fix this strange behaviour?

[GeneveyC]

Hi, anyone have a solution to standardization of Test Datased in the goal to use this code in real world?

[JoshuaShaw]

https://medium.com/machine-learning-world/neural-networks-for-algorithmic-trading-1-2-correct-time-series-forecasting-backtesting-9776bfd9e589