walk.forward uses only the [testing.timespan] subset from: [R-Forge #6258] bug in demo(luxor.8.walk.forward) in quantstrat 0.9.1687

[braverock]

walk.forward uses only the [testing.timespan] subset from: [R-Forge #6258] bug in demo(luxor.8.walk.forward) in quantstrat 0.9.1687

[joshuaulrich]

The essence of this issue is that the walk-forward analysis function will only use the testing timespan for all necessary calculations for the strategy. That means all indicators, signals, etc will only see the testing data for their calculations. This is a problem for long-term indicators (e.g. 200-day SMA) and indicators that require ~3x their window length to stabilize (e.g. EMA).

It makes sense to restrict the testing window to only the out-of-sample data in order to only run the rules on the test sample. But the strategy should use all available historical data to calculate indicators, at minimum. It may also make sense to run signals on all available data, but I have not thought about that much.

[evelynmitchell]

This appears to be related to the SMA calculation running out of data on the walk forward.

source('~/1CurrentProjects/quantstrat/demo/luxor.8.walk.forward.R')
Loading required package: quantstrat
Loading required package: quantmod
Loading required package: xts
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

    as.Date, as.Date.numeric

Loading required package: TTR
Version 0.4-0 included new data defaults. See ?getSymbols.
Loading required package: blotter
Loading required package: FinancialInstrument
Loading required package: PerformanceAnalytics

Package PerformanceAnalytics (1.5.0) loaded.
Copyright (c) 2004-2015 Peter Carl and Brian G. Peterson, GPL-2 | GPL-3
http://r-forge.r-project.org/projects/returnanalytics/


Attaching package: ‘PerformanceAnalytics’

The following object is masked from ‘package:graphics’:

    legend

Loading required package: foreach
foreach: simple, scalable parallel programming from Revolution Analytics
Use Revolution R for scalability, fault tolerance and more.
http://www.revolutionanalytics.com
loading  GBPUSD .....
Reading  2002.10.21.GBPUSD.rda ... done.
Reading  2002.10.22.GBPUSD.rda ... done.
Reading  2002.10.23.GBPUSD.rda ... done.
Reading  2002.10.24.GBPUSD.rda ... done.
Reading  2002.10.25.GBPUSD.rda ... done.
Reading  2002.10.28.GBPUSD.rda ... done.
Reading  2002.10.29.GBPUSD.rda ... done.
Reading  2002.10.30.GBPUSD.rda ... done.
Reading  2002.10.31.GBPUSD.rda ... done.
rbinding data ... done.
Loading required package: doMC
Loading required package: iterators
Loading required package: parallel
[1] "=== training WFA on 2002-10-21 00:30:00/2002-10-23 23:30:00"
numValues: 15, numResults: 0, stopped: TRUE
got results for task 1
numValues: 15, numResults: 1, stopped: TRUE
returning status FALSE
got results for task 2
numValues: 15, numResults: 2, stopped: TRUE
returning status FALSE
got results for task 3
numValues: 15, numResults: 3, stopped: TRUE
returning status FALSE
got results for task 4
numValues: 15, numResults: 4, stopped: TRUE
returning status FALSE
got results for task 5
numValues: 15, numResults: 5, stopped: TRUE
returning status FALSE
got results for task 6
numValues: 15, numResults: 6, stopped: TRUE
returning status FALSE
got results for task 7
numValues: 15, numResults: 7, stopped: TRUE
returning status FALSE
got results for task 8
numValues: 15, numResults: 8, stopped: TRUE
returning status FALSE
got results for task 9
numValues: 15, numResults: 9, stopped: TRUE
returning status FALSE
got results for task 10
numValues: 15, numResults: 10, stopped: TRUE
returning status FALSE
got results for task 11
numValues: 15, numResults: 11, stopped: TRUE
returning status FALSE
got results for task 12
numValues: 15, numResults: 12, stopped: TRUE
returning status FALSE
got results for task 13
numValues: 15, numResults: 13, stopped: TRUE
returning status FALSE
got results for task 14
numValues: 15, numResults: 14, stopped: TRUE
returning status FALSE
got results for task 15
numValues: 15, numResults: 15, stopped: TRUE
first call to combine function
evaluating call object to combine results:
  fun(result.1, result.2, result.3, result.4, result.5, result.6, 
    result.7, result.8, result.9, result.10, result.11, result.12, 
    result.13, result.14, result.15)
returning status TRUE
[1] "=== testing param.combo 5 on 2002-10-24/2002-10-24 23:30:00"  "=== testing param.combo 10 on 2002-10-24/2002-10-24 23:30:00"
   nFAST nSLOW
5      9    42
10     9    44
[1] "=== training WFA on 2002-10-22/2002-10-24 23:30:00"
numValues: 15, numResults: 0, stopped: TRUE
got results for task 1
numValues: 15, numResults: 1, stopped: TRUE
returning status FALSE
got results for task 2
numValues: 15, numResults: 2, stopped: TRUE
returning status FALSE
got results for task 3
numValues: 15, numResults: 3, stopped: TRUE
returning status FALSE
got results for task 4
numValues: 15, numResults: 4, stopped: TRUE
returning status FALSE
got results for task 5
numValues: 15, numResults: 5, stopped: TRUE
returning status FALSE
got results for task 6
numValues: 15, numResults: 6, stopped: TRUE
returning status FALSE
got results for task 7
numValues: 15, numResults: 7, stopped: TRUE
returning status FALSE
got results for task 8
numValues: 15, numResults: 8, stopped: TRUE
returning status FALSE
got results for task 9
numValues: 15, numResults: 9, stopped: TRUE
returning status FALSE
got results for task 10
numValues: 15, numResults: 10, stopped: TRUE
returning status FALSE
got results for task 11
numValues: 15, numResults: 11, stopped: TRUE
returning status FALSE
got results for task 12
numValues: 15, numResults: 12, stopped: TRUE
returning status FALSE
got results for task 13
numValues: 15, numResults: 13, stopped: TRUE
returning status FALSE
got results for task 14
numValues: 15, numResults: 14, stopped: TRUE
returning status FALSE
got results for task 15
numValues: 15, numResults: 15, stopped: TRUE
first call to combine function
evaluating call object to combine results:
  fun(result.1, result.2, result.3, result.4, result.5, result.6, 
    result.7, result.8, result.9, result.10, result.11, result.12, 
    result.13, result.14, result.15)
returning status TRUE
[1] "=== testing param.combo 5 on 2002-10-25/2002-10-25 13:30:00"  "=== testing param.combo 10 on 2002-10-25/2002-10-25 13:30:00" "=== testing param.combo 15 on 2002-10-25/2002-10-25 13:30:00"
   nFAST nSLOW
5      9    42
10     9    44
15     9    46
Error in runSum(x, n) : Invalid 'n'
In addition: Warning messages:
1: In beg:(n + beg - 1) :
  numerical expression has 2 elements: only the first used
2: In 1:(n - 1 + NAs) :
  numerical expression has 2 elements: only the first used
3: In beg:(n + beg - 1) :
  numerical expression has 2 elements: only the first used
4: In 1:(n - 1 + NAs) :
  numerical expression has 2 elements: only the first used
5: In beg:(n + beg - 1) :
  numerical expression has 3 elements: only the first used
6: In 1:(n - 1 + NAs) :
  numerical expression has 3 elements: only the first used
7: In `/.default`(runSum(x, n), n) :
  longer object length is not a multiple of shorter object length

[braverock]

I think that the 'easiest' solution to this will be to break up the calls inside walk.forward.

Each paramset should have the entire series used for indicators and signals, and then each subset should be evaluated separately for rules.

1. It seems like we could use applyIndicators and applySignals accross the entire period, and then only apply the rules to each subset.

2. Alternately, we could run full-period backtests for every paramset, and then use performance in the testing subset to select which one will be used 'out of sample'. One challenge of this second method is that the rules would not be fully path dependent, since you could have a residual position from prior periods under different parameter sets.

Assuming that the first method is the best, then applyIndicators and applySignals would first be called for every paramset, and stored. Then applyRules would be called using only the mktdata subset from the period being examined.

[edward-wilson]

Just chiming in to support braverock's approach above. Re-generating signals for each testing subset has been a roadblock for my strategies and WFA in quantstrat as outlined (admittedly quite poorly) in a comment on #44.

[braverock]

Just commenting on the ultimate solution here (more detail in the commits):

The way things work in the current code are that indicators and signals will cover the entire period, and applyRules will be called with the new optional argument rule.subset defining the training(testing) subset to use.

This means that entry/exit/etc signals for rule.subset will be checked.

This would still be in-exact for the training period for strategies with long holding periods that would tend to cross train/test boundaries. such strategies should probably call walk.forward with anchored=TRUE