robcarver17 / pysystemtrade Goto Github PK

View Code? Open in Web Editor NEW

2.5K 2.5K 792.0 932.21 MB

Systematic Trading in python

License: GNU General Public License v3.0

Python 97.89% Shell 0.11% CSS 0.08% JavaScript 0.61% HTML 0.30% Jupyter Notebook 1.02%

pysystemtrade's People

Contributors

Stargazers

Watchers

Forkers

cmorgan goldtpython eliotjackson canicola vt100 sanyaade-fintechnology bmaher74 balajivinodap nachobenitez uniwin adriantorrie codetradr rlcjj r0k3 tzaavi sahanduiuc expressoman greatgreatgreg yemintao anthonywise ehiggs flysoso trrolek skippyho yowtzu mattmorris-91 pszu sixkingdoms-op sixkingdoms frrp petevej js190 femtotrader tuxtag nfshr quant1729 dg32 arvindchandra mangini210 borundev ganesh-git2014 cymond altfund themltrader leedaga systemtrader sunphoenix18 fenildf oizma pkan0583 michaljirman dcurrents eurekar repotudou spyamine robuko timzhangau luoq lus8806 jphme matdorling miki0782 arunnairid johnsonc sizzles liulion rpatil524 mountaincountry herrold ifquant ericscheier samson-huang nieshaoshuai hejunbok garftalk jianfly tianhm falconzyx allensmile xcbat stevenlol red7hj win2cs ijeomaonuosa benjamesbabala egkachai placidocn myahmao reggiewong jaydenwhyte lcota wiehern gnuhub albertdatui sathishn ianmadlenya kurtas kikolev jonahhill huahuajhu

pysystemtrade's Issues

get_actual_position yields attribute error

Hi Rob,

system.portfolio.get_actual_position("CORN") seems to not work. I have tried using both fixed and estimated portfolios. The errors show up as follows:

AttributeError: 'PortfoliosFixed' object has no attribute 'accounts'
AttributeError: 'PortfoliosEstimated' object has no attribute 'accounts'

Same error shows up with get_actual_buffers_for_position as well. For what it's worth, I am only running one trading rule on one instrument (corn). Thank you very much.

Regards,
Tony

Data Update plus script for backadjusting futures

Hi Rob,

Would you be so kind as to update the CSV data? I would like to simulate performance over Brexit.

Additionally, are you able to provide and script for back-adjusting the futures contracts? I have attempted to create one so I can update the data myself. It is based on the traded price for the new future and subtracting the difference between the traded price of the new future from the price of the old future at the time of trade across the historical data series, but this is yielding some strange results for me. I assume you have something similar you must used when rolling contracts and back-adjusting the price series?

Thanks

ForecastCombine

Hi,

On https://github.com/robcarver17/pysystemtrade/blob/master/docs/introduction.md there is a line stating from systems.forecast_combine import ForecastCombine ; however, when I do this, I receive the error ImportError: cannot import name 'ForecastCombine'.

I checked the forecast_combine.py file and there are only two options in there: i) ForecastCombineFixed, and ii) ForecastCombineEstimated.

What should the relevant code be at the site mentioned above? Should it read: from systems.forecast_combine import ForecastCombine ?

Currency performance is different

Why do the returns and metrics in pysystemtrade change so markedly if you change the base currency? Shouldn't the returns be similar for various currencies given they are mostly USD instruments and most of the account stays in cash? The vol is often much lower than target vol when using non-USD currency balances. How do you grapple with this?

drawdown function in pdutils.py crashing Python

Sorry for another issue, but just wanted to bring up that every time I would try system.accounts.portfolio().stats(), python would crash. After reviewing the output to just up to the crash point, I isolated the issue to the drawdown function in pdutils.py. For some reason, changing the "window" to six digits did the trick and it doesn't crash anymore.
maxx = pd.rolling_max(x, 999999, min_periods=1) #six digits instead of 8 for python crashing error
If it matters, I'm running Python 3.4.4 on Windows 7.

Thank you,
Tony

intraday data

Need to handle conversion of daily / sparse intraday to daily.
Need to properly handle intraday data.

AttributeError: 'System' object has no attribute 'accounts'

I have encountered the following error message when I try running the tutorial codes:

from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)

File "c:\pysystemtrade\systems\forecast_combine.py", line 352, in get_SR_cost_for_instrument_forecast return self.parent.accounts.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)

AttributeError: 'System' object has no attribute 'accounts'

There is no 'accounts' attribute defined within \systems\basesystem.py

system.rawdata.daily_denominator_price() retrieving wrong price?

Hello,

In my pursuit to try to figure out why the carry data affects PandL/Stats(), I think I may have uncovered a bug (please forgive me if I have not, still learning Python). The code for daily_denominator_price in rawdata.py is essentially:

def _daily_denominator_returns(system, instrument_code, this_stage):

            dem_returns = this_stage.get_daily_prices(instrument_code)
            return dem_returns

To me, this just means getting the regular daily prices (although I note you state "This won't always be the same as the normal 'price'"). However, when I run the code it seems to be pulling the price from the carry data file:

>>> mysystem.rawdata.daily_denominator_price("CORN").head(5)
1981-09-25    326.75
1981-09-28    327.75
1981-09-29    332.75
1981-09-30    334.75
1981-10-01    334.00
Freq: B, Name: PRICE, dtype: float64

>>> mysystem.rawdata.get_daily_prices("CORN").head(5)
1981-09-25    693.50
1981-09-28    694.50
1981-09-29    699.50
1981-09-30    701.50
1981-10-01    700.75
Freq: B, Name: price, dtype: float64

For good measure, here is the output from the data stage:

>>> mysystem.data.daily_prices("CORN").head(5)
1981-09-25    693.50
1981-09-28    694.50
1981-09-29    699.50
1981-09-30    701.50
1981-10-01    700.75
Freq: B, Name: price, dtype: float64

Am I missing something (probably), or is this a bug? And if you could point me in the right direction as to my original pursuit (why carry would affect PandL/Stats() if I'm not using any carry trading rules), I would greatly appreciate it.

Thank you,
Tony

RuntimeWarning: Mean of empty slice

Hi Rob, I am trying to compute instrument weights for two instruments, US20 and SP500, with two rules on them, ewmac_8 and ewmac_32 (not estimated, equal weights).

from systems.provided.example.rules import ewmac_forecast_with_defaults as ewmac
from sysdata.configdata import Config
from systems.account import Account
from systems.forecast_combine import ForecastCombine
from systems.forecast_scale_cap import ForecastScaleCap
from systems.basesystem import System
from sysdata.csvdata import csvFuturesData
from systems.forecasting import Rules
from systems.forecasting import TradingRule
from systems.portfolio import PortfoliosEstimated
from systems.positionsizing import PositionSizing

data = csvFuturesData()
my_config = Config()
my_config.instruments=["US20", "SP500"]

ewmac_8 = TradingRule((ewmac, [], dict(Lfast=8, Lslow=32)))
ewmac_32 = TradingRule(dict(function=ewmac, other_args=dict(Lfast=8, Lslow=32)))
my_rules = Rules(dict(ewmac8=ewmac_8, ewmac32=ewmac_32))

my_system = System([Account(), PortfoliosEstimated(), PositionSizing(), ForecastScaleCap(), my_rules, ForecastCombine()], data, my_config)
my_system.config.forecast_weight_estimate['method']="equal_weights"
my_system.config.instrument_weight_estimate['method']="bootstrap"
my_system.config.instrument_weight_estimate["monte_runs"]=10
my_system.config.instrument_weight_estimate["bootstrap_length"]=250
print (my_system.portfolio.get_instrument_weights())
print(my_system.portfolio.get_instrument_diversification_multiplier())

When I run this system, I get

Cost multiplier of 0.0 will be ignored as equalising SR in optimisation (equalise_SR=True)
Zero cost multiplier and not applying cost weightings - so costs won't be used at all
Cost multiplier of %2.f is less than one and not applying cost weightings - effect of costs may be underestimated
WARNING: No forecast weights  - using equal weights of 0.5000 over all 2 trading rules in system
WARNING: No forecast weights  - using equal weights of 0.5000 over all 2 trading rules in system
....
...
Anaconda2\envs\py3k\lib\site-packages\numpy\lib\nanfunctions.py:675: RuntimeWarning: Mean of empty slice
  warnings.warn("Mean of empty slice", RuntimeWarning)
Anaconda2\envs\py3k\lib\site-packages\numpy\lib\nanfunctions.py:1136: RuntimeWarning: Degrees of freedom <= 0 for slice.
  warnings.warn("Degrees of freedom <= 0 for slice.", RuntimeWarning)
...
...

This output goes on for a while, and I ran it for a long time, it did not finish. Am I doing something wrong during the configuration, or did I stumble upon a bug?

Thanks,

get_raw_cost_data in csvdata invalid type

Hello Rob,

New version of the method get_raw_cost_data in csvdata returns objects of different type (either dict or tuple) depending on cost data availability. This leads to a runtime crash later in account.py _get_SR_cost.

File: sysdata\csvdata.py
Method: get_raw_cost_data
Line: 117, 123
Still returns tuples, not dicts.

Best regards,
Pavel

systems.basesystem

Hi,

I just synced my version of pysystemtrade with the latest version in order to test on the latest data, and experiment with the added trading rules, but now unfortunately I receive an error. It seems that systems.basesystem has changed.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-10-48c5ec1b28eb> in <module>()
----> 1 from systems.provided.futures_chapter15.basesystem import *
      2 from syscore.pdutils import align_to_joint, uniquets, divide_df_single_column
      3 from syscore.dateutils import generate_fitting_dates
      4 from syscore.algos import robust_vol_calc
      5 

H:\pysystemtrade-master\systems\provided\futures_chapter15\basesystem.py in <module>()
      9 
     10 from systems.forecasting import Rules
---> 11 from systems.basesystem import System
     12 from systems.forecast_combine import ForecastCombine
     13 from systems.forecast_scale_cap import ForecastScaleCap

H:\pysystemtrade-master\systems\basesystem.py in <module>()
      9 
     10 from systems.forecasting import Rules
---> 11 from systems.basesystem import System
     12 from systems.forecast_combine import ForecastCombine
     13 from systems.forecast_scale_cap import ForecastScaleCap

ImportError: cannot import name 'System'

thresholding

is ForecastWithThreshold (examples/smallaccountsize) still a valid way to achieve thresholding? is the API still valid?

Feature: progress bars

breakout method

Hi, I am trying to utilise the breakout method you outlined (code below). I notice that when I use this on my own csv data, the code seems to work; however, when I try to use this on pysystemtrade data using e.g., system.rawdata.get_daily_prices("CRUDE_W") it yields error messages. This means that I cannot incorporate the breakout into pysystemtrade. I believe it may be related to the index associated with system.rawdata.get_daily_prices("CRUDE_W"). I have checked and both my own csv data and system.rawdata.get_daily_prices("CRUDE_W") yield pandas dataframes with Datetime indexes. Do you have any advice on how to adapt the breakout code so that it works with pysystemtrade and pysystemtrade data?

Here is the code:

def breakout(x, ws):
    max_x = pd.rolling_max(x, ws, min_periods=min(len(x), int(ws/2)))
    min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))

    sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
    sig = pd.Series(sig, index=x.index)
    sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))

    return sig

def breakout_one_row(idx, x, max_x, min_x):
    r_px = x[idx]
    r_min = min_x[idx]
    r_max = max_x[idx]
    return 40*(r_px - np.mean([r_min, r_max]))/(r_max - r_min)

Here is the error using pysystemtrade:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\pandas\indexes\base.py in get_loc(self, key, method, tolerance)
   1875             try:
-> 1876                 return self._engine.get_loc(key)
   1877             except KeyError:
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4027)()
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:3891)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12408)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12359)()
KeyError: 0
During handling of the above exception, another exception occurred:
KeyError                                  Traceback (most recent call last)
<ipython-input-69-079ba8216ec5> in <module>()
----> 1 breakout(pd.DataFrame(system.rawdata.get_daily_prices("CRUDE_W")), 320)
<ipython-input-27-61d55c59b2f4> in breakout(x, ws)
      3     min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
      4 
----> 5     sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
      6     sig = pd.Series(sig, index=x.index)
      7     sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))
<ipython-input-27-61d55c59b2f4> in <listcomp>(.0)
      3     min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
      4 
----> 5     sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
      6     sig = pd.Series(sig, index=x.index)
      7     sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))
<ipython-input-27-61d55c59b2f4> in breakout_one_row(idx, x, max_x, min_x)
     10 
     11 def breakout_one_row(idx, x, max_x, min_x):
---> 12     r_px = x[idx]
     13     r_min = min_x[idx]
     14     r_max = max_x[idx]
C:\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
   1990             return self._getitem_multilevel(key)
   1991         else:
-> 1992             return self._getitem_column(key)
   1993 
   1994     def _getitem_column(self, key):
C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_column(self, key)
   1997         # get column
   1998         if self.columns.is_unique:
-> 1999             return self._get_item_cache(key)
   2000 
   2001         # duplicate columns & possible reduce dimensionality
C:\Anaconda3\lib\site-packages\pandas\core\generic.py in _get_item_cache(self, item)
   1343         res = cache.get(item)
   1344         if res is None:
-> 1345             values = self._data.get(item)
   1346             res = self._box_item_values(item, values)
   1347             cache[item] = res
C:\Anaconda3\lib\site-packages\pandas\core\internals.py in get(self, item, fastpath)
   3223 
   3224             if not isnull(item):
-> 3225                 loc = self.items.get_loc(item)
   3226             else:
   3227                 indexer = np.arange(len(self.items))[isnull(self.items)]
C:\Anaconda3\lib\site-packages\pandas\indexes\base.py in get_loc(self, key, method, tolerance)
   1876                 return self._engine.get_loc(key)
   1877             except KeyError:
-> 1878                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   1879 
   1880         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4027)()
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:3891)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12408)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12359)()
KeyError: 0

Instrument weights with non pooled scalars

Hi Rob,
with your new code the estimation or cleaning of instrument weights behaves different when using non pooled scalars. The code calculates weights only when scalar values exist for all instruments e.g. US10 and SP500. Before your code changes all weight before 1998 went to US10 as no values existed for SP500. Now no weight is given at all (neither to US10 nor SP500) and weights are calculated when both instruments have scalar values.
Regards
Robooeko

AttributeError: 'System' object has no attribute 'accounts'

Thank you for doing this project. I get the error when I test this example in the introduction,

from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5)
my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5)

Please help.

Crash if no rules are cheap enough for an instrument

Hi Rob,

I am testing new version of the code (0.12.0) and noticed one thing. If there are no cheap enough rules for an instrument available, the calc crashes in syscore\accounting.py. To reproduce the bug, it is enough to increase instrument costs in cost_analysis.csv column Slippage.

Best regards,
Pavel
PS. I am attaching the stack (long read):

Error: ValueError: No objects to concatenate

Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3066, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 2, in
acc_curve_group=system.accounts.portfolio()
File "C:\Project\Project\systems\account.py", line 1769, in portfolio
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1757, in _portfolio
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 1757, in
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 1101, in pandl_for_instrument
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1056, in _pandl_for_instrument
positions = this_stage.get_buffered_position(instrument_code, roundpositions = roundpositions)
File "C:\Project\Project\systems\account.py", line 991, in get_buffered_position
flags="roundpositions%s" % TorF(roundpositions))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 977, in _get_buffered_position
optimal_position=this_stage.get_notional_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 364, in get_notional_position
return self.parent.portfolio.get_notional_position(instrument_code)
File "C:\Project\Project\systems\portfolio.py", line 347, in get_notional_position
"get_notional_position", instrument_code, _get_notional_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 330, in _get_notional_position
idm = this_stage.get_instrument_diversification_multiplier()
File "C:\Project\Project\systems\portfolio.py", line 719, in get_instrument_diversification_multiplier
self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 710, in _get_instrument_div_multiplier
correlation_list_object=this_stage.get_instrument_correlation_matrix()
File "C:\Project\Project\systems\portfolio.py", line 673, in get_instrument_correlation_matrix
self, corr_func, *_corr_params)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, *args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 646, in _get_instrument_correlation_matrix
pandl=this_stage.pandl_across_subsystems().to_frame()
File "C:\Project\Project\systems\portfolio.py", line 776, in pandl_across_subsystems
return self.parent.accounts.pandl_across_subsystems()
File "C:\Project\Project\systems\account.py", line 945, in pandl_across_subsystems
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 933, in _pandl_across_subsystems
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 933, in
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 891, in pandl_for_subsystem
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 860, in _pandl_for_subsystem
positions = this_stage.get_aligned_subsystem_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 257, in get_aligned_subsystem_position
_get_aligned_subsystem_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 249, in _get_aligned_subsystem_position
sspos=this_stage.get_subsystem_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 233, in get_subsystem_position
return self.parent.positionSize.get_subsystem_position(instrument_code)
File "C:\Project\Project\systems\positionsizing.py", line 493, in get_subsystem_position
'get_subsystem_position', instrument_code, _get_subsystem_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\positionsizing.py", line 484, in _get_subsystem_position
forecast = this_stage.get_combined_forecast(instrument_code)
File "C:\Project\Project\systems\positionsizing.py", line 76, in get_combined_forecast
return self.parent.combForecast.get_combined_forecast(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 538, in get_combined_forecast
'get_combined_forecast', instrument_code, _get_combined_forecast, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 512, in _get_combined_forecast
forecast_weights = this_stage.get_forecast_weights(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 392, in get_forecast_weights
'get_forecast_weights', instrument_code, _get_forecast_weights, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 376, in _get_forecast_weights
instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 1092, in get_raw_forecast_weights
self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 1086, in _get_raw_forecast_weights
return this_stage.calculation_of_raw_forecast_weights(instrument_code).weights
File "C:\Project\Project\systems\forecast_combine.py", line 875, in calculation_of_raw_forecast_weights
return self.calculation_of_raw_forecast_weights_for_instrument(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 959, in calculation_of_raw_forecast_weights_for_instrument
self, codes_to_use, weighting_func, pool_costs, *_weighting_params)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, *args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 907, in _calculation_of_raw_forecast_weights
for code in codes_to_use]
File "C:\Project\Project\systems\forecast_combine.py", line 907, in
for code in codes_to_use]
File "C:\Project\Project\systems\forecast_combine.py", line 773, in get_returns_for_optimisation
return self.parent.accounts.pandl_for_instrument_rules_unweighted(instrument_code, rule_list)
File "C:\Project\Project\systems\account.py", line 1311, in pandl_for_instrument_rules_unweighted
delayfill))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1300, in _pandl_for_instrument_rules_unweighted
weighted_flag=False)
File "C:\Project\Project\syscore\accounting.py", line 1323, in init
curve_type="net")
File "C:\Project\Project\syscore\accounting.py", line 1058, in init
(acc_total, capital)=total_from_list(acc_curve_for_type_list, asset_columns, capital)
File "C:\Project\Project\syscore\accounting.py", line 979, in total_from_list
pdframe=acc_list_to_pd_frame(list_of_ac_curves, asset_columns)
File "C:\Project\Project\syscore\accounting.py", line 955, in acc_list_to_pd_frame
ans=pd.concat(list_of_df, axis=1, join="outer")
File "D:\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 834, in concat
copy=copy)
File "D:\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 867, in init
raise ValueError('No objects to concatenate')
ValueError: No objects to concatenate

Correlation Estimate - pysystemtrade/syscore/correlations.py error in fit_period.period_end

Hi,
I updated data in pysystemtrade today, so the last data entry was 2016-09-26.
Each time I run pysytemtrade I get an error during the correlation estimate. The last data point in all of the legacycsv files I have is 2016-09-26 (I have triple checked this). When pysytemtrade goes to estimate the correlations, for some reason it attempts to estimate out to 2016-10-02 even though there is no data after 2016-09-26. My log states:
"Estimating from 2016-01-31 00:00:00 to 2016-10-02 00:00:00"
Why is it trying to estimate out to 2016-10-02?
I believe it is an error in fit_period. I've done this every day for a month and this is the first time I've encountered this. I am using expanding estimates.

Value Error when getting combined forecast

Hello, Rob. Thanks for the book and this repo.

I am running the following code:

from systems.provided.futures_chapter15.estimatedsystem import futures_system

system=futures_system(log_level="on")
system.combForecast.get_combined_forecast("EDOLLAR")

And getting the following error:

Traceback (most recent call last):

  File "<ipython-input-1-15484267b926>", line 1, in <module>
    runfile('C:/Users/paulj/Documents/Upwork/Ravi_Balsa/futures_system.py', wdir='C:/Users/paulj/Documents/Upwork/Ravi_Balsa')

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 880, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/paulj/Documents/Upwork/Ravi_Balsa/futures_system.py", line 6, in <module>
    system.combForecast.get_combined_forecast("EDOLLAR")

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 685, in wrapper
    **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 582, in calc_or_cache
    value = func(this_stage, *args, **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 899, in get_combined_forecast
    raw_multiplied_combined_forecast = self._get_raw_combined_forecast(instrument_code)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 914, in _get_raw_combined_forecast
    instrument_code)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 656, in wrapper
    return func(*args, **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 849, in get_forecast_diversification_multiplier
    return self.get_forecast_diversification_multiplier_estimated(instrument_code)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 685, in wrapper
    **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 582, in calc_or_cache
    value = func(this_stage, *args, **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 838, in get_forecast_diversification_multiplier_estimated
    **div_mult_params)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\syscore\divmultipliers.py", line 92, in diversification_multiplier_from_list
    corrmatrix, weights, **kwargs)

  File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\syscore\divmultipliers.py", line 38, in diversification_mult_single_period
    (float(np.dot(np.dot(weights, corrmatrix), weights.transpose()))**.5),

ValueError: shapes (1,) and (4,1) not aligned: 1 (dim 0) != 4 (dim 0)

That happens in an advanced stage. After the system outputs this:

Calculating forecast correlations over CORN, EDOLLAR, EUROSTX, MXP, US10, V2X
[================================================================================] 100.0%
Estimating correlations

Error after recent changes to algos.py

Hi Rob,

Your book is terrific -- it's the most thorough, logical, and practical one I've ever read on systematic trading and futures. Your blog and EliteTrader posts are also very helpful, and the pysystemtrade code is a great resource. Thank you for sharing your knowledge.

After the latest update to algos.py (in commit 2a68e99), this error started to appear: "'Series' object has no attribute 'columns'." Python and YAML for reproducing the bug are attached, along with a traceback.

The error goes away if algos.py is reverted to the version in the previous commit (f899ab1).

It's possible that the error is related to my using later versions of Python and Pandas (3.4, 0.18) than the ones on which pysystemtrade was developed, but I thought I should mention it for future reference.

-- Michael

test1_python.txt
test1_traceback.txt
test1_yaml.txt

AttributeError: 'BlockManager' object has no attribute 'index'

Hi, I am trying to examine the equity curves of subsystems as per your post "Instrument Weights"
referring to the following code:

import pandas as pd 
instrument_codes=system.get_instrument_list()
pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
        for code in instrument_codes]
pandl=pd.concat(pandl_subsystems, axis=1)
pandl.columns=instrument_codes
pandl=pandl.cumsum().plot()
show()

This yields the following error: AttributeError: 'BlockManager' object has no attribute 'index'

Here is the error message:

AttributeError                            Traceback (most recent call last)
<ipython-input-8-f5d38cdd5c65> in <module>()
      2 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
      3         for code in instrument_codes]
----> 4 pandl=pd.concat(pandl_subsystems, axis=1)
C:\Anaconda3\lib\site-packages\pandas\tools\merge.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
    833                        verify_integrity=verify_integrity,
    834                        copy=copy)
--> 835     return op.get_result()
    836 
    837 
C:\Anaconda3\lib\site-packages\pandas\tools\merge.py in get_result(self)
   1027                 new_data._consolidate_inplace()
   1028 
-> 1029             return (self.objs[0]._from_axes(new_data, self.new_axes)
   1030                     .__finalize__(self, method='concat'))
   1031 
C:\Anaconda3\lib\site-packages\pandas\core\generic.py in _from_axes(cls, data, axes, **kwargs)
    294         # for construction from BlockManager
    295         if isinstance(data, BlockManager):
--> 296             return cls(data, **kwargs)
    297         else:
    298             if cls._AXIS_REVERSED:
C:\Anaconda3\pysystemtrade\syscore\accounting.py in __init__(self, price, percentage, cash_costs, SR_cost, capital, ann_risk_target, weighting, apply_weight_to_costs_only, **kwargs)
    709         """
    710 
--> 711         returns_data=pandl_with_data(price,  capital=capital, ann_risk_target=ann_risk_target, **kwargs)
    712 
    713         (cum_trades, trades, instr_ccy_returns,
C:\Anaconda3\pysystemtrade\syscore\accounting.py in pandl_with_data(price, trades, marktomarket, positions, delayfill, roundpositions, get_daily_returns_volatility, forecast, fx, capital, ann_risk_target, value_of_price_point)
    139     if fx is None:
    140         # assume it's 1.0
--> 141         fx = pd.Series([1.0] * len(price.index),
    142                        index=price.index).to_frame("fx")
    143 
AttributeError: 'BlockManager' object has no attribute 'index'

Is there an error within pysystemtrade or am I missing something?

Issues with install

Hi Rob,

I recently bought your book and have read it once and will soon start reading it again as I think there are a lot of concepts which are going to be really useful as I start looking at trading my own money systematically so thank you for that.

My issue is that when I try and install the package (using python3 setup.py install from the terminal on my mac os x machine - when in the relevant directory) I get the following error:

error: can't copy 'examples/longonly.py': doesn't exist or not a regular file

I know you're not "tech support" but I'm at a loss as to how to fix this as I tried just copying the files into the directory where all the other packages are but it seems to not work either - when I try and import a function in iPython it doesn't like it and says it doesn't exist.

If there is something I'm missing or another resource you can point to to help me out that would be great as I'm at a loss as to how to fix this now.

Thanks

Solution to Error in example1: asimpletradingrule.py

Hi Rob
I really to reallywant to get a grip with Algo trading and your book is such a valuable ressource. I am just starting with your package, however I was having difficulties with the 1st line of code (great strart I know..)
from sysdata.csvdata import csvFuturesData

File "/home/olivier/pysystemtrade/sysdata/csvdata.py", line 15, in
from sysdata.futuresdata import FuturesData

File "/home/olivier/pysystemtrade/sysdata/futuresdata.py", line 512, in
class futuresData(pd.dataFrame):
I am ample to change it once I adopt CamelStyle type of writing on DataFrame (as opposed to dataFrame). But as I am relatively new to the whole Python language I wanted to check with you if this was the right edit?
Thanks again

Position time series contains values on weekend

The function get_instrument_value_vol generate a time series that contains the weekend dates.
That happen because the get_instrument_currency_vol is multiply with the get_fx_rate, this create time series with weekend date.
this is propagate up to:
get_instrument_value_vol
get_volatility_scalar
get_subsystem_position
get_notional_position
get_buffers_for_position

get_buffers_for_position is called within get_buffered_position this generate pos_buffers time series that contains weekend with NaN values.

This time series then pass into apply_buffer that call ffill() on pos_buffers this generate buffered positions the contains positions on the weekend.

I don’t know it that’s a problem or not in the end stats and curve, but it’s just something I notice.

Some more info:
I'm running the simplesystemconfig.yaml with base currency USD, I don’t see the same problem when base currency is GBP

several errors

When I try to load the asimpletradingrule.py , it gives me the following error.

FileNotFoundError Traceback (most recent call last)
C:\Users\jngri\Documents\Python Scripts\pysystemtrade\examples\introduction\asimpletradingrule.py in ()
62
63 import pandas as pd
---> 64 from syscore.algos import robust_vol_calc
65
66

C:\Users\jngri\Anaconda3\lib\site-packages\syscore\algos.py in ()
11
12 from syscore.genutils import str2Bool
---> 13 from systems.defaults import system_defaults
14
15 LARGE_NUMBER_OF_DAYS = 250 * 100 * 100

C:\Users\jngri\Anaconda3\lib\site-packages\systems\defaults.py in ()
24 return default_dict
25
---> 26 system_defaults = get_system_defaults()
27
28 if name == 'main':

C:\Users\jngri\Anaconda3\lib\site-packages\systems\defaults.py in get_system_defaults()
19 """
20 default_file = get_filename_for_package(DEFAULT_FILENAME)
---> 21 with open(default_file) as file_to_parse:
22 default_dict = yaml.load(file_to_parse)
23

FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\jngri\Anaconda3\lib\site-packages\systems\provided\defaults.yaml'

I try to run other notebooks like prebakedsystem and I get this..

ModuleNotFoundError Traceback (most recent call last)
C:\Users\jngri\Documents\Python Scripts\pysystemtrade\examples\introduction\prebakedsystems.py in ()
----> 1 from systems.provided.example.simplesystem import simplesystem
2
3 my_system = simplesystem(log_level="on")
4 print(my_system)
5 print(my_system.portfolio.get_notional_position("EDOLLAR").tail(5))

ModuleNotFoundError: No module named 'systems.provided'

I am on a windows 10 machine using anaconda. I added the environment path for for pysystemtrade and it seems to be having trouble findings things. ALL Required packages are installed Just an FYI- I am teaching myself python, so I could quite possibly be down something wrong.

Repost: AttributeError related to system.accounts

I post the code below. It is the same appeared in the introduction of pysystemtrade.

# coding: utf-8

import syscore

from syscore.algos import robust_vol_calc

import pandas as pd
import numpy as np
import matplotlib as plt
#import pyyaml
import scipy


from sysdata.csvdata import csvFuturesData
data=csvFuturesData()



from systems.provided.example.rules import ewmac_forecast_with_defaults as ewmac
from systems.forecasting import Rules
my_rules=Rules(dict(ewmac=ewmac))
my_rules.trading_rules()

from systems.basesystem import System
my_system=System([my_rules], data)
my_system


# We can now get forecasts:

my_system.rules.get_raw_forecast("EDOLLAR", "ewmac").tail(5)



from systems.forecasting import TradingRule
ewmac_rule=TradingRule(ewmac)
my_rules=Rules(dict(ewmac=ewmac_rule))
ewmac_rule



ewmac_8=TradingRule((ewmac, [], dict(Lfast=8, Lslow=32))) ## as a tuple (function, data, other_args) notice the empty element in the middle
ewmac_32=TradingRule(dict(function=ewmac, other_args=dict(Lfast=32, Lslow=128)))  ## as a dict
my_rules=Rules(dict(ewmac8=ewmac_8, ewmac32=ewmac_32))
my_rules.trading_rules()['ewmac32']


from sysdata.configdata import Config
my_config=Config()
my_config
empty_rules=Rules()
my_config.trading_rules=dict(ewmac8=ewmac_8, ewmac32=ewmac_32)
my_system=System([empty_rules], data, my_config)



from systems.forecast_scale_cap import ForecastScaleCap
## By default we pool esimates across instruments. It's worth telling the system what instruments we want to use:
#
my_config.instruments=["EDOLLAR", "US10", "EDOLLAR", "CORN", "SP500"]

## this parameter ensures we estimate:
my_config.use_forecast_scale_estimates=True

fcs=ForecastScaleCap()
my_system = System([fcs, my_rules], data, my_config)
print(my_system.forecastScaleCap.get_forecast_scalar("EDOLLAR", "ewmac32").tail(5))


my_config.forecast_scalars=dict(ewmac8=5.3, ewmac32=2.65)

## this parameter ensures we don't estimate:
#my_config.use_forecast_scale_estimates=False

## we need a new object
fcs=ForecastScaleCap()

my_system=System([fcs, empty_rules], data, my_config)
my_system.forecastScaleCap.get_capped_forecast("EDOLLAR", "ewmac32")


from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5)
my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5)


from systems.account import Account
my_account = Account()

## let's use naive markowitz to get more interesting results...
my_config.forecast_weight_estimate=dict(method="one_period") 
my_config.use_forecast_weight_estimates=True

combiner = ForecastCombine()
my_system = System([my_account, fcs, my_rules, combiner], data, my_config)

## this is a bit slow, better to know what's going on
my_system.set_logging_level("on")

print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))



my_config.forecast_weights=dict(ewmac8=0.5, ewmac32=0.5)
my_config.forecast_div_multiplier=1.1
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)


from systems.positionsizing import PositionSizing
possizer=PositionSizing()

my_config.percentage_vol_target=25
my_config.notional_trading_capital=500000
my_config.base_currency="GBP"

my_system=System([ fcs, empty_rules, combiner, possizer], data, my_config)

my_system.positionSize.get_subsystem_position("EDOLLAR").tail(5)

file resolution doesn't work in interactive pycharm session

Running Accounts stage snippet in the introduction

Robert, running the script below gives me an error. Full trace is attached. Thanks. -Zhen

from systems.account import Account
my_account = Account()

## let's use naive markowitz to get more interesting results...
my_config.forecast_weight_estimate=dict(method="one_period") 
my_config.use_forecast_weight_estimates=True

combiner = ForecastCombine()
my_system = System([my_account, fcs, my_rules, combiner], data, my_config)

## this is a bit slow, better to know what's going on
my_system.set_logging_level("on")

print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))

Calculating forecast weights for EDOLLAR
Calculating raw forecast weights for EDOLLAR
Calculating capped forecast for CORN ewmac32
Calculating raw forecast CORN for ewmac32
Calculating capped forecast for EDOLLAR ewmac32
Calculating raw forecast EDOLLAR for ewmac32
Calculating capped forecast for SP500 ewmac32
Calculating raw forecast SP500 for ewmac32
Calculating capped forecast for US10 ewmac32
Calculating raw forecast US10 for ewmac32
Calculating capped forecast for CORN ewmac8
Calculating raw forecast CORN for ewmac8
Calculating capped forecast for EDOLLAR ewmac8
Calculating raw forecast EDOLLAR for ewmac8
Calculating capped forecast for SP500 ewmac8
Calculating raw forecast SP500 for ewmac8
Calculating capped forecast for US10 ewmac8
Calculating raw forecast US10 for ewmac8
Only this set of rules ['ewmac32'] is cheap enough to trade for EDOLLAR
Only this set of rules ['ewmac32', 'ewmac8'] is cheap enough to trade for CORN
Only this set of rules ['ewmac32', 'ewmac8'] is cheap enough to trade for SP500
Only this set of rules ['ewmac32', 'ewmac8'] is cheap enough to trade for US10
Calculating raw forecast weights for EDOLLAR, over EDOLLAR
Calculating pandl for instrument rules for EDOLLAR
Calculating pandl for instrument forecast for EDOLLAR ewmac32
Using cost multiplier on optimisation of 0.00
Optimising...
Optimising for data from 1983-10-02 00:00:00 to 1984-10-31 00:00:00
Optimising for data from 1984-10-31 00:00:00 to 1985-10-31 00:00:00
Optimising for data from 1985-10-31 00:00:00 to 1986-10-31 00:00:00
Optimising for data from 1986-10-31 00:00:00 to 1987-10-31 00:00:00
Optimising for data from 1987-10-31 00:00:00 to 1988-10-31 00:00:00
Optimising for data from 1988-10-31 00:00:00 to 1989-10-31 00:00:00
Optimising for data from 1989-10-31 00:00:00 to 1990-10-31 00:00:00
Optimising for data from 1990-10-31 00:00:00 to 1991-10-31 00:00:00
Optimising for data from 1991-10-31 00:00:00 to 1992-10-31 00:00:00
Optimising for data from 1992-10-31 00:00:00 to 1993-10-31 00:00:00
Optimising for data from 1993-10-31 00:00:00 to 1994-10-31 00:00:00
Optimising for data from 1994-10-31 00:00:00 to 1995-10-31 00:00:00
Optimising for data from 1995-10-31 00:00:00 to 1996-10-31 00:00:00
Optimising for data from 1996-10-31 00:00:00 to 1997-10-31 00:00:00
Optimising for data from 1997-10-31 00:00:00 to 1998-10-31 00:00:00
Optimising for data from 1998-10-31 00:00:00 to 1999-10-31 00:00:00
Optimising for data from 1999-10-31 00:00:00 to 2000-10-31 00:00:00
Optimising for data from 2000-10-31 00:00:00 to 2001-10-31 00:00:00
Optimising for data from 2001-10-31 00:00:00 to 2002-10-31 00:00:00
Optimising for data from 2002-10-31 00:00:00 to 2003-10-31 00:00:00
Optimising for data from 2003-10-31 00:00:00 to 2004-10-31 00:00:00
Optimising for data from 2004-10-31 00:00:00 to 2005-10-31 00:00:00
Optimising for data from 2005-10-31 00:00:00 to 2006-10-31 00:00:00
Optimising for data from 2006-10-31 00:00:00 to 2007-10-31 00:00:00
Optimising for data from 2007-10-31 00:00:00 to 2008-10-31 00:00:00
Optimising for data from 2008-10-31 00:00:00 to 2009-10-31 00:00:00
Optimising for data from 2009-10-31 00:00:00 to 2010-10-31 00:00:00
Optimising for data from 2010-10-31 00:00:00 to 2011-10-31 00:00:00
Optimising for data from 2011-10-31 00:00:00 to 2012-10-31 00:00:00
Optimising for data from 2012-10-31 00:00:00 to 2013-10-31 00:00:00
Optimising for data from 2013-10-31 00:00:00 to 2014-10-31 00:00:00
Optimising for data from 2014-10-31 00:00:00 to 2015-10-31 00:00:00
Optimising for data from 2015-10-31 00:00:00 to 2016-10-31 00:00:00
Optimising for data from 2016-10-31 00:00:00 to 2016-11-13 00:00:00
Applying cost weighting to optimisation results
ewmac32
2016-11-07 1.0
2016-11-08 1.0
2016-11-09 1.0
2016-11-10 1.0
2016-11-11 1.0
Calculating forecast div multiplier for EDOLLAR
Calculating forecast correlations over EDOLLAR

AttributeError Traceback (most recent call last)
in ()
13
14 print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
---> 15 print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_diversification_multiplier(self, instrument_code)
764 forecast_div_multiplier = self.parent.calc_or_cache(
765 'get_forecast_diversification_multiplier', instrument_code, _get_forecast_div_multiplier,
--> 766 self)
767 return forecast_div_multiplier
768

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_div_multiplier(system, instrument_code, this_stage)
752
753 correlation_list_object = this_stage.get_forecast_correlation_matrices(
--> 754 instrument_code)
755 weight_df = this_stage.get_forecast_weights(instrument_code)
756

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_correlation_matrices(self, instrument_code)
705 'get_forecast_correlation_matrices', instrument_code_ref, label,
706 _get_forecast_correlation_matrices,
--> 707 self, codes_to_use, corr_func, **corr_params)
708
709 return forecast_corr_list

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache_nested(self, itemname, instrument_code, keyname, func, this_stage, *args, **kwargs)
684 this_stage,
685 *args,
--> 686 **kwargs)
687 self.set_item_in_cache(value, cache_ref, instrument_code, keyname)
688

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_correlation_matrices(system, NotUsed1, NotUsed2, this_stage, codes_to_use, corr_func, **corr_params)
668
669 return corr_func(forecast_data, log=self.log.setup(
--> 670 call="correlation"), **corr_params)
671
672 # Get some useful stuff from the config

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\syscore\correlations.py in init(self, data, log, frequency, date_method, rollyears, dict_group, boring_offdiag, cleaning, **kwargs)
278 column_names = list(data.columns)
279
--> 280 data = frequency.resample.last()
281
282 # Generate time periods

AttributeError: 'str' object has no attribute 'resample'

Optimisation.py error

In [50]: pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
...: for code in instrument_codes]

TypeError Traceback (most recent call last)
in ()
1 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
----> 2 for code in instrument_codes]

in (.0)
1 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
----> 2 for code in instrument_codes]

TypeError: pandl_for_subsystem() got an unexpected keyword argument 'percentage'

Need to check turnover calculation is correct

fyi - pandas - ewma changed in 0.18

Hi Rob,
I've bought your book and started looking at the code, working through the examples. Since I normally use Python 2.7 I set up a fresh environment with the latest... 3.5 and everything else. In the introduction there is an implementation of the ewma forecast which accesses a pandas function .ewma(...). It turns out that in pandas 0.18 they have changed the functionality so that you have to create an .ewm object then calculate its .mean(). See --> http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.ewm.html.
When I downgraded to 0.17.1 this problem went away, so you might want to flag that as a requirement. Currently pysystemtrade/requirements.txt states pandas >= 0.17.0. (One other note - I reset to 3.4 to dup your environment as best I could).
Getting another error also but I'll comment on that in the file where the error is occurring.
Best
Davis

Phantom costs

If I run Ch. 15 trading system I get two datasets of my interest: gross and net. You documented it well, and I find the idea very good. Then I delete "costs_analysis.csv" file and I assume that in this case there will be no costs included into the calculations. Indeed, I get the warning when I run calculations - "Cost file not found sysdata\legacycsv\costs_analysis.csv". That is great. But when I look into .net.daily.stats() and gross.daily.stats()stats, I see different numbers. Assuming that the costs are zeroed, I expected those stats to be identical. Am I wrong? I am attaching the code and output in the end of this email.

from systems.provided.futures_chapter15.basesystem import futures_system
from matplotlib.pyplot import show

**system = futures_system(log_level="on")

acc_curve_group=system.accounts.portfolio(

percentage

True

)

(

acc_curve_group

.gross.daily.stats())

(

acc_curve_group

.net.daily.stats())
system = futures_system(log_level="on")

acc_curve_group=system.accounts.portfolio(

percentage

True

)

(

acc_curve_group

.gross.daily.stats())

(

acc_curve_group

.net.daily.stats())

possible issue with base_capital not being assigned

Hi Rob - I am walking through your introduction.md document. Intermittently (when I run from scratch) when I run line 134-136:-
from syscore.accounting import accountCurve
account = accountCurve(price, forecast=ewmac, percentage=True)
account.percent().stats()
I get an error:-
pysystemtrade/syscore/accounting.py in resolve_capital(ts_to_scale_to, capital, ann_risk_target)
931
932 ## might be a float or a Series, depending on capital
--> 933 daily_risk_capital = base_capital * ann_risk_target / ROOT_BDAYS_INYEAR
934
935 if type(base_capital) is float or type(base_capital) is int:

UnboundLocalError: local variable 'base_capital' referenced before assignment

If I make a change to the resolve_capital function within accounting.py as follows, that seems to fix it.
if capital is None:
capital=DEFAULT_CAPITAL
base_capital = copy(capital) #add this line
else:
base_capital = copy(capital)

Full disclosure - not that confident that this is the cause of my error - rusty in programming and quite new to python.

Problem with asimpletradingrule.py

Hi there, when I run examples/introduction/asimpletradingrule.py, the graphs are fine, but I also get,

File "../..\syscore\accounting.py", line 932, in resolve_capital
daily_risk_capital = base_capital * ann_risk_target / ROOT_BDAYS_INYEAR
UnboundLocalError: local variable 'base_capital' referenced before assignment

When I pass capital=10000 to accountCurve, I received a segmentation fault (at least on Windows).
After some tries, I was able to narrow it down to accountCurveSingleElementOneFreq class, to stats(self) function; in the loop while the stats are being calculated, the avg_drawdown stat, time_in_drawdown, calmar, avg_return_to_drawdown are the stats that are causing problems. If these are skipped, then I get the output fine, it says 0.5084 for Sharpe which matches the output in the doc.

I have the latest code.

Maybe use concurrent.futures for speed?

I thought I'd let you know that using concurrent.futures ProcessPoolExecutor sped up bootstrap performance by about 10x. I prep all the samples in advance and then bootstrap them in parallel.

If you're interested in code, it's here https://gist.github.com/chrism2671/256ec51515b6849dd695f15fb28caecf (start reading at def bootstrap() )

Questions: time frames, commissions

Dear Rob, Dear colleagues,

Thank you very much such a good library. I have some questions on its usage.

I have some 10 trading rules, that work well on 15-minute data bars time frame. I see that the library works with daily data. I am wondering if the library has some feature that would allow me to build my portfolio of 20 cheap instruments with 10 trading rules using 15 minute pricing data bars. If not, what would be your recommendation on how I could add this feature myself?
In my case above, I have noticed that the best performance I am getting not when I trade to update my positions every 15 minutes (even including position inertia), but at some fixed time frame of larger scale: e.g. every 4 hours. That means that I trade every 4 hours, and when it is time to trade, I take the latest position info available (16 of them is produced per every 4 hour interval). Wanted to ask your thoughts if this is possible to have with current implementation, and if not, what would be your recommendation on how I could add this feature myself while working with your library?
And the last question is about commissions. Many brokers, including IB, have non-flat commissions for some of the markets that depend on monthly volumes of the traded instruments and other factors. As far as I understand, currently I can only specify some flat numbers (estimates) in the library configuration file. I have checked that with one of my systematic portfolios if I specify the flat numbers, then the resulting P/L is effectively eaten out by the commissions. If I model the commissions more precisely and separately (after the gross curves are produced), I see that commissions eat out only 12% of the performance. The difference is remarkable. But I believe it is not be the right approach because this information about commissions was not used by the library itself during its calculations. Wanted to ask you what would be your recommendation on how I could model more precise non-linear commissions (to stay in style of your library design)?

I would really appreciate your thoughts on those three points.

Best regards,
Max

KeyError: 'private'

I am trying to use pysystemtrade to backtest your estimated system across all instruments you include in the csv files (on Python 3 in Windows 10).

I create new config in pysystemtrade/private/my_system/futuresestimateconfigme.yaml which exactly copies your pysystemtrade\systems\provided\futures_chapter15\futuresestimateconfig.yaml file, but adds in all of the instruments in the last item in the yaml file as a list. instruments: ['CRUDE_W', 'GOLD', 'PLAT', 'CORN', 'KR3', 'SOYBEAN', 'BOBL', 'SHATZ'...etc]
I then run the following code:

from sysdata.configdata import Config
from systems.provided.futures_chapter15.estimatedsystem import futures_system
my_config = Config("private.my_system.futuresestimateconfigme.yaml")

Each time I try to run this, I obtain the following 'KeyError: 'private' .

I have tried this with and without __init__.py files in the 'private' and 'my_system' folders.

What am I missing?

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-11-24c926ebb4e0> in <module>()
----> 1 my_config = Config("private.my_system.futuresestimateconfigme.yaml")

C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\sysdata\configdata.py in __init__(self, config_object)
     52                 self._create_config_from_item(config_item)
     53         else:
---> 54             self._create_config_from_item(config_object)
     55 
     56         ## this will normally be overriden by the base system

C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\sysdata\configdata.py in _create_config_from_item(self, config_item)
     65         elif isinstance(config_item, str):
     66             # must be a file YAML'able, from which we load the
---> 67             filename = get_filename_for_package(config_item)
     68             with open(filename) as file_to_parse:
     69                 dict_to_parse = yaml.load(file_to_parse)

C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\syscore\fileutils.py in get_filename_for_package(name_with_dots)
     24         path_as_list[-1] = path_as_list[-2] + "." + path_as_list.pop()
     25 
---> 26     return get_pathname_for_package_from_list(path_as_list)
     27 
     28 

C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\syscore\fileutils.py in get_pathname_for_package_from_list(path_as_list)
     54     package_name = path_as_list[0]
     55     paths_or_files = path_as_list[1:]
---> 56     d = os.path.dirname(sys.modules[package_name].__file__)
     57 
     58     if len(paths_or_files) == 0:

KeyError: 'private'

get_instrument_price

Hi,
I have cloned pysystemtrade and am examining your (ipynb) example in examples/introduction/asimpletradingrule.ipynb. When running price = data.get_instrument_price(instrument_code) I receive the error AttributeError: 'csvFuturesData' object has no attribute 'get_instrument_price'
When I examined csvFuturesData I see a method called get_daily_price, which I believe should be get_instrument_price. Is this correct?

Capital and position sizing

I’m not sure if this is an issue or not.

When daily cash volatility trader is calculated the trading capital is constant throughout time. It doesn’t take in account history profit and loses.

Another thing regarding capital is that the system ignore buying power when generate positions. And in some scenarios you end up buying more than your capital allow.

AttributeError related to system.accounts

I cannot figure out how to solve this when running the script below. The full trace is attached. Thanks.

my_config.forecast_weights=dict(ewmac8=0.5, ewmac32=0.5) my_config.forecast_div_multiplier=1.1 my_system=System([fcs, empty_rules, combiner], data, my_config) my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)

Adding config defaults
Calculating combined forecast for EDOLLAR
Calculating forecast weights for EDOLLAR
Calculating raw forecast weights for EDOLLAR

AttributeError Traceback (most recent call last)
in ()
2 my_config.forecast_div_multiplier=1.1
3 my_system=System([fcs, empty_rules, combiner], data, my_config)
----> 4 my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_combined_forecast(self, instrument_code)
541
542 combined_forecast = self.parent.calc_or_cache(
--> 543 'get_combined_forecast', instrument_code, _get_combined_forecast, self)
544 return combined_forecast
545

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_combined_forecast(system, instrument_code, this_stage)
513 instrument_code=instrument_code)
514
--> 515 forecast_weights = this_stage.get_forecast_weights(instrument_code)
516 rule_variation_list = list(forecast_weights.columns)
517

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_weights(self, instrument_code)
392
393 forecast_weights = self.parent.calc_or_cache(
--> 394 'get_forecast_weights', instrument_code, _get_forecast_weights, self)
395 return forecast_weights
396

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_weights(system, instrument_code, this_stage)
375
376 forecast_weights = this_stage.get_raw_forecast_weights(
--> 377 instrument_code)
378 rule_variation_list = list(forecast_weights.columns)
379 forecasts = this_stage.get_all_forecasts(

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_raw_forecast_weights(self, instrument_code)
1132 'get_raw_forecast_weights', instrument_code,
1133 _get_raw_forecast_weights,
-> 1134 self)
1135
1136 return raw_forecast_weights

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_raw_forecast_weights(system, instrument_code, this_stage)
1126
1127 return this_stage.calculation_of_raw_forecast_weights(
-> 1128 instrument_code).weights
1129
1130 ##

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in calculation_of_raw_forecast_weights(self, instrument_code)
894 # could still be using pooled returns
895 return self.calculation_of_raw_forecast_weights_for_instrument(
--> 896 instrument_code)
897
898 def calculation_of_raw_forecast_weights_for_instrument(

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in calculation_of_raw_forecast_weights_for_instrument(self, instrument_code)
976 if pooling_returns:
977 # find set of instruments with same trading rules as I have
--> 978 codes_to_use = self.has_same_cheap_rules_as_code(instrument_code)
979 else:
980 codes_to_use = [instrument_code]

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in has_same_cheap_rules_as_code(self, instrument_code)
861 """
862
--> 863 my_rules = self.apply_cost_weighting(instrument_code)
864 instrument_list = self.parent.get_instrument_list()
865

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in apply_cost_weighting(self, instrument_code)
846 'apply_cost_weighting', instrument_code,
847 _apply_cost_weighting,
--> 848 self, ceiling_cost_SR)
849
850 return cheap_rules

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _apply_cost_weighting(system, instrument_code, this_stage, ceiling_cost_SR)
824 rule_list = this_stage.get_trading_rule_list(instrument_code)
825 SR_cost_list = [this_stage.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
--> 826 for rule_variation_name in rule_list]
827
828 cheap_rule_list = [rule_name for (rule_name, rule_cost) in zip(rule_list, SR_cost_list)

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in (.0)
824 rule_list = this_stage.get_trading_rule_list(instrument_code)
825 SR_cost_list = [this_stage.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
--> 826 for rule_variation_name in rule_list]
827
828 cheap_rule_list = [rule_name for (rule_name, rule_cost) in zip(rule_list, SR_cost_list)

C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_SR_cost_for_instrument_forecast(self, instrument_code, rule_variation_name)
804 """
805
--> 806 return self.parent.accounts.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
807
808 def apply_cost_weighting(self, instrument_code):

AttributeError: 'System' object has no attribute 'accounts'

generatesystems.py

under "small account size" and for the above referenced file:

from systems.provided.futures_chapter15.estimatedsystem import PortfoliosEstimated

ImportError: cannot import name 'PortfoliosEstimated'

P&L over weekends

https://www.elitetrader.com/et/threads/oh-no-not-another-python-backtester.296589/page-4#post-4391887

Correlations.py RuntimeWarning

When trying to run the 'optimisation.py':

In [86]: print(system.accounts.portfolio().stats())
Calculating pandl for portfolio
Calculating instrument div. multiplier
Calculating instrument correlations
C:\Anaconda3\lib\site-packages\systems\portfolio.py:665: FutureWarning:
.resample() is now a deferred operation
You called diff(...) on this deferred object which materialized it into a dataframe
by implicitly taking the mean. Use .resample(...).mean() instead
pandl = pandl.cumsum().resample(frequency).diff()
Correlation estimate
C:\Anaconda3\lib\site-packages\syscore\correlations.py:181: RuntimeWarning: invalid value encountered in less
corrmat[corrmat < 0] = 0.0

Example for 4th example for user guide doesn't work: "How do I....See how profitable a backtest was"

Traceback (most recent call last):
File "experiment2.py", line 6, in
system.accounts.portfolio().pandl_for_instrument("US10").percent().stats() #

produce % statistics for a 10 year bond

File "C:\Users\xxxxx\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-
packages\pandas\core\generic.py", line 2672, in getattr
return object.getattribute(self, name)
AttributeError: 'accountCurveGroup' object has no attribute 'pandl_for_instrument'

One possibility is it might be my conda environment (my Python3 is conda installed, my native installation is Python 2.7) causing an error - any one facing a similar issue?

ValueError: shapes (1,) and (2,1) not aligned: 1 (dim 0) != 2 (dim 0)

Hi Rob, while running the simplesystem.py, it got interrupted halfway and threw out this error above. I google to search for it and it seems that the data set is not in the correct form? Am I heading in the right direction to resolve this? Thanks.

Instrument Weights Examples

Hi Rob,
tested your new code with the breakout example and the standard estimated system example using your defaults. In both cases the resulting instrument weights seem to be incorrect.

Code:
from systems.provided.futures_chapter15.estimatedsystem import futures_system
from sysdata.configdata import Config
my_config=Config("examples.breakout.breakoutfuturesestimateconfig.yaml")
system=futures_system(config=my_config)
system.set_logging_level("on")

In your previous version a similar issue occurred when increasing the notional trading capital (above $1M). Now it seems to occur independent of the trading capital chosen. I tested with your default and 100000.

Regards
robooeko

Error calculating covariance matrix after downloading latest version of Anaconda/Python

Hi,

I just downloaded the latest version of Anaconda/python and have received the following error in pysystemtrade. Any thoughts on debugging this?

Estimating from 2001-01-31 00:00:00 to 2002-01-31 00:00:00
C:\Users\me\Google Drive\pysystemtrade\syscore\correlations.py:175: RuntimeWarning: invalid value encountered in less
corrmat[corrmat<0]=0.0

ValueError Traceback (most recent call last)
in ()
16 = System([Account(), Portfolios(), PositionSizing(), FuturesRawData(), ForecastCombine(), ForecastScaleCap(), Rules()], csvFuturesData(), config_)
17 .set_logging_level("on")
---> 18 _portfolio = File.accounts.portfolio(delayfill=False)

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in portfolio(self, percentage, delayfill, roundpositions)
1219 port_pandl = self.parent.calc_or_cache(
1220 itemname, ALL_KEYNAME, _portfolio, self, percentage, delayfill,
-> 1221 roundpositions)
1222
1223 return port_pandl

C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _portfolio(system, not_used, this_stage, percentage, delayfill, roundpositions)
1208 percentage=percentage,
1209 delayfill=delayfill,
-> 1210 roundpositions=roundpositions) for instrument_code in instruments]
1211
1212 port_pandl = accountCurveGroup(port_pandl, instruments)

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in (.0)
1208 percentage=percentage,
1209 delayfill=delayfill,
-> 1210 roundpositions=roundpositions) for instrument_code in instruments]
1211
1212 port_pandl = accountCurveGroup(port_pandl, instruments)

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in pandl_for_instrument(self, instrument_code, percentage, delayfill, roundpositions)
884 TorF(percentage), TorF(delayfill), TorF(roundpositions))
885 instr_pandl = self.parent.calc_or_cache(
--> 886 itemname, instrument_code, _pandl_for_instrument, self, percentage, delayfill, roundpositions)
887
888 return instr_pandl

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _pandl_for_instrument(system, instrument_code, this_stage, percentage, delayfill, roundpositions)
844
845 price = this_stage.get_daily_price(instrument_code)
--> 846 positions = this_stage.get_buffered_position(instrument_code, roundpositions = roundpositions)
847 fx = this_stage.get_fx_rate(instrument_code)
848 value_of_price_point = this_stage.get_value_of_price_move(

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in get_buffered_position(self, instrument_code, roundpositions)
775
776 buffered_position = self.parent.calc_or_cache(
--> 777 itemname, instrument_code, _get_buffered_position, self, roundpositions)
778
779 return buffered_position

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _get_buffered_position(system, instrument_code, this_stage, roundpositions)
761
762 this_stage.log.msg("Calculating buffered positions")
--> 763 optimal_position=this_stage.get_notional_position(instrument_code)
764 pos_buffers=this_stage.get_buffers_for_position(instrument_code)
765 trade_to_edge=system.config.buffer_trade_to_edge

C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in get_notional_position(self, instrument_code)
276 >>>
277 """
--> 278 return self.parent.portfolio.get_notional_position(instrument_code)
279
280

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_notional_position(self, instrument_code)
341
342 notional_position = self.parent.calc_or_cache(
--> 343 "get_notional_position", instrument_code, _get_notional_position, self)
344 return notional_position
345

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_notional_position(system, instrument_code, this_stage)
321 instrument_code=instrument_code)
322
--> 323 idm = this_stage.get_instrument_diversification_multiplier()
324 instr_weights = this_stage.get_instrument_weights()
325 subsys_position = this_stage.get_subsystem_position(

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_instrument_diversification_multiplier(self)
641 instrument_div_multiplier = self.parent.calc_or_cache(
642 'get_instrument_diversification_multiplier', ALL_KEYNAME, _get_instrument_div_multiplier,
--> 643 self)
644 return instrument_div_multiplier
645

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_instrument_div_multiplier(system, NotUsed, this_stage)
630 idm_func=resolve_function(div_mult_params.pop("func"))
631
--> 632 correlation_list_object=this_stage.get_instrument_correlation_matrix()
633 weight_df=this_stage.get_instrument_weights()
634

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_instrument_correlation_matrix(self)
593 'get_instrument_correlation_matrix', ALL_KEYNAME,
594 _get_instrument_correlation_matrix,
--> 595 self, corr_func, **corr_params)
596
597 return forecast_corr_list

C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_instrument_correlation_matrix(system, NotUsed, this_stage, corr_func, **corr_params)
576 pandl=pandl.cumsum().resample(frequency).diff()
577
--> 578 return corr_func(pandl, log=this_stage.log.setup(call="correlation"), **corr_params)
579
580 ## Get some useful stuff from the config

C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in init(self, data, log, frequency, date_method, rollyears, dict_group, boring_offdiag, cleaning, **kwargs)
306
307 # means we can use earlier correlations with sensible values
--> 308 corrmat=clean_correlation(corrmat, corr_with_no_data, must_haves)
309
310 corr_list.append(corrmat)

C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in clean_correlation(corrmat, corr_with_no_data, must_haves)
99 size_range=range(corrmat.shape[0])
100
--> 101 avgcorr=get_avg_corr(corrmat)
102
103 def _good_correlation(i,j,corrmat, avgcorr, must_haves, corr_with_no_data):

C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in get_avg_corr(sigma)
24 """
25 new_sigma=copy(sigma)
---> 26 np.fill_diagonal(new_sigma,np.nan)
27 if np.all(np.isnan(new_sigma)):
28 return np.nan

C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\index_tricks.py in fill_diagonal(a, val, wrap)
757 """
758 if a.ndim < 2:
--> 759 raise ValueError("array must be at least 2-d")
760 end = None
761 if a.ndim == 2:

ValueError: array must be at least 2-d`

Quality

Hello Rob,

I have noticed that this project has ups and downs in terms of test pass rate and test coverage. Pass rate is often not 100%. Currently 16 tests are red. Do you have any plans to improve software quality of the project? e.g. higher coverage, CI (most importantly), regression tests, etc. I am not a python expert, but I am sure those things will be valued by end-users.

My environment: Win10, Python 3.5, Pandas 0.18.0, Matplotlib 1.5.1, pyyaml 3.11, numpy 1.10.4, scipy 0.17.0.

Red tests:

Best regards,
Pavel

Correlation not working

I am finding that the correlation module is not working as expected. It's likely to be me but I have checked just using 2 instruments and get a one dimensional array with
'ans = system.portfolio.get_instrument_correlation_matrix()' whereas before it worked beautifully.

use_SR_costs: False yields error when using pandl_for_subsystem

When setting use_SR_costs to false, an error was popping up when using pandl_for_subsystem, I think because SR_cost would not be defined in that situation. I seemed to have fixed it by doing the following:

if SR_cost_per_turnover is not None:
    SR_cost = self.subsystem_SR_costs(instrument_code, roundpositions)
else:
    SR_cost = None

Regards,
Tony

robcarver17 / pysystemtrade Goto Github PK

pysystemtrade's People

Contributors

Stargazers

Watchers

Forkers

pysystemtrade's Issues

I post the code below. It is the same appeared in the introduction of pysystemtrade.

Adding config defaults Calculating combined forecast for EDOLLAR Calculating forecast weights for EDOLLAR Calculating raw forecast weights for EDOLLAR

produce % statistics for a 10 year bond

Estimating from 2001-01-31 00:00:00 to 2002-01-31 00:00:00 C:\Users\me\Google Drive\pysystemtrade\syscore\correlations.py:175: RuntimeWarning: invalid value encountered in less corrmat[corrmat<0]=0.0

Recommend Projects

Recommend Topics

Recommend Org

Adding config defaults
Calculating combined forecast for EDOLLAR
Calculating forecast weights for EDOLLAR
Calculating raw forecast weights for EDOLLAR

Estimating from 2001-01-31 00:00:00 to 2002-01-31 00:00:00
C:\Users\me\Google Drive\pysystemtrade\syscore\correlations.py:175: RuntimeWarning: invalid value encountered in less
corrmat[corrmat<0]=0.0