anfederico/clairvoyant

Software designed to identify and monitor social/historical cues for short term stock movement

clairvoyant's Issues

Could you supply the file("SBUX.csv")?

I am very interetes in your programe,but there is a little difficult for me to get the file in China. Could you please send the file("SUBX.csv") to my email("[email protected]")?

#HELP# Some errors appeared unusually

Hi，
Some errors appeared unusually.

I used the example code to test environment. But, when I use the library, SyntaxError: invalid syntax note appeared.

charmve@charmve-virtual-machine:~/Clairvoyant$ python test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from clairvoyant.engine import Backtest
  File "/home/charmve/.local/lib/python2.7/site-packages/clairvoyant/engine.py", line 16
    self.svc = SVC(**kwargs, probability=True)
                           ^
SyntaxError: invalid syntax

test.py

from clairvoyant.engine import Backtest
import pandas as pd

features  = ["EMA", "SSO"]   # Financial indicators of choice
trainStart = 0               # Start of training period
trainEnd   = 700             # End of training period
testStart  = 701             # Start of testing period
testEnd    = 1000            # End of testing period
buyThreshold  = 0.65         # Confidence threshold for predicting buy (default = 0.65) 
sellThreshold = 0.65         # Confidence threshold for predicting sell (default = 0.65)
continuedTraining = False    # Continue training during testing period? (default = false)

# Initialize backtester
backtest = Backtest(features, trainStart, trainEnd, testStart, testEnd, buyThreshold, sellThreshold, continuedTraining)

# A little bit of pre-processing
data = pd.read_csv("SBUX.csv", date_parser=['date'])
data = data.round(3)                                    

# Start backtesting and optionally modify SVC parameters
# Available paramaters can be found at: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
backtest.start(data, kernel='rbf', C=1, gamma=10)
backtest.conditions()
backtest.statistics()  
backtest.visualize('SBUX')

Bug in predict_proba?

Anthony,

I suspect there may be an upstream bug in SVC.predict_proba. I believe it outputs the classifier probabilities in reverse order than what is stated in their documentation. I discovered this when setting my training period equal to my testing period. I would expect that you should get highly accurate buy and sells if you do this. However,

1. Backtesting was producing the opposite recommendations..

2. So I tried updating the Predict function to use model.predict_proba([Xs])[0] consistent with my suspicions.

3. Now I'm getting correct recommendations.

Here's my data.

Note: I do not have a lot of data so I'm breaking up the modeling frequency into 30 minute intervals.

IndexError: single positional indexer is out-of-bounds

Hi,
I use the data.csv and following indicators in the test code, but get above error.
features = ["X1", "X2"]

here is the data.head():

date open high low close volume X1 X2
0 4/15/16 60.24 60.625 60.010 60.51 5965310 59.959 0.268
1 4/18/16 60.69 61.070 60.355 60.89 7228573 60.048 0.009
2 4/19/16 61.16 61.250 60.480 60.90 7283570 60.129 0.505
3 4/20/16 61.04 61.430 60.850 60.90 5558770 60.203 1.190
4 4/21/16 60.90 61.100 60.480 60.64 12799083 60.244 2.586

Error when running example: ufunc 'isfinite' not supported for the input types

I'm trying to run the example, though it has been modified as per the guidance in other issues. Here is the code:

from clairvoyant.engine import Backtest
import pandas as pd

features  = ["X1", "X2"]   # Financial indicators of choice
trainStart = 0               # Start of training period
trainEnd   = 100             # End of training period
testStart  = 200             # Start of testing period
testEnd    = 300            # End of testing period
buyThreshold  = 0.65         # Confidence threshold for predicting buy (default = 0.65) 
sellThreshold = 0.65         # Confidence threshold for predicting sell (default = 0.65)
continuedTraining = False    # Continue training during testing period? (default = false)

# Initialize backtester
backtest = Backtest(features, trainStart, trainEnd, testStart, testEnd, buyThreshold, sellThreshold, continuedTraining)

# A little bit of pre-processing
data = pd.read_csv("clair-data.csv", date_parser=['date'])
data = data.round(3)                                    

# Start backtesting and optionally modify SVC parameters
# Available paramaters can be found at: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html
backtest.start(data, kernel='rbf', C=1, gamma=10)
backtest.conditions()
backtest.statistics()  
backtest.visualize('X1','X2')

and this is the output I'm getting:

------------ Data Features ------------

X1: X1
X2: X2

---------------------------------------

----------- Model Arguments -----------

kernel: rbf
C: 1
gamma: 10

---------------------------------------

---------  Engine Conditions ----------

Training: 4/15/16 -- 9/7/16
Testing:  1/31/17 -- 6/23/17
Buy Threshold: 65.0%
Sell Threshold: 65.0%
Continued Training: False

---------------------------------------

------------- Statistics --------------

Total Buys: 0
Buy Accuracy: 0.0%
Total Sells: 11
Sell Accuracy: 27.27%

---------------------------------------

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-480e8274c951> in <module>
     23 backtest.conditions()
     24 backtest.statistics()
---> 25 backtest.visualize('X1','X2')

~/.local/lib/python3.6/site-packages/clairvoyant/engine.py in visualize(self, name, width, height, stepsize)
    170         xx, yy = meshgrid(arange(x_min, x_max, stepsize), arange(y_min, y_max, stepsize))
    171 
--> 172         pyplot.figure(figsize=(width, height))
    173         cm = pyplot.cm.RdBu  # Red/Blue gradients
    174         rb = ListedColormap(['#FF312E', '#6E8894']) # Red = 0 (Negative) / Blue = 1 (Positve)

~/.local/lib/python3.6/site-packages/matplotlib/pyplot.py in figure(num, figsize, dpi, facecolor, edgecolor, frameon, FigureClass, clear, **kwargs)
    543                                         frameon=frameon,
    544                                         FigureClass=FigureClass,
--> 545                                         **kwargs)
    546 
    547         if figLabel:

~/.local/lib/python3.6/site-packages/matplotlib/backend_bases.py in new_figure_manager(cls, num, *args, **kwargs)
   3256         from matplotlib.figure import Figure
   3257         fig_cls = kwargs.pop('FigureClass', Figure)
-> 3258         fig = fig_cls(*args, **kwargs)
   3259         return cls.new_figure_manager_given_figure(num, fig)
   3260 

~/.local/lib/python3.6/site-packages/matplotlib/figure.py in __init__(self, figsize, dpi, facecolor, edgecolor, linewidth, frameon, subplotpars, tight_layout, constrained_layout)
    346             frameon = rcParams['figure.frameon']
    347 
--> 348         if not np.isfinite(figsize).all():
    349             raise ValueError('figure size must be finite not '
    350                              '{}'.format(figsize))

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Do you have any idea what this could be? Is there other things that I need to adjust to run the example?

Stocks.csv format

I created a Stocks.csv file for the first test code you had on the readme page, but I can't figure out the format required for the script to work. Do you have a template?

KeyError: 'EMA'

Hi! Thank you for uploading this project.

I am having an issue running Backtest using the code in your README.md file. Using all of your original code and your example data, I receive the following error (I've edited out the traceback parts):

TypeError: 'str' object cannot be interpreted as an integer

  File "pandas\_libs\index.pyx", line 106, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 114, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'EMA'

Am I setting up the features incorrectly or do you know why I might be receiving this error? I am running python 3.6.4 and all of the dependencies are installed and up to date, as per the versions listed in the SyntaxError issue posted.

Cannot Run Code

Traceback (most recent call last):
File "C:/Users/14753/OneDrive/Desktop/stock.py", line 17, in
data = pd.read_csv("SBUX.csv", date_parser=['date'])
File "C:\Users\14753\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pandas\io\parsers.py", line 702, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\14753\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pandas\io\parsers.py", line 429, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\14753\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pandas\io\parsers.py", line 895, in init
self._make_engine(self.engine)
File "C:\Users\14753\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pandas\io\parsers.py", line 1122, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\14753\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pandas\io\parsers.py", line 1853, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 387, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'SBUX.csv' does not exist: b'SBUX.csv'

Having this error, whats the problem

TypeError: can't multiply sequence by non-int of type 'float'

from clairvoyant import Backtest
from pandas import read_csv

Testing performance on a single stock

variables = ["SSO", "SSC"] # Financial indicators of choice
trainStart = '2013-03-01' # Start of training period
trainEnd = '2015-07-15' # End of training period
testStart = '2015-07-16' # Start of testing period
testEnd = '2016-07-16' # End of training period
buyThreshold = 0.65 # Confidence threshold for predicting buy (default = 0.65)
sellThreshold = 0.65 # Confidence threshold for predicting sell (default = 0.65)
C = 1 # Penalty parameter (default = 1)
gamma = 10 # Kernel coefficient (default = 10)
continuedTraining = False # Continue training during testing period? (default = false)

backtest = Backtest(variables, trainStart, trainEnd, testStart, testEnd)

data = read_csv(r"H:/Python/ClairVoyantTest/sbux.csv") # Read in data
data = data.round(3) # Round all values
backtest.stocks.append("SBUX") # Inform the model which stock is being tested
for i in range(0,10): # Run the model 10-15 times
backtest.runModel(data)

backtest.displayConditions()
backtest.displayStats()

---

I have run the above code: The issue coming was

File "", line 1, in runfile('H:/Python/StockPredictionUsingClairVoyant.py', wdir='H:/Python')

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile execfile(filename, namespace)

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile exec(compile(scripttext, filename, 'exec'), glob, loc)

File "H:/Python/StockPredictionUsingClairVoyant.py", line 30, in backtest.runModel(data)

File "C:\Anaconda2\lib\site-packages\clairvoyant\Backtest.py", line 72, in runModel data['Date'] = to_datetime(data['Date'])

File "C:\Anaconda2\lib\site-packages\pandas\core\frame.py", line 1914, in getitem return self._getitem_column(key)

File "C:\Anaconda2\lib\site-packages\pandas\core\frame.py", line 1921, in _getitem_column return self._get_item_cache(key)

File "C:\Anaconda2\lib\site-packages\pandas\core\generic.py", line 1090, in _get_item_cache values = self._data.get(item)

File "C:\Anaconda2\lib\site-packages\pandas\core\internals.py", line 3102, in get loc = self.items.get_loc(item)

File "C:\Anaconda2\lib\site-packages\pandas\core\index.py", line 1692, in get_loc return self._engine.get_loc(_values_from_object(key))

File "pandas\index.pyx", line 137, in pandas.index.IndexEngine.get_loc (pandas\index.c:3979)

File "pandas\index.pyx", line 157, in pandas.index.IndexEngine.get_loc (pandas\index.c:3843)

File "pandas\hashtable.pyx", line 668, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12265)

File "pandas\hashtable.pyx", line 676, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12216)

KeyError: 'Date'

runfile('H:/Python/StockPredictionUsingClairVoyant.py', wdir='H:/Python') Traceback (most recent call last):

File "", line 1, in runfile('H:/Python/StockPredictionUsingClairVoyant.py', wdir='H:/Python')

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile execfile(filename, namespace)

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile exec(compile(scripttext, filename, 'exec'), glob, loc)

File "H:/Python/StockPredictionUsingClairVoyant.py", line 27, in data = data.round(3) # Round all values

File "C:\Anaconda2\lib\site-packages\pandas\core\frame.py", line 4335, in round new_cols = [np.round(self[col], decimals) for col in self]

File "C:\Anaconda2\lib\site-packages\numpy\core\fromnumeric.py", line 2782, in round_ return round(decimals, out)

File "C:\Anaconda2\lib\site-packages\pandas\core\series.py", line 1234, in round result = _values_from_object(self).round(decimals, out=out)

TypeError: can't multiply sequence by non-int of type 'float'

What would be the cause of the error.

Does trainStart and trainEnd not support date formats anymore?

So basicly im downloading yahoo data and then convert the date format because I've seen in previously issues in this repo that you don't support yyyy-mm-dd?
So i convert from yyyy-mm-dd to mm-dd-yyyy.

Then i assign variables in the same date format.
I see from your example in readme that you are using integers here instead?
So is date not supported anymore?
As you can see in the traceback at the bottom dates don't work for me.

variables = ["Open", "Close", "High", "Low", "Volume"]
trainStart = '11-01-2015'       # Start of training period
trainEnd   = '11-01-2017'       # End of training period
testStart  = '11-02-2017'       # Start of testing period
testEnd    = '11-17-2017'       # End of testing period

So here is the data from the csv file before i read it.

1 Date,Open,High,Low,Close,Volume
2 2015-01-02,5.05,5.18,4.83,5.14,1542452.0
3 2015-01-05,5.15,5.23,4.8,4.99,1207386.0
4 2015-01-06,4.89,4.97,4.75,4.92,704102.0
5 2015-01-07,4.82,4.93,4.75,4.8,614919.0

data = read_csv("file.csv")                                                                                                                                      
data = data.round(3)                    # Round all values                                                                                                               
data['Date'] = to_datetime(data.Date)   #convert Date coloum data to datetime                                                                                            
data['Date'] = data['Date'].dt.strftime('%m-%d-%Y') #convert format of dates

output of print(data)

0    01-02-2015  5.05  5.18  4.83   5.14   1542452.0
1    01-05-2015  5.15  5.23  4.80   4.99   1207386.0
2    01-06-2015  4.89  4.97  4.75   4.92    704102.0
3    01-07-2015  4.82  4.93  4.75   4.80    614919.0
4    01-08-2015  4.85  4.98  4.65   4.72    686451.0

Traceback (most recent call last):
File "clair2.py", line 35, in <module>
    backtest.start(data, kernel='rbf', C=1, gamma=10)
  File "/home/kodi/.local/lib/python3.5/site-packages/clairvoyant/engine.py", line 197, in start
    Engine.start(self, data, **kwargs)
  File "/home/kodi/.local/lib/python3.5/site-packages/clairvoyant/engine.py", line 50, in start
    for i in range(self.trainStr, self.trainEnd+1):
TypeError: Can't convert 'int' object to str implicitly

backtest.start(data, kernel='rbf', C=1, gamma=10)                                                                                                                       
backtest.conditions()                                                                                                                                                   
backtest.statistics()                                                                                                                                                   
backtest.visualize('thin.ol')

IndexError: single positional indexer is out-of-bounds

When trying to run your code I got the above error in this line:
backtest.start(data, kernel='rbf', C=1, gamma=10)

any idea why?

Files with sample data

It would be great to have files with sample data included in repo.
It doesn't have to be real data, but it would be helpful to making Clairvoyant up and running with my own data (e.g. I don't know if it should be minute data, or daily OHLCV).

Portfolio Simulation

I have been trying to run through your Demo and I am stuck on the Portfolio Simulation section, first time I've used Python so probably doing something wrong somewhere. I got the first Demo working and it was returning.

RELIANCE | Training: 02/01/2013-07/15/2015 Testing: 07/16/2015-07/15/2016

Total Buys: 3690
Buy Accuracy: 50.81%
Total Sells: 0
Sell Accuracy: 0.0%

With the Portfolio one I only get back
Run #1 Buying Power: $1000000.0 Shares: 0 Total Value: $1000000.0 Run #2 Buying Power: $1000000.0 Shares: 0 Total Value: $1000000.0 Run #3 Buying Power: $1000000.0 Shares: 0 Total Value: $1000000.0 Run #4 Buying Power: $1000000.0 Shares: 0 Total Value: $1000000.0 Run #5 Buying Power: $1000000.0 Shares: 0 Total Value: $1000000.0 Performance across all runs Runs: 5 Average Performance: 0.0%

Here is my Code.

from clairvoyant import Portfolio
from pandas import read_csv
from nsepy import get_history
from datetime import date

variables  = ["Open", "Close", "High", "Low", "Volume"]     # Financial indicators of choice # "SSO", "SSC"
trainStart = '2013-03-01'            # Start of training period
trainEnd   = '2015-07-15'            # End of training period
testStart  = '2015-07-16'            # Start of testing period
testEnd    = '2016-07-16'            # End of testing period
buyThreshold  = 0.65                 # Confidence threshold for predicting buy (default = 0.65) 
sellThreshold = 0.65                 # Confidence threshold for predicting sell (default = 0.65)
C = 1                                # Penalty parameter (default = 1)
gamma = 10                           # Kernel coefficient (default = 10)
continuedTraining = False            # Continue training during testing period? (default = false)
startingBalance = 1000000            # Starting balance of portfolio

#User defined trading logic (see below)
def buyLogic(self, confidence, data, testDay): 
    quote = data["Close"][testDay]                           # Leave as is

    if confidence >= 0.2:                                   # If model signals buy
        shareOrder = int((self.buyingPower*0.3)/quote)       # and is 75-100% confident
        self.buyShares(shareOrder, quote)                    # invest 30% of buying power    

    elif confidence >= 0.1:                                 # If model is 70-75% confident
        shareOrder = int((self.buyingPower*0.2)/quote)       # invest 20% of buying power
        self.buyShares(shareOrder, quote)

    elif confidence >= 0.05:                                 # If model is 65-70% confident
        shareOrder = int((self.buyingPower*0.1)/quote)       # invest 10% of buying power
        self.buyShares(shareOrder, quote)


def sellLogic(self, confidence, data, testDay):
    quote = data["Close"][testDay]                       

    if confidence >= 0.65:                                   # If model signals sell
        self.sellShares(self.shares, quote)                  # and is 65-100% confident
                                                             # sell all shares    

def nextDayLogic(self, prediction, nextDayPerformance, data, testDay):
    quote = data["Close"][testDay]                        

    # Case 1: Prediction is buy, price increases
    if prediction == 1 and nextDayPerformance > 0:

        if nextDayPerformance >= 0.025:                      # If I bought shares
            self.sellShares(self.shares, quote)              # and price increases >= 2.5%
                                                             # sell all shares

    # Case 2: Prediction is buy, price decreases
    elif prediction == 1 and nextDayPerformance <= 0: pass 

                                                             # If I bought shares
                                                             # and price decreases
                                                             # hold position

    # Case 3: Prediction is sell, price decreases
    elif prediction == -1 and nextDayPerformance <= 0:

        if nextDayPerformance <= -0.025:                     # If I sold shares
            shareOrder = int((self.buyingPower*0.2)/quote)   # and price decreases >= 2.5%
            self.buyShares(shareOrder, quote)                # reinvest 20% of buying power

    # Case 4: Prediction is sell, price increases
    elif prediction == -1 and nextDayPerformance > 0: pass

                                                             # If I sold shares
                                                             # and price increases
                                                             # hold position
    # Case 5: No confident prediction was made

portfolio = Portfolio(variables, trainStart, trainEnd, testStart, testEnd)

history = get_history(symbol="RELIANCE",
                      start=date(2013, 1, 1),
                      end=date(2017, 1, 1),
                      index=False)

history[['Open','High','Low','Close','Volume','Close']].to_csv("RELIANCE.csv")

data = read_csv("RELIANCE.csv")
data = data.round(3)
for i in range(0,5):
    portfolio.runModel(data, startingBalance, buyLogic, sellLogic, nextDayLogic)
    portfolio.displayLastRun()

portfolio.displayAllRuns()

SyntaxError: invalid syntax : "self.svc = SVC(**kwargs, probability=True)"

First of all, let me start by thanking you very much for all your hard work and dedication. Your work is truly magnificent.

I am trying to run the example from your instructions. I am getting the error:

File "C:\Python27\lib\site-packages\clairvoyant\model.py", line 22
self.svc = SVC(**kwargs, probability=True)

SyntaxError: invalid syntax

What could be the cause of this error ?

I am running Python 2.7.15

Date format error

I got new info and entered it in the SBUX.csv file, but I keep getting this every time I run it:

File "test.py", line 17, in <module>
    backtest = Backtest(variables, trainStart, trainEnd, testStart, testEnd)
  File "/root/clair/clairvoyant/Backtest.py", line 47, in __init__
    self.trainEnd           = to_datetime(trainEnd)
  File "/usr/local/lib/python2.7/dist-packages/pandas/util/decorators.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/tools.py", line 430, in to_datetime
    return _convert_listlike(np.array([arg]), box, format)[0]
  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/tools.py", line 413, in _convert_listlike
    raise e
ValueError: month must be in 1..12

I changed the date format and everything. Here is what my data looks like:

Date,Open,High,Low,Close,Volume
2-1-2015,12.33,12.84,11.87,12.67,221293
5-1-2015,12.57,13.34,12.19,13.02,335037
6-1-2015,13,13.45,12.29,12.8,369810
7-1-2015,12.95,13.23,12.05,13.1,561895
8-1-2015,13.19,14.07,12.91,14.05,539694
9-1-2015,13.98,14.2,13.39,13.61,626020
12-1-2015,13.7,14.25,13.39,13.52,176644
13-1-2015,13.7,14.17,12.94,13.21,457432
14-1-2015,13.06,14.18,12.1,14.17,310948
15-1-2015,14.19,14.5,13.41,13.44,486042
16-1-2015,13.38,13.87,13.21,13.54,283274

ImportError: No module named 'backtester'

I m trying to run the example from your instructions. I am getting the error:
ImportError: No module named 'backtester'

Kindly, let me know what is the cause of this?

Standard Data Format

I am using standard Yahoo downloads in the format

Date,Open,High,Low,Close,Volume,Close
2015-01-01,312.45,315.0,310.7,314.0,6138488,314.0

When I run sample program (as per README) I get the following error -

Traceback (most recent call last):
File "clairvoyance.py", line 35, in <module>
backtest.runModel(data)
File "/Users/8288/source_code/finance/venv/lib/python2.7/site-packages/clairvoyant/Backtest.py", line 75, in runModel
trainStart = DateIndex(data, stock, self.trainStart, False)
File "/Users/8288/source_code/finance/venv/lib/python2.7/site-packages/clairvoyant/Backtest.py", line 16, in DateIndex
lowbound = data["Date"][0]
File "/Users/8288/source_code/finance/venv/lib/python2.7/site-packages/pandas/core/series.py", line 603, in __getitem__
result = self.index.get_value(self, key)
File "/Users/8288/source_code/finance/venv/lib/python2.7/site-packages/pandas/indexes/base.py", line 2175, in get_value
return tslib.get_value_box(s, key)
File "pandas/tslib.pyx", line 946, in pandas.tslib.get_value_box (pandas/tslib.c:19053)
File "pandas/tslib.pyx", line 962, in pandas.tslib.get_value_box (pandas/tslib.c:18770)
IndexError: index out of bounds 

Model continually yields 0 buys and 0 sells

Thank you for the great work on the project! I'm really excited to test it out.

I downloaded data from five stocks through Yahoo finance and am attempting to use the model on them, but it only returns 0 buys and 0 sells no matter how I adjust the parameters.

I've attached my code, the output I see, and a sample of my data (abridged). I've been experimenting with different buy and sell thresholds trying to force it to sell, but I haven't been able to.

Code:

   1  from clairvoyant import Backtest
   2  from pandas import read_csv
   3 
   4 # Testing performance on a single stock
   5 
   6 variables = ["Close", "Volume"]
   7 trainStart = '2012-05-21'       # Start of training period
   8 trainEnd   = '2015-06-15'       # End of training period
   9 testStart  = '2015-06-16'       # Start of testing period
  10 testEnd    = '2017-02-9'       # End of testing period
  11 buyThreshold  = 0.1            # Confidence threshold for predicting buy (default = 0.65)
  12 sellThreshold = 0.1            # Confidence threshold for predicting sell (default = 0.65)
  13 C = 1                           # Penalty parameter (default = 1)
  14 gamma = 10                      # Kernel coefficient (default = 10)
  15 continuedTraining = False       # Continue training during testing period? (default = false)
  16 
  17 backtest = Backtest(variables, trainStart, trainEnd, testStart, testEnd)
  18 
  19 # Testing performance across multiple stocks
  20 
  21 stocks = ["msft2", "fb2", "appl2", "amzn2", "goog2"]
  22 
  23 for stock in stocks:
  24     data = read_csv('Stocks/%s.csv' % stock)
  25     data = data.round(3)
  26     backtest.stocks.append(stock)
  27     for i in range(0,10):
  28         backtest.runModel(data)
  29 
  30 backtest.displayConditions()
  31 backtest.displayStats()

Output:

Conditions
X1: Close
X2: Volume
Buy Threshold: 65.0%
Sell Threshold: 65.0%
C: 1
gamma: 10
Continued Training: False
Stats
Stock(s):
msft2 | Training: 05/21/2012-06/15/2015 Testing: 06/16/2015-02/09/2017
fb2 | Training: 05/21/2012-06/15/2015 Testing: 06/16/2015-02/09/2017
appl2 | Training: 05/21/2012-06/15/2015 Testing: 06/16/2015-02/09/2017
amzn2 | Training: 05/21/2012-06/15/2015 Testing: 06/16/2015-02/09/2017
goog2 | Training: 05/21/2012-06/15/2015 Testing: 06/16/2015-02/09/2017

Total Buys: 0
Buy Accuracy: 0.0%
Total Sells: 0
Sell Accuracy: 0.0%

Sample data:

Date,Open,High,Low,Close,Volume,Adj Close
    02/10/2012,183.419998,187.630005,182.520004,185.539993,5797100,185.539993
    02/13/2012,187.169998,192.50,185.679993,191.589996,6071400,191.589996
    02/14/2012,191.080002,193.570007,186.100006,191.300003,9527100,191.300003
    02/15/2012,191.289993,191.539993,183.259995,184.470001,7794400,184.470001
    02/16/2012,177.789993,181.679993,175.139999,179.929993,12563800,179.929993
    02/17/2012,180.089996,183.410004,179.360001,182.50,7418100,182.50

data named 'SBUX.csv'

Hi, where can I find your data 'SBUX.csv' in the example?

Stock prediction tool

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.

I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.

The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction). https://github.com/Leci37/LecTrade/tree/develop
For this I have used big data tools like pandas python, stock technical patterns market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.

With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).

I think it could be useful to you, to improve, I would like to share it with you, and if you are interested in improving and collaborating we could, who knows how to make beautiful things.

Thank you for your time
I'm sorry to contact you here ,by issues, I don't know how I would be able to do it.
mail : [email protected] or https://github.com/Leci37/stocks-Machine-learning-RealTime-telegram/discussions

Getting Clairvoyant to work

I am having the following issue in getting the Clairvoyant project, that identifies and monitors social/historical cues for short term stock movement, to run successfully:
The file hashtable_class_helper.pxl could not be opened.
Here is the exception generated:
Exception "unhandled KeyError"
'SSO'
File: pandas/src/hashtable_class_helper.pxi, Line: 740

I think it might have something to do with the fact that this is a cython-based project and I'm wanting to implement in Python 2.7x in Ubuntu 16.04

Any help would be appreciated.

error using 1st line of code example

ImportError: cannot import name relativedelta

When i run sample code, following error occured:

ImportError: cannot import name relativedelta

ImportError Traceback (most recent call last)
in ()
----> 1 from clairvoyant import Backtest

.\venv\Lib\site-packages\clairvoyant_init_.py in ()
----> 1 from Backtest import Backtest

.\venv\Lib\site-packages\clairvoyant\Backtest.py in ()
10 from numpy import vstack, hstack
11 from csv import DictWriter
---> 12 from pandas.tslib import relativedelta
13 from dateutil.parser import parse

---

My Pandas version : 0.14.1

TypeError: 'str' object cannot be interpreted as an integer when running backtest.start()

Outdated dependencies

Is there any specific reason that the required modules are so outdated and strict (== vs >= in requirements.txt)? This makes it tough to use in existing projects.

Project dependencies have API risk issues

Hi, In clairvoyant, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

bokeh>=0.12.4
matplotlib>=2.0.0
numpy>=1.11.3
pandas>=0.19.2
scikit-learn>=0.18.1

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict.
The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project,
The version constraint of dependency scikit-learn can be changed to >=0.17b1,<=0.21.3.

The above modification suggestions can reduce the dependency conflicts as much as possible,
and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

The calling methods from the matplotlib

Axes.scatter
Axes.set_ylim
pyplot.savefig
Axes.contourf
Axes.set_xlim
Axes.set_title
pyplot.figure
pyplot.subplot

The calling methods from the scikit-learn

RobustScaler.fit

The calling methods from the all methods

setup
ClosedTrade
self.model.predict
pyplot.savefig
Axes.contourf
TypeError
self.account.Equity.append
print
super.__init__
exchange.Account
round
RobustScaler.fit
xx.min
pyplot.figure
copy.deepcopy
enumerate
pyplot.subplot
self.account.TotalValue
vstack
helpers.change
X.append
yy.max
yy.min
LongPosition
Axes.set_xlim
show
Engine.start
super
SVC
Axes.set_title
output_file
format
helpers.profit
change
Model
range
len
self.account.PurgePositions
self.Positions.append
Engine.__init__
self.OpenedTrades.append
figure
sys.path.append
self.model.fit
self.model.svc.fit
self.model.svc.decision_function
Z.reshape
hstack
self.svc.fit
self.svc.predict_proba
xx.max
X.min
meshgrid
ShortPosition
p.line
p.Show
Temporary.ClosePosition
self.model.scaler.transform
Axes.scatter
y.append
Position.Close
float
logic
ValueError
Axes.set_ylim
arange
self.buyStats
RobustScaler
self.ClosedTrades.append
X.max
OpenedTrade
yy.ravel
xx.ravel
ListedColormap
self.scaler.transform
self.sellStats

@developer
Could please help me check this issue?
May I pull a request to fix it?
Thank you very much.

TypeError: a float is required: "self.svc.fit(self.scaler.transform(self.XX), self.yy)"

I am trying to run the example from your instructions. data comes from example/data.csv

I am getting the error:

Traceback (most recent call last):
File "test.py", line 22, in
backtest.start(data, kernel='rbf', C=1, gamma=10)
File "/home/machol/virtualenv/local/lib/python2.7/site-packages/clairvoyant/engine.py", line 197, in start
Engine.start(self, data, **kwargs)
File "/home/machol/virtualenv/local/lib/python2.7/site-packages/clairvoyant/engine.py", line 60, in start
self.model.fit(X, y)
File "/home/machol/virtualenv/local/lib/python2.7/site-packages/clairvoyant/engine.py", line 22, in fit
self.svc.fit(self.scaler.transform(self.XX), self.yy)
File "/home/machol/virtualenv/local/lib/python2.7/site-packages/sklearn/svm/base.py", line 187, in fit
fit(X, y, sample_weight, solver_type, kernel, random_seed=seed)
File "/home/machol/virtualenv/local/lib/python2.7/site-packages/sklearn/svm/base.py", line 254, in _dense_fit
max_iter=self.max_iter, random_seed=random_seed)
File "sklearn/svm/libsvm.pyx", line 59, in sklearn.svm.libsvm.fit
TypeError: a float is required

My python version is 2.7.9。

Dimensionality reduction with t-SNE

https://lvdmaaten.github.io/tsne/

Code seems incorrect

in Engine.start for below code:

Xs = [ data.iloc[i][var] for var in self.features ]

features are mentioned as ["EMA","SSO"] # i assume we need to map them to X1, X2.
iloc takes indexes as integers, it will not work as i.e; data.iloc[0]["EMA"] .

NameError: name 'backtester' is not defined

Hi Federico,

I am new to Python, and I am trying the second example , Simulate a Trading Strategy , and I am getting the following Error:

Traceback (most recent call last):
File "stock6.py", line 60, in
simulation = backtester.Simulation(features, trainStart, trainEnd, testStart, testEnd, buyThreshold, sellThreshold, continuedTraining)
NameError: name 'backtester' is not defined

If I try to edit it and use backtest.simulation I get the same error
NameError: name 'backtest' is not defined

Is there a workaround for this ?

Working example

Hello,

Great package. Can you provide sample data with the acceptable data layout.

Thanks,

Amir