scrtlabs / catalyst Goto Github PK
View Code? Open in Web Editor NEWAn Algorithmic Trading Library for Crypto-Assets in Python
Home Page: http://enigma.co
License: Apache License 2.0
An Algorithmic Trading Library for Crypto-Assets in Python
Home Page: http://enigma.co
License: Apache License 2.0
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
$ 2.7
$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
(catalyst) C:\Users\user C-137>catalyst ingest
Traceback (most recent call last):
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Users\user C-137\AppData\Local\conda\conda\envs\catalyst\Scripts\catalyst.exe_main.py", line 5, in
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst_init.py", line 25, in
from .utils.run_algo import run_algorithm
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\utils\run_algo.py", line 11, in
from catalyst.exchange.bittrex.bittrex import Bittrex
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\exchange\bittrex\bittrex.py", line 11, in
from catalyst.exchange.exchange import Exchange
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\exchange\exchange.py", line 13, in
from catalyst.data.data_portal import BASE_FIELDS
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\data\data_portal.py", line 47, in
from catalyst.data.resample import (
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\data\resample.py", line 29, in
from catalyst.data.minute_bars import MinuteBarReader
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\catalyst\data\minute_bars.py", line 29, in
import tables
File "c:\users\user c-137\appdata\local\conda\conda\envs\catalyst\lib\site-packages\tables_init_.py", line 74, in
' that it can be found in the system path' % hdf5_dlls)
ImportError: Could not load any of ['hdf5.dll', 'hdf5dll.dll'], please ensure that it can be found in the system path
(catalyst) C:\Users\user C-137>
Provide generic base classes for building pricing datasets. Many of the default parameters can be reused across different bundles, this allows data bundles to be more readily and properly extended.
The BasePricingBundle
incorporates the relevant dtypes for pricing data and its associated metadata. The BaseCryptoPricingBundle
and BaseEquityPricingBundle
are similar, but feature different default calendar and minutes per day. This saves users from having to specify these parameters separately when registering a bundle.
This commit b82f719 contains the initial work on this issue.
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
2017-01-01
. If I select 2017-09-01
all markets work as expected. I have manually filters pairs that are not in the current change of the backtest to ensure that data for that day in catalyst existed.BTC Market works for both starting dates, but XMR, eth, usdt markets are not working for 2017-01-01
similar to the previous issues in v0.3.2
I have only tested in Poloniex but my script can be used with any exchange by changing context.exchange so feel free to use this script as a unit test.
"""
Requires Catalyst version 0.3.0 or above
Tested on Catalyst version 0.3.2
These example aims to provide and easy way for users to learn how to collect data from the different exchanges.
You simply need to specify the exchange and the market that you want to focus on.
You will all see how to create a universe and filter it base on the exchange and the market you desire.
The example prints out the closing price of all the pairs for a given market-exchange every 30 minutes.
The example also contains the ohlcv minute data for the past seven days which could be used to create indicators
Use this as the backbone to create your own trading strategies.
Variables lookback date and date are used to ensure data for a coin existed on the lookback period specified.
"""
import numpy as np
import pandas as pd
from datetime import timedelta
from catalyst import run_algorithm
from catalyst.exchange.exchange_utils import get_exchange_symbols
from catalyst.api import (
symbols,
)
def initialize(context):
context.i = -1 # counts the minutes
context.exchange = 'poloniex' # must match the exchange specified in run_algorithm
context.base_currency = 'eth' # must match the base currency specified in run_algorithm
def handle_data(context, data):
lookback = 60 * 24 * 7 # (minutes, hours, days) of how far to lookback in the data history
context.i += 1
# current date formatted into a string
today = context.blotter.current_dt
date, time = today.strftime('%Y-%m-%d %H:%M:%S').split(' ')
lookback_date = today - timedelta(days=(lookback / (60 * 24))) # subtract the amount of days specified in lookback
lookback_date = lookback_date.strftime('%Y-%m-%d %H:%M:%S').split(' ')[0] # get only the date as a string
# update universe everyday
new_day = 60 * 24
if not context.i % new_day:
context.universe = universe(context, lookback_date, date)
# get data every 30 minutes
minutes = 30
if not context.i % minutes and context.universe:
# we iterate for every pair in the current universe
for coin in context.coins:
pair = str(coin.symbol)
# 30 minute interval ohlcv data (the standard data required for candlestick or indicators/signals)
# 30T means 30 minutes re-sampling of one minute data. change to your desire time interval.
open = fill(data.history(coin, 'open', bar_count=lookback, frequency='1m')).resample('30T').first()
high = fill(data.history(coin, 'high', bar_count=lookback, frequency='1m')).resample('30T').max()
low = fill(data.history(coin, 'low', bar_count=lookback, frequency='1m')).resample('30T').min()
close = fill(data.history(coin, 'price', bar_count=lookback, frequency='1m')).resample('30T').last()
volume = fill(data.history(coin, 'volume', bar_count=lookback, frequency='1m')).resample('30T').sum()
# close[-1] is the equivalent to current price
# displays the minute price for each pair every 30 minutes
print(today, pair, open[-1], high[-1], low[-1], close[-1], volume[-1])
# ----------------------------------------------------------------------------------------------------------
# -------------------------------------- Insert Your Strategy Here -----------------------------------------
# ----------------------------------------------------------------------------------------------------------
def analyze(context=None, results=None):
pass
# Get the universe for a given exchange and a given base_currency market
# Example: Poloniex BTC Market
def universe(context, lookback_date, current_date):
json_symbols = get_exchange_symbols(context.exchange) # get all the pairs for the exchange
universe_df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str) # convert into a dataframe
universe_df['base_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[1],
axis=1)
universe_df['market_currency'] = universe_df.apply(lambda row: row.symbol.split('_')[0],
axis=1)
# Filter all the exchange pairs to only the ones for a give base currency
universe_df = universe_df[universe_df['base_currency'] == context.base_currency]
# Filter all the pairs to ensure that pair existed in the current date range
universe_df = universe_df[universe_df.start_date < lookback_date]
universe_df = universe_df[universe_df.end_daily >= current_date]
context.coins = symbols(*universe_df.symbol) # convert all the pairs to symbols
print(universe_df.head(), len(universe_df))
return universe_df.symbol.tolist()
# Replace all NA, NAN or infinite values with its nearest value
def fill(series):
if isinstance(series, pd.Series):
return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
elif isinstance(series, np.ndarray):
return pd.Series(series).replace([np.inf, -np.inf], np.nan).ffill().bfill().values
else:
return series
if __name__ == '__main__':
start_date = pd.to_datetime('2017-01-01', utc=True)
end_date = pd.to_datetime('2017-10-15', utc=True)
performance = run_algorithm(start=start_date, end=end_date,
capital_base=10000.0,
initialize=initialize,
handle_data=handle_data,
analyze=analyze,
exchange_name='poloniex',
data_frequency='minute',
base_currency='eth',
live=False,
live_graph=False,
algo_namespace='simple_universe')
"""
Run in Terminal (inside catalyst environment):
python simple_universe.py
"""
[2017-10-26 23:14:30.107384] INFO: exchange_bundle: pricing data for [u'bcn_xmr'] not found in range 1989-05-28 00:00:00+00:00 to 2017-01-01 00:00:00+00:00, updating the bundles.
[====================================] Fetching poloniex daily candles: : 100%
Error traceback: /home/avn3r/applications/conda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py (line 676)
PricingDataNotLoadedError: Pricing data open for trading pairs bcn_xmr trading on exchange poloniex since 2014-07-23 00:00:00+00:00 is unavailable. The bundle data is either out-of-date or has not been loaded yet. Please ingest data using the command `catalyst ingest-exchange -x poloniex -f daily -i bcn_xmr`. See catalyst documentation for details.
Sincerely,
avn3r
Adapt existing quandl bundle to use new bundle logic. This is helpful for testing that the base bundle is generic enough to ingestion for as many types of assets as possible. This commit d705af6 contains the implementation.
Dear Catalyst Maintainers,
Hello!
Before I tell you about my issue, let me describe my environment:
$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))' 64 (python27)
pip
)Now that you know a little about me, let me tell you about the issue I am
having:
KeyError: Equity(71, symbol=u'USDT_BTC', asset_name='', exchange=u'POLO', start_date=Timestamp('2015-02-19 00:00:00+0000', tz='UTC'), end_date=Timestamp('2017-08-09 00:00:00+0000', tz='UTC'), first_traded=None, auto_close_date=None, exchange_full=u'POLO')
Here is how you can reproduce this issue on your machine:
deleting .catalyst folder
Thank you!
C:\Anaconda3\envs\python27\lib\site-packages\ipykernel_launcher.py:50: ZiplineDeprecationWarning: `data[sid(N)]` is deprecated. Use `data.current`.
i: 1
KeyErrorTraceback (most recent call last)
<ipython-input-4-c2ead63b7fce> in <module>()
76 end=end,
77 capital_base=100000,
---> 78 bundle='poloniex'
79 )
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\utils\run_algo.pyc in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, default_extension, extensions, strict_extensions, environ)
412 print_algo=False,
413 local_namespace=False,
--> 414 environ=environ,
415 )
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\utils\run_algo.pyc in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, print_algo, local_namespace, environ)
231 ).run(
232 data,
--> 233 overwrite_sim_params=False,
234 )
235
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\algorithm.pyc in run(self, data, overwrite_sim_params)
740 try:
741 perfs = []
--> 742 for perf in self.get_generator():
743 perfs.append(perf)
744
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\gens\tradesimulation.pyc in transform(self)
221 for dt, action in self.clock:
222 if action == BAR:
--> 223 for capital_change_packet in every_bar(dt):
224 yield capital_change_packet
225 elif action == SESSION_START:
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\gens\tradesimulation.pyc in every_bar(dt_to_use, current_data, handle_data)
134 perf_tracker.process_commission(commission)
135
--> 136 handle_data(algo, current_data, dt_to_use)
137
138 # grab any new orders from the blotter, then clear the list.
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\utils\events.pyc in handle_data(self, context, data, dt)
214 context,
215 data,
--> 216 dt,
217 )
218
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\utils\events.pyc in handle_data(self, context, data, dt)
233 """
234 if self.rule.should_trigger(dt):
--> 235 self.callback(context, data)
236
237
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\algorithm.pyc in handle_data(self, data)
475 def handle_data(self, data):
476 if self._handle_data:
--> 477 self._handle_data(self, data)
478
479 # Unlike trading controls which remain constant unless placing an
<ipython-input-4-c2ead63b7fce> in handle_data(context, data)
48
49 # Retrieve current asset price from pricing data
---> 50 price = data[context.asset].price
51
52 # Check if still buying and could (approximately) afford another purchase
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\_protocol.pyx in catalyst._protocol.SidView.__getattr__ (catalyst\_protocol.c:12347)()
869 return self.sid
870
--> 871 return self.data_portal.get_spot_value(
872 self.asset,
873 column,
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\data_portal.pyc in get_spot_value(self, assets, field, dt, data_frequency)
551
552 if assets_is_scalar:
--> 553 return get_single_asset_value(assets)
554 else:
555 return list(map(get_single_asset_value, assets))
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\data_portal.pyc in get_single_asset_value(asset)
534 else:
535 return self._get_daily_spot_value(
--> 536 asset, field, session_label,
537 )
538 else:
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\data_portal.pyc in _get_daily_spot_value(self, asset, column, dt)
760 try:
761 value = reader.get_value(
--> 762 asset, found_dt, "close"
763 )
764 if not isnull(value):
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\dispatch_bar_reader.pyc in get_value(self, sid, dt, field)
95 asset = self._asset_finder.retrieve_asset(sid)
96 r = self._readers[type(asset)]
---> 97 return r.get_value(asset, dt, field)
98
99 def get_last_traded_dt(self, asset, dt):
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\us_equity_pricing.py in get_value(self, sid, dt, field)
731 0.
732 """
--> 733 ix = self.sid_day_index(sid, dt)
734 price = self._spot_col(field)[ix]
735 if field != 'volume':
C:\Anaconda3\envs\python27\lib\site-packages\catalyst\data\us_equity_pricing.py in sid_day_index(self, sid, day)
700 raise NoDataOnDate("day={0} is outside of calendar={1}".format(
701 day, self.sessions))
--> 702 offset = day_loc - self._calendar_offsets[sid]
703 if offset < 0:
704 raise NoDataBeforeDate(
KeyError: Equity(71, symbol=u'USDT_BTC', asset_name='', exchange=u'POLO', start_date=Timestamp('2015-02-19 00:00:00+0000', tz='UTC'), end_date=Timestamp('2017-08-09 00:00:00+0000', tz='UTC'), first_traded=None, auto_close_date=None, exchange_full=u'POLO')
``
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
$ uname --all
)Darwin Mack.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64
FreeBSD cronus 11.1-STABLE FreeBSD 11.1-STABLE #0 r321518: Wed Jul 26 07:20:06 MDT 2017 root@cronus:/usr/obj/usr/src/sys/GENERIC amd64
pip
, conda
, or other (please explain)
) On MacOS, installed using brew. On FreeBSD, installed through ports$ pip freeze
or $ conda list
MacOS Python2 below:alembic==0.9.3
bcolz==0.12.1
Bottleneck==1.2.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.26
decorator==4.1.2
empyrical==0.2.1
enigma-catalyst==0.1.dev6
functools32==3.2.3.post2
idna==2.5
intervaltree==2.1.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.0.2
multipledispatch==0.4.9
networkx==1.11
numexpr==2.6.2
numpy==1.13.1
pandas==0.19.2
pandas-datareader==0.5.0
patsy==0.4.1
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.2
requests==2.18.2
requests-file==1.4.2
requests-ftp==0.3.1
scipy==0.19.1
six==1.10.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.12
statsmodels==0.8.0
subprocess32==3.2.7
tables==3.4.2
toolz==0.8.2
urllib3==1.22
Now that you know a little about me, let me tell you about the issue I am
having:
Running the following command fails to download data (backtrace below)
catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
Following getting started guide.
$ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
[2017-07-28 17:59:24.303662] INFO: Loader: Loading benchmark data for 'USDT_BTC' from 1989-12-31 00:00:00+00:00 to 2017-07-26 00:00:00+00:00
[2017-07-28 17:59:24.317006] INFO: Loader: Loading data for /Users/randy/.catalyst/data/USDT_BTC_benchmark.csv failed with error [Unknown string format].
[2017-07-28 17:59:24.317162] INFO: Loader: Cache at /Users/randy/.catalyst/data/USDT_BTC_benchmark.csv does not have data from 1990-01-01 00:00:00+00:00 to 2017-07-26 00:00:00+00:00.
[2017-07-28 17:59:24.317488] INFO: Loader: Downloading benchmark data for 'USDT_BTC' from 1989-12-31 00:00:00+00:00 to 2017-07-26 00:00:00+00:00
Traceback (most recent call last):
File "/Users/randy/.virtualenvs/catalyst2/bin/catalyst", line 11, in <module>
sys.exit(main())
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/__main__.py", line 97, in _
return f(*args, **kwargs)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/__main__.py", line 240, in run
environ=os.environ,
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 196, in _run
env, data = get_trading_env_and_data(bundles)
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 156, in get_trading_env_and_data
environ=environ,
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/finance/trading.py", line 99, in __init__
self.bm_symbol,
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/data/loader.py", line 135, in load_crypto_market_data
environ,
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/data/loader.py", line 367, in ensure_crypto_benchmark_data
if not has_data_for_dates(daily_close, first_date, last_date):
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/catalyst/data/loader.py", line 96, in has_data_for_dates
first, last = dts[[0, -1]]
File "/Users/randy/.virtualenvs/catalyst2/lib/python2.7/site-packages/pandas/tseries/base.py", line 276, in __getitem__
result = getitem(key)
IndexError: index 0 is out of bounds for axis 1 with size 0
Here is how you can reproduce this issue on your machine:
I have confirmed that this works as expected on Debian Linux. Confirmed that it failed on these two platforms for both py2 and py3. Other users have confirmed the same behavior on MacOS in Slack #dev
**I got this error with "catalyst run -f buy_btc_simple.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_btc_simple_out.pickle" in linux mint .
details as follows:**
raceback (most recent call last):
File "/usr/local/bin/catalyst", line 11, in
load_entry_point('enigma-catalyst==0+unknown', 'console_scripts', 'catalyst')()
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/main.py", line 101, in _
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/main.py", line 275, in run
live_graph=False
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/utils/run_algo.py", line 180, in _run
portfolio=portfolio
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/exchange/bitfinex/bitfinex.py", line 50, in init
self.load_assets()
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/exchange/exchange.py", line 254, in load_assets
symbol_map = self.fetch_symbol_map()
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/exchange/exchange.py", line 235, in fetch_symbol_map
return get_exchange_symbols(self.name)
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/exchange/exchange_utils.py", line 102, in get_exchange_symbols
download_exchange_symbols(exchange_name, environ)
File "/usr/local/lib/python2.7/dist-packages/enigma_catalyst-0+unknown-py2.7-linux-i686.egg/catalyst/exchange/exchange_utils.py", line 78, in download_exchange_symbols
response = request.urlretrieve(url=url, filename=filename)
File "/usr/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
fp = self.open(url, data)
File "/usr/lib/python2.7/urllib.py", line 213, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 350, in open_http
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 859, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 836, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 575, in create_connection
raise err
IOError: [Errno socket error] [Errno 111] Connection refused
I'm able to successfully run the buy_and_hodl.py (note typo) sample algorithm using the command line interface as suggested in the documentation.
However, I also need to ability to run it using the run_algorithm() interface for more control during the execution.
Here is my adapted version of the algorithm:
from catalyst.utils.run_algo import run_algorithm
from datetime import datetime
import pytz
from catalyst.api import (
order_target_value,
symbol,
record,
cancel_order,
get_open_orders,
)
def initialize(context):
context.ASSET_NAME = 'USDT_BTC'
context.TARGET_HODL_RATIO = 0.8
context.RESERVE_RATIO = 1.0 - context.TARGET_HODL_RATIO
# For all trading pairs in the poloniex bundle, the default denomination
# currently supported by Catalyst is 1/1000th of a full coin. Use this
# constant to scale the price of up to that of a full coin if desired.
context.TICK_SIZE = 1000.0
context.is_buying = True
context.asset = symbol(context.ASSET_NAME)
context.i = 0
def handle_data(context, data):
context.i += 1
print 'i:', context.i
starting_cash = context.portfolio.starting_cash
target_hodl_value = context.TARGET_HODL_RATIO * starting_cash
reserve_value = context.RESERVE_RATIO * starting_cash
# Cancel any outstanding orders
orders = get_open_orders(context.asset) or []
for order in orders:
cancel_order(order)
# Stop buying after passing the reserve threshold
cash = context.portfolio.cash
if cash <= reserve_value:
context.is_buying = False
# Retrieve current asset price from pricing data
price = data[context.asset].price
# Check if still buying and could (approximately) afford another purchase
if context.is_buying and cash > price:
# Place order to make position in asset equal to target_hodl_value
order_target_value(
context.asset,
target_hodl_value,
limit_price=price * 1.1,
stop_price=price * 0.9,
)
record(
price=price,
cash=cash,
starting_cash=context.portfolio.starting_cash,
leverage=context.account.leverage,
)
start = datetime(2015, 3, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2017, 6, 28, 0, 0, 0, 0, pytz.utc)
run_algorithm(
initialize=initialize,
handle_data=handle_data,
start=start,
end=end,
capital_base=100000,
bundle='poloniex'
)
Here is the output:
[2017-08-16 17:11:57.438224] INFO: Loader: Loading benchmark data for 'USDT_BTC' from 2015-02-28 00:00:00+00:00 to 2017-08-14 00:00:00+00:00
[2017-08-16 17:11:57.478054] WARNING: Loader: Refusing to download new treasury data because a download succeeded at 2017-08-16 16:19:53+00:00.
Traceback (most recent call last):
File "catalyst/examples/buy_and_hold_live.py", line 78, in <module>
bundle='poloniex'
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/catalyst/utils/run_algo.py", line 414, in run_algorithm
environ=environ,
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/catalyst/utils/run_algo.py", line 197, in _run
env, data = get_trading_env_and_data(bundles)
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/catalyst/utils/run_algo.py", line 168, in get_trading_env_and_data
adjustment_reader=bundle_data.adjustment_reader,
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/catalyst/data/data_portal.py", line 319, in __init__
if self._first_trading_day is not None else (None, None)
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/catalyst/utils/calendars/trading_calendar.py", line 695, in open_and_close_for_session
sched.at[session_label, 'market_open'].tz_localize('UTC'),
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/python2/lib/python2.7/site-packages/pandas/core/indexing.py", line 1665, in __getitem__
return self.obj.get_value(*key, takeable=self._takeable)
File "/Users/fredfortier/Code/enigma-projects/catalyst-baseline/python2/lib/python2.7/site-packages/pandas/core/frame.py", line 1900, in get_value
return engine.get_value(series.get_values(), index)
File "pandas/index.pyx", line 98, in pandas.index.IndexEngine.get_value (pandas/index.c:3557)
File "pandas/index.pyx", line 106, in pandas.index.IndexEngine.get_value (pandas/index.c:3240)
File "pandas/index.pyx", line 583, in pandas.index.DatetimeEngine.get_loc (pandas/index.c:11566)
KeyError: Timestamp('2012-02-01 00:00:00+0000', tz='UTC')
Do you get the same error on your side?
Extend support for daily ingestion to also support five minute ingestion. This will require having a functional FiveMinuteBarWriter and passing it appropriately to the ingestion function. Otherwise, the PoloniexBundle is currently capable of curating five minute data from the poloniex API, it just needs a method of writing it properly to disk.
Mac OSX
Python 3.6.1
64 bit
pip
pip (9.0.1) setuptools (36.5.0) wheel (0.30.0)
catalyst-venv) ankit@Ankits-MacBook-Pro ~/crypto pip install enigma-catalyst
Collecting enigma-catalyst
Downloading enigma-catalyst-0.2.dev5.tar.gz (691kB)
100% |████████████████████████████████| 696kB 74kB/s
v Complete output from command python setup.py egg_info:
Download error on https://pypi.python.org/simple/Cython/: [Errno 65] No route to host -- Some packages may not be found!
Couldn't find index page for 'Cython' (maybe misspelled?)
No local packages or working download links found for Cython>=0.25.2
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/85/tdct6g4s3fq5zdwkctczl6ym0000gn/T/pip-build-2yj9d9ar/enigma-catalyst/setup.py", line 322, in <module>
**conditional_arguments
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/setuptools/dist.py", line 335, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/setuptools/dist.py", line 456, in fetch_build_eggs
replace_conflicting=True,
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 863, in resolve
replace_conflicting=replace_conflicting
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1141, in best_match
return self.obtain(req, installer)
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1153, in obtain
return installer(requirement)
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/setuptools/dist.py", line 522, in fetch_build_egg
return cmd.easy_install(req)
File "/Users/ankit/crypto/catalyst-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 666, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('Cython>=0.25.2')
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/85/tdct6g4s3fq5zdwkctczl6ym0000gn/T/pip-build-2yj9d9ar/enigma-catalyst/
The following error occurs with requesting daily historical data on Bittrex:
PricingDataNotLoadedError: Pricing data close for trading pairs btc_usdt trading on exchange bittrex since 2015-12-11 00:00:00+00:00 is unavailable. The bundle data is either out-of-date or has not been loaded yet. Please ingest data using the command `catalyst ingest-exchange -x bittrex -f daily -i btc_usdt`. See catalyst documentation for details.The `data.history()` call looks this, where `context.base_market` equals btc_usdt and `context.hist_window` equals 60.
btc_price = data.history(symbol(context.base_market),bar_count=context.hist_window,fields='price'
,frequency='1d')
In some systems this command fails:
catalyst run -f buy_and_hodl.py --start 2015-03-01 --end 2017-06-28 --capital-base 100000 -o bah.pickle
while this succeeds:
catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-7-25 --capital-base 100000 -o bah.pickle
While we are investigating this issue, if you encounter this error, use the second command.
While we are adding more built-in exchange price data, users may want to backtest with their own. For example, someone may have purchased rare historical data from Coinigy. While this should work in theory with the regular ingest
command, we have been focused on exchange bundles so this need to be re-validated and fully supported.
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
Linux 4.11.3-1-ARCH #1 SMP PREEMPT Sun May 28 10:40:17 CEST 2017 x86_64 GNU/Linux
)Python 2.7.13
64
pip
)$ pip freeze
alembic==0.9.6
backports-abc==0.5
backports.functools-lru-cache==1.4
backports.shutil-get-terminal-size==1.0.0
bcolz==0.12.1
bleach==2.1.1
Bottleneck==1.2.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
configparser==3.5.0
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.27.2
decorator==4.1.2
empyrical==0.2.1
enigma-catalyst==0.3
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
html5lib==1.0b10
idna==2.6
intervaltree==2.1.0
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.0.3
Jinja2==2.9.6
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.1.0
mistune==0.8
multipledispatch==0.4.9
nbconvert==5.3.1
nbformat==4.4.0
networkx==2.0
notebook==5.2.0
numexpr==2.6.4
numpy==1.13.3
pandas==0.19.2
pandas-datareader==0.5.0
pandocfilters==1.4.2
pathlib2==2.3.0
patsy==0.4.1
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.5.2
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.3
pyzmq==16.0.3
qtconsole==4.3.1
requests==2.18.4
requests-file==1.4.2
requests-ftp==0.3.1
scandir==1.6
scipy==1.0.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.14
statsmodels==0.8.0
subprocess32==3.2.7
tables==3.4.2
terminado==0.6
testpath==0.3.1
toolz==0.8.2
tornado==4.5.2
traitlets==4.3.2
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==3.0.6
Now that you know a little about me, let me tell you about the issue I am
having:
The example catalyst/catalyst/examples/buy_low_sell_high_live.py
should run live, however it is not. if I remove the comments at the bottom of the file (and comment the prior section with start and end dates) and try to run it with catalyst run -f buy_low_sell_high_live.py -o buy_low_sell_high_live.pickle
, I am having the following error: Error: must specify dates with '-s' / '--start' and '-e' / '--end' needs a start and end date.
To run live.
I received an error
Here is how you can reproduce this issue on your machine:
catalyst run -f buy_low_sell_high_live.py -o buy_low_sell_high_live.pickle
None, I believe that zipline might need some patching to make the start and end date parameters optional
Sincerely,
Albert.
Dear Catalyst Maintainers and @lacabra
I'm trying to use context.portfolio.portfolio_value
and context.portfolio.cash
to limit when I can buy I sell assets and control my leverage and handle all kind of exception but there are two problems with these values.
context.portfolio
and context.account
take too long to be updated. These are updated in the next cycle of scheduling function. However, if you decide to purchase multiple assets/coins on the cycle it can lead to errors.
Ideally, once the order has been successfully completed (i.e someone bought my coins I bought the coins I wanted) it should instantly update the values of both context.portfolio
and context.account
.
Fixed with enigma-catalyst 0.2.dev4
Sincerely,
avn3r
No way to edit submitted strategies
Dear Catalyst Maintainers,
Is there a way to load my own csv quotes into catalyst database? I have found the sample to do this on zipline - http://www.prokopyshen.com/create-custom-zipline-data-bundle, but it is not worked on catalyst...
entering viacsv. tuSymbols= ('BTC1', 'BTC2')
about to return ingest function
Traceback (most recent call last):
File "/home/user/anaconda3/bin/catalyst", line 11, in <module>
sys.exit(main())
File "/home/user/anaconda3/lib/python3.5/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/user/anaconda3/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/user/anaconda3/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/user/anaconda3/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.5/site-packages/catalyst/__main__.py", line 374, in ingest
compile_locally,
File "/home/user/anaconda3/lib/python3.5/site-packages/catalyst/data/bundles/core.py", line 556, in ingest
pth.data_path([name, timestr], environ=environ),
TypeError: ingest() takes from 9 to 11 positional arguments but 13 were given
Can only buy/sell whole coins in strategy
Run this unit test:
tests/exchange/test_bundle.py:457
With these parameters:
exchange_name = 'bitfinex'
data_frequency = 'minute'
period = '2017-06'
symbol = 'etc_btc'
Notice that the first minute of the bundle is missing:
| close | high | low | open | volume
-- | -- | -- | -- | -- | --
2017-06-01 00:00:00+00:00 | | | | | 0
2017-06-01 00:01:00+00:00 | 0.09981799 | 0.09981799 | 0.09981799 | 0.09981799 | 2.25752429
2017-06-01 00:02:00+00:00 | 0.09981799 | 0.09981799 | 0.09981799 | 0.09981799 | 0
2017-06-01 00:03:00+00:00 | 0.0993 | 0.0993 | 0.0993 | 0.0993 | 1621.348101
2017-06-01 00:04:00+00:00 | 0.098705 | 0.0993 | 0.098593 | 0.0993 | 1790.962076
2017-06-01 00:05:00+00:00 | 0.099 | 0.099 | 0.098578 | 0.098705 | 149.5418242
I suspect that it's because there is not volume in this particular minute. Since it's at the beginning of the bundle, Pandas cannot ffill the data. A solution might be to make the make the start date earlier by a first periods, then slice out the trailing data.
There needs to be some universal naming convention for currency pairs that makes it easy for the user to find the available pairs for backtesting and live trading. Currently, the error message that one gets when trying to retrieve an currency pair is confusing:
catalyst.errors.SymbolNotFound: Symbol 'XRP_USD' was not found.
Whereas a more informative and precise message would be something to the like of:
catalyst.errors.SymbolNotFound: Pair XRP_USD is not listed in Bundle: Poloniex.
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
pip
Now that you know a little about me, let me tell you about the issue I am
having:
What did you expect to happen?
Running ‘’catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle‘’ would successfully produce the graph as shown in the sample.
What happened instead?
ERROR: Loader: failed to cache treasury data
The program failed to obtain the treasury data from http://www.federalreserve.gov/, while the connection is intact.
Below is the connectivity test,
* Rebuilt URL to: www.federalreserve.gov/
* Trying 132.200.148.151...
* Connected to www.federalreserve.gov (132.200.148.151) port 80 (#0)
> GET / HTTP/1.1
> Host: www.federalreserve.gov
> User-Agent: curl/7.47.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 301 Moved Permanently
< Location: https://www.federalreserve.gov/
* HTTP/1.0 connection set to keep alive!
< Connection: Keep-Alive
< Content-Length: 0
<
* Connection #0 to host www.federalreserve.gov left intact
Here is how you can reproduce this issue on your machine:
I have not been able to reproduce the issue in a different network environment. This may just be a connectivity issue of that particular network.
I changed my connection environment later and the program was able to successfully obtain the data.
Another solution is downloading the required data manually of the federal reserve website.
I am located in China.
Sincerely,
Ricereaper
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
5.6.0 Darwin Kernel Version 15.6.0: Tue Apr 11 16:00:51 PDT 2017; root:xnu-3248.60.11.5.3~1/RELEASE_X86_64 x86_64
Python 2.7.10
64
pip install enigma-catalyst
alembic==0.9.3
bcolz==0.12.1
Bottleneck==1.2.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cyordereddict==1.0.0
Cython==0.26
decorator==4.1.2
empyrical==0.2.1
enigma-catalyst==0.1.dev6
idna==2.5
intervaltree==2.1.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
multipledispatch==0.4.9
networkx==1.11
numexpr==2.6.2
numpy==1.13.1
pandas==0.19.2
pandas-datareader==0.5.0
patsy==0.4.1
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.2
requests==2.18.2
requests-file==1.4.2
requests-ftp==0.3.1
scipy==0.19.1
six==1.10.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.12
statsmodels==0.8.0
tables==3.4.2
toolz==0.8.2
urllib3==1.22
Now that you know a little about me, let me tell you about the issue I am
having:
Catalyst fails when running an algo (starting with the buy_and_hodl.py suggested in the tutorial).
Trading algorithm should have run with no errors.
Catalyst throws the following error:
$ catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
[2017-07-30 08:20:48.089358] INFO: Loader: Loading benchmark data for 'USDT_BTC' from 1989-12-31 00:00:00+00:00 to 2017-07-28 00:00:00+00:00
[2017-07-30 08:20:48.089732] INFO: Loader: Cache at /Users/marta/.catalyst/data/USDT_BTC_benchmark.csv does not have data from 1990-01-01 00:00:00+00:00 to 2017-07-28 00:00:00+00:00.
[2017-07-30 08:20:48.090117] INFO: Loader: Downloading benchmark data for 'USDT_BTC' from 1989-12-31 00:00:00+00:00 to 2017-07-28 00:00:00+00:00
[2017-07-30 08:20:48.093315] DEBUG: catalyst.curate.poloniex: Getting data for USDT_BTC
Traceback (most recent call last):
File "/Users/marta/Victor/env-c/bin/catalyst", line 11, in <module>
sys.exit(main())
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/__main__.py", line 97, in _
return f(*args, **kwargs)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/__main__.py", line 240, in run
environ=os.environ,
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 196, in _run
env, data = get_trading_env_and_data(bundles)
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 156, in get_trading_env_and_data
environ=environ,
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/finance/trading.py", line 99, in __init__
self.bm_symbol,
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/data/loader.py", line 135, in load_crypto_market_data
environ,
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/data/loader.py", line 367, in ensure_crypto_benchmark_data
if not has_data_for_dates(daily_close, first_date, last_date):
File "/Users/marta/Victor/env-c/lib/python2.7/site-packages/catalyst/data/loader.py", line 97, in has_data_for_dates
return (first <= first_date) and (last >= last_date)
File "pandas/tslib.pyx", line 1106, in pandas.tslib._Timestamp.__richcmp__ (pandas/tslib.c:20503)
File "pandas/tslib.pyx", line 1167, in pandas.tslib._Timestamp._assert_tzawareness_compat (pandas/tslib.c:21519)
TypeError: Cannot compare tz-naive and tz-aware timestamps
Here is how you can reproduce this issue on your machine:
First time you run: catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
If you run the same command a second time, the error disappears.
No
Sincerely,
Victor
When running any trading algorithm, Catalyst throws a weekly warning to the like of:
[2017-07-30 10:57:20.808486] WARNING: Risk: No rate within 1 trading day of end date = 2015-03-08 00:00:00+00:00 and term = 10year. Using 2015-03-06 00:00:00+00:00. Check that date doesn't exceed treasury history range.
This is caused by the fact that is using regular stocks trading data as some part of the benchmark that doesn't trade on weekends. It's just a warning, and you can safely ignore it for now, but we should get rid of it.
Discord user voodeye said:
"hello, so with version 0.2.dev5, on live trading, catalyst retrives minutely candles from exchange without a problem. However, on version 0.3.3 i receive an error saying that data not available after '2017-10-22'. i am guess on the latest version live trading doesn't fetch minute data after that date. While backtesting i can use minute candles on prior dates."
Before 0.3, historical data was always retrieved using the exchange api. Since then, we try to get as much data as available in the bundle and fetch only the difference from the exchange. It looks like some conditions are causing this to fail.
Dear Catalyst Maintainers,
Currently, we have 3 exchanges which are among the most well know exchanges in terms of Volume and activity. This is great for more stable investing, but for those interested in daily and high interval trading a bigger universe will help capitalize in smaller coins that may lead to higher returns in shorter periods.
For this reason, I propose the addition of the following exchanges:
With the addition of three exchanges, we pretty much cover the entire Cryptocurrency market.
Sincerely,
avn3r
This messages usually appears more than once when running an algo:
RuntimeWarning: invalid value encountered in double_scalars
sortino = mu / dsr
RuntimeWarning: divide by zero encountered in log1p
df_cum = np.exp(nancumsum(np.log1p(returns)))
It's related to cumulative calculations probably because of the 24/7 calendar.
What did you expect to happen?
that the tutorial runs fine
catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
What happened instead?
it stops with this error:
File ".../enigmampc/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/data/loader.py", line 367, in ensure_crypto_benchmark_data
if not has_data_for_dates(daily_close, first_date, last_date):
File ".../enigmampc/catalyst/catalyst-venv/local/lib/python2.7/site-packages/catalyst/data/loader.py", line 97, in has_data_for_dates
return (first <= first_date) and (last >= last_date)
File "pandas/tslib.pyx", line 1106, in pandas.tslib._Timestamp.__richcmp__ (pandas/tslib.c:20503)
File "pandas/tslib.pyx", line 1167, in pandas.tslib._Timestamp._assert_tzawareness_compat (pandas/tslib.c:21519)
then when I re-run it, with this error
File ".../enigmampc/catalyst/catalyst-venv/local/lib/python2.7/site-packages/six.py", line 82, in _import_module
__import__(name)
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 42, in <module>
raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package
which I could fix with
sudo apt install python-tk
... and then ... it worked, and showed the diagrams.
What I had done before:
mkdir enigmampc
cd enigmampc
git clone https://github.com/enigmampc/catalyst.git
cd catalyst
virtualenv catalyst-venv
source ./catalyst-venv/bin/activate
pip install enigma-catalyst matplotlib
catalyst ingest
cd catalyst/examples/
catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
my system looks like this
uname --all
Linux ... 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux
python --version
Python 2.7.13
python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
64
installed via virtualenv and pip
pip freeze
alembic==0.9.4
bcolz==0.12.1
Bottleneck==1.2.1
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.26
decorator==4.1.2
empyrical==0.2.1
enigma-catalyst==0.1.dev6
functools32==3.2.3.post2
idna==2.5
intervaltree==2.1.0
Logbook==1.1.0
lru-dict==1.1.6
Mako==1.0.7
MarkupSafe==1.0
matplotlib==2.0.2
multipledispatch==0.4.9
networkx==1.11
numexpr==2.6.2
numpy==1.13.1
pandas==0.19.2
pandas-datareader==0.5.0
patsy==0.4.1
pyparsing==2.2.0
python-dateutil==2.6.1
python-editor==1.0.3
pytz==2017.2
requests==2.18.3
requests-file==1.4.2
requests-ftp==0.3.1
scipy==0.19.1
six==1.10.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.13
statsmodels==0.8.0
subprocess32==3.2.7
tables==3.4.2
toolz==0.8.2
urllib3==1.22
hope that helps?
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
alembic 0.9.6
backports.functools-lru-cache 1.4
bcolz 0.12.1
Bottleneck 1.2.1
certifi 2016.2.28 py27_0
certifi 2017.7.27.1
chardet 3.0.4
click 6.7
contextlib2 0.5.5
cycler 0.10.0
cyordereddict 1.0.0
Cython 0.27.1
decorator 4.1.2
empyrical 0.2.1
enigma-catalyst 0.3
idna 2.6
intervaltree 2.1.0
Logbook 1.1.0
lru-dict 1.1.6
Mako 1.0.7
MarkupSafe 1.0
matplotlib 2.1.0
mkl 2017.0.3 0
multipledispatch 0.4.9
networkx 2.0
numexpr 2.6.4
numpy 1.13.1 py27_0
openssl 1.0.2l 0
pandas 0.19.2
pandas-datareader 0.5.0
patsy 0.4.1
pip 9.0.1 py27_1
pudb 2017.1.4
Pygments 2.2.0
pyparsing 2.2.0
python 2.7.13 0
python-dateutil 2.6.1
python-editor 1.0.3
pytz 2017.2
readline 6.2 2
requests 2.18.4
requests-file 1.4.2
requests-ftp 0.3.1
scipy 0.19.1 np113py27_0
setuptools 36.4.0 py27_1
six 1.11.0
sortedcontainers 1.5.7
SQLAlchemy 1.1.14
sqlite 3.13.0 0
statsmodels 0.8.0
subprocess32 3.2.7
TA-Lib 0.4.10
tables 3.4.2
tk 8.5.18 0
toolz 0.8.2
urllib3 1.22
urwid 1.3.1
wheel 0.29.0 py27_0
zlib 1.2.11 0
Now that you know a little about me, let me tell you about the issue I am
having:
What did you expect to happen?
To ingest poloniex data via catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt
What happened instead?
$ catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt
Ingesting exchange bundle poloniex...
Exception ValueError: ValueError('chunkfile /Users/administrator/.catalyst/data/exchanges/poloniex/minute_bundle/00/35/003541.bcolz/close/data/__-21.blp not found',) in 'bcolz.carray_ext.carray.getitem_cache' ignored
[= ] Fetching poloniex minute candles: : 3%
Traceback (most recent call last):
File "/anaconda/envs/catalyst/bin/catalyst", line 11, in
sys.exit(main())
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/main.py", line 508, in ingest_exchange
show_progress=show_progress
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 453, in ingest
show_progress)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 431, in ingest_assets
empty_rows_behavior='strip'
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 232, in ingest_ctable
end_dt=end_dt
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 51, in load_raw_arrays
return self._load_raw_arrays(fields, start_dt, end_dt, sids)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 54, in _load_raw_arrays
start_idx = self._find_position_of_minute(start_dt)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/data/minute_bars.py", line 1209, in _find_position_of_minute
False,
File "catalyst/data/_minute_bar_internal.pyx", line 81, in catalyst.data._minute_bar_internal.find_position_of_minute
IndexError: Out of bounds on buffer access (axis 0)
Here is how you can reproduce this issue on your machine:
catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt
$ ls -l /Users/administrator/.catalyst/data/exchanges/poloniex/minute_bundle/00/35/003541.bcolz/close/data/
-rw-r--r-- 1 apoehlmann staff 4105 Oct 20 23:49 __0.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __1.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __10.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __11.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __12.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __13.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __14.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __15.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __16.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __17.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __18.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __19.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __2.blp
-rw-r--r-- 1 apoehlmann staff 3144 Oct 20 23:49 __20.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __3.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __4.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __5.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __6.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __7.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __8.blp
-rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __9.blp
I then moved the data folder since it's already from the previous version but this didn't solve the problem, however the logs are now slightly different, the Exception ValueError: ValueError('chunkfile [...])
part (beginning of the previous logs) is gone:
$ catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt
Ingesting exchange bundle poloniex...
[= ] Fetching poloniex minute candles: : 3%
Traceback (most recent call last):
File "/anaconda/envs/catalyst/bin/catalyst", line 11, in
sys.exit(main())
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/main.py", line 508, in ingest_exchange
show_progress=show_progress
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 453, in ingest
show_progress)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 431, in ingest_assets
empty_rows_behavior='strip'
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 232, in ingest_ctable
end_dt=end_dt
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 51, in load_raw_arrays
return self._load_raw_arrays(fields, start_dt, end_dt, sids)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 54, in _load_raw_arrays
start_idx = self._find_position_of_minute(start_dt)
File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/data/minute_bars.py", line 1209, in _find_position_of_minute
False,
File "catalyst/data/_minute_bar_internal.pyx", line 81, in catalyst.data._minute_bar_internal.find_position_of_minute
IndexError: Out of bounds on buffer access (axis 0)
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
* Operating System: (Windows Subsystem For Linux )-- Linux DESKTOP 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64 x86_64 GNU/Linux
* Python Version: Python 2.7.12
* Python Bitness: 64
* How did you install Catalyst: pip (virtual environment)
* Python packages: `$ pip freeze`
alembic==0.7.7
backports.functools-lru-cache==1.4
bcolz==0.12.1
Bottleneck==1.0.0
certifi==2017.7.27.1
chardet==3.0.4
click==4.0
contextlib2==0.4.0
cycler==0.10.0
cyordereddict==0.2.2
Cython==0.25.2
decorator==4.0.0
empyrical==0.2.1
enigma-catalyst==0.2.dev5
idna==2.6
intervaltree==2.1.0
Logbook==0.12.5
lru-dict==1.1.4
Mako==1.0.1
MarkupSafe==0.23
matplotlib==2.1.0
multipledispatch==0.4.8
networkx==1.9.1
numexpr==2.6.1
numpy==1.13.1
pandas==0.19.2
pandas-datareader==0.2.1
patsy==0.4.0
pkg-resources==0.0.0
pyparsing==2.2.0
python-dateutil==2.4.2
python-editor==1.0.3
pytz==2016.4
requests==2.9.1
requests-file==1.4.1
requests-ftp==0.3.1
scipy==0.17.1
six==1.10.0
sortedcontainers==1.4.4
SQLAlchemy==1.0.8
statsmodels==0.6.1
subprocess32==3.2.7
tables==3.3.0
toolz==0.8.2
urllib3==1.22
Now that you know a little about me, let me tell you about the issue I am
having:
No able to run. It seems to be display issue.
ethdev@DESKTOP-D3T3450 { ~/catalyst-venv/catalyst/catalyst/examples }$ catalyst run -f buy_and_hodl.py --start 2015-3-2 --end 2017-7-31 --capital-base 100000 -o bah.pickle
[2017-10-14 20:06:26.544295] INFO: run_algo: running algo in backtest mode
[2017-10-14 20:06:26.688107] INFO: Loader: Loading benchmark data for 'USDT_BTC' from 2015-03-02 00:00:00+00:00 to 2017-10-03 00:00:00
[2017-10-14 20:06:30.169306] INFO: Performance: Simulated 883 trading days out of 883.
[2017-10-14 20:06:30.169431] INFO: Performance: first open: 2015-03-02 00:00:00+00:00
[2017-10-14 20:06:30.169596] INFO: Performance: last close: 2017-07-31 23:59:00+00:00
Traceback (most recent call last):
File "/home/ethdev/catalyst-venv/bin/catalyst", line 11, in <module>
sys.exit(main())
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/click/core.py", line 664, in __call__
return self.main(*args, **kwargs)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/click/core.py", line 991, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/catalyst/__main__.py", line 98, in _
return f(*args, **kwargs)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/catalyst/__main__.py", line 294, in run
live_graph=live_graph
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 362, in _run
overwrite_sim_params=False,
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/catalyst/algorithm.py", line 749, in run
self.analyze(daily_stats)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/catalyst/algorithm.py", line 490, in analyze
self._analyze(self, perf)
File "buy_and_hodl.py", line 83, in analyze
ax1 = plt.subplot(611)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 1052, in subplot
fig = gcf()
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 588, in gcf
return figure()
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/matplotlib/pyplot.py", line 534, in figure
**kwargs)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 170, in new_figure_manager
return cls.new_figure_manager_given_figure(num, fig)
File "/home/ethdev/catalyst-venv/local/lib/python2.7/site-packages/matplotlib/backends/backend_tkagg.py", line 1049, in new_figure_manager_given_figure
window = Tk.Tk(className="matplotlib")
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1818, in __init__
self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: no display name and no $DISPLAY environment variable
Here is how you can reproduce this issue on your machine:
catalyst run -f buy_and_hodl.py --start 2015-3-2 --end 2017-7-31 --capital-base 100000 -o bah.pickle
export DISPLAY=:0
and run,same errorexport DISPLAY=0.0
and run, same errorexport DISPLAY=:0.0
and run, same errorexport DISPLAY
ethdev@DESKTOP { ~/.config/matplotlib }$ cat matplotlibrc
backend: TkAgg
echo "backend: TkAgg" > ~/.matplotlib/matplotlibrc
No additional details
Sincerely,
mesnia
I got this error with the get_candles()
method of Poloniex.
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
The error occurs when using the catalyst
command but not when calling the python process directly.
In Zipline, performance calculations are daily. While it is currently possible to run a minutely backtest with daily performance charts (zipline calculates the totals), I find it less meaningful. For example, representing a technical indicator this way does not accurately show the signals which may lead to a trade.
Since Catalyst already include a minutely stats implementation in live mode, we should port it to backtest.
The goal of the BaseBundle is to provide an extensible abstract class that encapsulates reusable ingestion logic. This will serve to help standardize custom data bundles, and make gathering new data sources simpler.
In addition to some simple parameters, the BaseBundle exposes two primary methods: fetch_raw_metadata_frame
and fetch_raw_symbol_frame
. Each of these allows an inheriting class to retrieve data and format it before handing it back to the BaseBundle to be written to disk.
The primary responsibility of fetch_raw_metadata_frame
is to obtain the list of asset pairs, or the analogous list for other datasets, in addition to other metadata pertaining to the items in the dataset. The fetch_raw_symbol_frame
is used to retrieve data about an asset during a particular time interval and at a given data frequency.
In our experience with designing the poloniex bundle, we realized that not every API will readily incorporate the metadata Catalyst requires before writing the metadata to disk. To work around this, we also expose an optional post_process_symbol_metadata
, which allows a user to finalize an asset's metadata after fetching the raw symbol data. In the case of the poloniex bundle, we use this method to determine the proper start and end dates for each asset pair, which is inferable once we can determine the first and last record of each asset in the ingested data.
The initial work for this feature began here 1f24750 and later improved by this commit b82f719. As of the latter commit, we have also reimplemented our poloniex bundle using this new template, and have confirmed that the datasets are compatible back to version 0.1.dev4 (last major change to db format).
Ingest bundle data like this:
catalyst ingest-exchange -x poloniex -f daily -i xcp_btc
Run an algo containing a data.history() call with these parameters:
%%catalyst -x poloniex --start 2015-3-2 --end 2017-8-31 --capital-base 1
I don't have the algo but the call looks like this:
data.history(symbol(str(LP.iloc[i]['coin'])),bar_count=1,fields='close',frequency='1d')
Here is the error (stack trace attached):
NoDataAvailableOnExchange: Requested data for trading pair [u'xcp_btc'] is not available on exchange ['poloniex'] in `daily` frequency at this time. Check `http://enigma.co/catalyst/status` for market coverage.[fff.txt](https://github.com/enigmampc/catalyst/files/1443544/fff.txt)
Run:
catalyst clean-exchange -x bitfinex
catalyst ingest-exchange -x bitfinex -i btc_usd
Notice this entry in ~/.catalyst/data/exchanges/bitfinex/symbols.json
:
"btcusd":{
"end_daily":"2017-11-01",
"end_minute":"2017-11-01",
"start_date":"2013-03-31",
"symbol":"btc_usd"
}
The end_daily
value does not match the content of the bundle:
2017-10-29 00:00:00+00:00 6150.0 6345.0 5660.0 5719.7 61967.826833
2017-10-30 00:00:00+00:00 6130.0 6239.0 6006.2 6150.0 33217.404640
2017-10-31 00:00:00+00:00 6455.1 6481.5 6082.5 6127.9 38821.109194
2017-11-01 00:00:00+00:00 6739.6 6739.9 6344.5 6455.0 57203.787342
2017-11-02 00:00:00+00:00 7018.8 7350.0 6707.3 6739.2 97337.200722
A bundle like this one:
exchange_name = 'bitfinex'
data_frequency = 'minute'
period = '2017-06'
symbol = 'etc_btc'
has zero volume between 2017-06-18 and 2017-06-30
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
$ python --version
$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
pip
, conda
)So I tried using the command :catalyst ingest-exchange -x {bittrex}
However I get this back.
(C:\ana) C:\Users\user C-137>activate catalyst
(catalyst) C:\Users\user C-137>catalyst ingest-exchange -x {bittrex}
Usage: catalyst ingest-exchange [OPTIONS]
Error: Invalid value for "-x" / "--exchange-name": invalid choice: {bittrex}. (choose from poloniex, bitfinex, bittrex)
(catalyst) C:\Users\user C-137>catalyst ingest-exchange -x {exchange-bittrex}
Usage: catalyst ingest-exchange [OPTIONS]
Error: Invalid value for "-x" / "--exchange-name": invalid choice: {exchange-bittrex}. (choose from poloniex, bitfinex, bittrex)
(catalyst) C:\Users\user C-137>catalyst ingest-exchange -x {exchange_bittrex}
Usage: catalyst ingest-exchange [OPTIONS]
Error: Invalid value for "-x" / "--exchange-name": invalid choice: {exchange_bittrex}. (choose from poloniex, bitfinex, bittrex)
(catalyst) C:\Users\user C-137>catalyst ingest-exchange -x {Poloniex}
Usage: catalyst ingest-exchange [OPTIONS]
Error: Invalid value for "-x" / "--exchange-name": invalid choice: {Poloniex}. (choose from poloniex, bitfinex, bittrex)
(catalyst) C:\Users\user C-137>catalyst ingest-exchange -x {Poloniex}
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
$ anaconda 3
$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
pip
, conda
, or other (please explain)
)$ pip freeze
or $ conda list
The ingest function is not working tried multiple times specifying exchange howevwewr it does not work
(C:\ana) C:\Users\user C-137>activate catalyst
(catalyst) C:\Users\user C-137>catalyst ingest --bittrex-b
Error: no such option: --bittrex-b
(catalyst) C:\Users\user C-137>catalyst ingest --bittrex -b
Error: no such option: --bittrex
(catalyst) C:\Users\user C-137>catalyst ingest -b
Error: -b option requires an argument
(catalyst) C:\Users\user C-137>catalyst ingest
Error: No bundle registered with the name None
(catalyst) C:\Users\user C-137>
set_max_leverage(1.0) does not allow order_target_percent(context.asset, 1.0)
Should be <= not <
Expose list of currency pairs in Catalyst so that a user can query and obtain that information by exchange.
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
Ingest Data not being loaded for many markets. I have only focused on Poloniex minute data but here is a list of pairs that currently do not work with version 0.3.1
Data History is not accurate. data prices are not matching exchanges prices on that date at that time.
This may be because data from the 3 exchanges are being now aggregated, but that's not the only issue.
Data values are spiking/jumpy. Example prices increase from 1.4e-6 to 1.6e-6 in 30 minutes when the expected behavior (on exchange candle data) was that it will incrementally increase from 1.4e-6 to 1.43 to 1.5 to 1.55 to 1.6. These are just hypothetical numbers but I have seen this behavior in most currencies since the update.
These two errors did NOT exist on version 0.2.X
Here is how you can reproduce this issue on your machine:
import pandas as pd
from catalyst import run_algorithm
from catalyst.exchange.exchange_utils import get_exchange_symbols
from catalyst.api import (
symbols,
)
def initialize(context):
context.i = -1
context.base_currency = 'btc'
def handle_data(context, data):
lookback = 60 * 24 * 7 # (minutes, hours, days)
context.i += 1
if context.i < lookback:
return
today = context.blotter.current_dt.strftime('%Y-%m-%d %H:%M:%S')
try:
# update universe everyday
new_day = 60 * 24
if not context.i % new_day:
context.universe = universe(context, today)
# get data every 30 minutes
minutes = 30
if not context.i % minutes and context.universe:
for coin in context.coins:
pair = str(coin.symbol)
# ohlcv data
open = data.history(coin, 'open', lookback, '1m').ffill().bfill().resample('30T').first()
high = data.history(coin, 'high', lookback, '1m').ffill().bfill().resample('30T').max()
low = data.history(coin, 'low', lookback, '1m').ffill().bfill().resample('30T').min()
close = data.history(coin, 'price', lookback, '1m').ffill().bfill().resample('30T').last()
volume = data.history(coin, 'volume', lookback, '1m').ffill().bfill().resample('30T').sum()
print(today, pair, close[-1])
except Exception as e:
print(e)
def analyze(context=None, results=None):
pass
def universe(context, today):
json_symbols = get_exchange_symbols('poloniex')
poloniex_universe_df = pd.DataFrame.from_dict(json_symbols).transpose().astype(str)
poloniex_universe_df['base_currency'] = poloniex_universe_df.apply(lambda row: row.symbol.split('_')[1],
axis=1)
poloniex_universe_df['market_currency'] = poloniex_universe_df.apply(lambda row: row.symbol.split('_')[0],
axis=1)
poloniex_universe_df = poloniex_universe_df[poloniex_universe_df['base_currency'] == context.base_currency]
poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'gas_btc']
# Markets currently not working on Catalyst 0.3.1
# 2017-01-01
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'bcn_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'burst_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'dgb_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'doge_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'emc2_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'pink_btc']
# poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.symbol != 'sc_btc']
date = str(today).split(' ')[0]
poloniex_universe_df = poloniex_universe_df[poloniex_universe_df.start_date < date]
context.coins = symbols(*poloniex_universe_df.symbol)
print(len(poloniex_universe_df))
return poloniex_universe_df.symbol.tolist()
if __name__ == '__main__':
start_date = pd.to_datetime('2017-01-01', utc=True)
end_date = pd.to_datetime('2017-10-15', utc=True)
performance = run_algorithm(start=start_date, end=end_date,
capital_base=10000.0,
initialize=initialize,
handle_data=handle_data,
analyze=analyze,
exchange_name='poloniex',
data_frequency='minute',
base_currency='btc',
live=False,
live_graph=False,
algo_namespace='test')
[2017-10-24 19:06:46.909017] INFO: exchange_bundle: pricing data for [u'bcn_btc'] not found in range 2017-01-01 05:30:00+00:00 to 2017-01-08 05:30:00+00:00, updating the bundles.
[====================================] Fetching poloniex minute candles: : 100%
Pricing data open for trading pairs bcn_btc trading on exchange poloniex since 2014-05-20 00:00:00+00:00 is unavailable. The bundle data is either out-of-date or has not been loaded yet. Please ingest data using the command `catalyst ingest-exchange -x poloniex -f minute -i bcn_btc`. See catalyst documentation for details.
Since it crashes on bcn_btc pair all other pairs in that 30 minute cycle skipped, without printing ther values.
Sincerely,
avn3r
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
alembic 0.9.6
backports.functools-lru-cache 1.4
bcolz 0.12.1
Bottleneck 1.2.1
certifi 2017.7.27.1
certifi 2016.2.28 py27_0
chardet 3.0.4
click 6.7
contextlib2 0.5.5
cycler 0.10.0
cyordereddict 1.0.0
Cython 0.27.1
decorator 4.1.2
empyrical 0.2.1
enigma-catalyst 0.3.1
idna 2.6
intervaltree 2.1.0
libgfortran 3.0.0 1
Logbook 1.1.0
lru-dict 1.1.6
Mako 1.0.7
MarkupSafe 1.0
matplotlib 2.1.0
mkl 2017.0.3 0
multipledispatch 0.4.9
networkx 2.0
numexpr 2.6.4
numpy 1.13.1 py27_0
openssl 1.0.2l 0
pandas 0.19.2
pandas-datareader 0.5.0
patsy 0.4.1
pip 9.0.1 py27_1
pyparsing 2.2.0
python 2.7.13 0
python-dateutil 2.6.1
python-editor 1.0.3
pytz 2017.2
readline 6.2 2
requests 2.18.4
requests-file 1.4.2
requests-ftp 0.3.1
scipy 0.19.1 np113py27_0
setuptools 36.4.0 py27_1
six 1.11.0
sortedcontainers 1.5.7
SQLAlchemy 1.1.14
sqlite 3.13.0 0
statsmodels 0.8.0
subprocess32 3.2.7
tables 3.4.2
tk 8.5.18 0
toolz 0.8.2
urllib3 1.22
wheel 0.29.0 py27_0
zlib 1.2.11 0
Now that you know a little about me, let me tell you about the issue I am
having:
Run buy_and_hold.py example:
catalyst run -f buy_and_hodl.py -x bitfinex --start 2016-1-1 --end 2017-9-30 -o buy_and_hodle_out.pickle
What did you expect to happen?
I expect to it should work the example
What happened instead?
I get this error:
[2017-10-26 20:27:18.844831] INFO: run_algo: running algo in backtest mode
[2017-10-26 20:27:19.814061] INFO: exchange_algorithm: initialized trading algorithm in backtest mode
Traceback (most recent call last):
File "/Users/gnc/miniconda2/envs/catalyst/bin/catalyst", line 11, in
sys.exit(main())
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/main.py", line 101, in _
return f(*args, **kwargs)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/main.py", line 275, in run
live_graph=False
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/run_algo.py", line 336, in _run
overwrite_sim_params=False,
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/algorithm.py", line 725, in run
for perf in self.get_generator():
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform
for capital_change_packet in every_bar(dt):
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar
handle_data(algo, current_data, dt_to_use)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/events.py", line 216, in handle_data
dt,
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/utils/events.py", line 235, in handle_data
self.callback(context, data)
File "/Users/gnc/miniconda2/envs/catalyst/lib/python2.7/site-packages/catalyst/algorithm.py", line 474, in handle_data
self._handle_data(self, data)
File "buy_and_hodl.py", line 59, in handle_data
price = data[context.asset].price
File "catalyst/_protocol.pyx", line 817, in catalyst._protocol.BarData.getitem
File "catalyst/_protocol.pyx", line 211, in catalyst._protocol.BarData._get_equity_price_view
AttributeError: 'NoneType' object has no attribute 'retrieve_asset'
Here is how you can reproduce this issue on your machine:
...
...
...
Sincerely,
$ whoami
The Zipline open calendar starts on 2015-03-01 which currently causes a limitation in Catalyst.
A single user might want to run algos against more than one account of the same exchange. Currently, it not possible because the naming convention under .catalyst
. Add an alias parameter which can reference specific accounts of an exchange.
Hello
I was reading the documentation regarding this library but I'm curious about the library of mpc that you use.
In the whitepaper you mention the mpc to anonymize the information, but in the external library you do not mention any mpc library. Which mpc library do you use ?
Thanks in advance.
Dear Zipline Maintainers,
Before I tell you about my issue, let me describe my environment:
(catalyst-venv) bash-3.2$ uname -a
Darwin Bens-MacBook-Pro.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64
(catalyst-venv) bash-3.2$ python --version
python --version
Python 3.6.1
(catalyst-venv) bash-3.2$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
64
How did you install Zipline: Installed from source using README.rst
.
Python packages:
(catalyst-venv) bash-3.2$ pip freeze
alembic==0.9.3
bcolz==0.12.1
Bottleneck==1.2.1
certifi==2017.4.17
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.25.2
decorator==4.0.11
empyrical==0.2.1
enigma-catalyst==0.1.dev5
idna==2.5
intervaltree==2.1.0
Logbook==1.0.0
lru-dict==1.1.6
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.2
multipledispatch==0.4.9
networkx==1.11
numexpr==2.6.2
numpy==1.13.1
pandas==0.19.2
pandas-datareader==0.4.0
patsy==0.4.1
pyparsing==2.2.0
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
requests==2.18.1
requests-file==1.4.2
requests-ftp==0.3.1
scipy==0.19.1
six==1.10.0
sortedcontainers==1.5.7
SQLAlchemy==1.1.11
statsmodels==0.8.0
tables==3.4.2
toolz==0.8.2
urllib3==1.21.1
Now that you know a little about me, let me tell you about the issue I am
having:
Example
What did you expect to happen?
Expected library to import with no errors.
What happened instead?
Got an error about an invalid literal. Looks like a typo
Here is how you can reproduce this issue on your machine:
(catalyst-venv) Bens-MacBook-Pro:catalyst bkc$ git branch
* master
(catalyst-venv) Bens-MacBook-Pro:catalyst bkc$ git pull
Already up-to-date.
(catalyst-venv) Bens-MacBook-Pro:catalyst bkc$ python
Python 3.6.1 (default, Apr 4 2017, 09:40:21)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from catalyst.api import (cancel_order, symbol, record, order_target_value, get_open_orders)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/bkc/Documents/code/projects/catalyst/catalyst/__init__.py", line 20, in <module>
from . import data
File "/Users/bkc/Documents/code/projects/catalyst/catalyst/data/__init__.py", line 1, in <module>
from . import loader
File "/Users/bkc/Documents/code/projects/catalyst/catalyst/data/loader.py", line 36, in <module>
from catalyst.curate.poloniex import PoloniexCurator
File "/Users/bkc/Documents/code/projects/catalyst/catalyst/curate/poloniex.py", line 9
DT_START = time.mktime(datetime(2010, 01, 01, 0, 0).timetuple())
^
SyntaxError: invalid token
Should be a simple fix. Just logging here
Nope
Sincerely,
Ben
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
Now that you know a little about me, let me tell you about the issue I am
having:
Traceback (most recent call last):
File "/Users/leotreasure/catalyst-venv/bin/catalyst", line 11, in <module>
sys.exit(main())
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/catalyst/__main__.py", line 97, in _
return f(*args, **kwargs)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/catalyst/__main__.py", line 240, in run
environ=os.environ,
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 229, in _run
'script': algotext,
File "/Users/leotreasure/catalyst-venv/lib/python3.6/site-packages/catalyst/algorithm.py", line 377, in __init__
code = compile(self.algoscript, filename, 'exec')
File "buy_and_hodl.py", line 45
print 'i:', context.i
^
SyntaxError: Missing parentheses in call to 'print'
Here is how you can reproduce this issue on your machine:
Run:
catalyst run -f buy_and_hodl.py --start 2015-3-1 --end 2017-6-28 --capital-base 100000 -o bah.pickle
Dear Catalyst Maintainers,
Before I tell you about my issue, let me describe my environment:
.yml
filechardet 3.0.4
click 6.7
contextlib2 0.5.5
cycler 0.10.0
cyordereddict 1.0.0
Cython 0.27.1
decorator 4.1.2
empyrical 0.2.1
enigma-catalyst 0.2.dev5
idna 2.6
intervaltree 2.1.0
Logbook 1.1.0
lru-dict 1.1.6
Mako 1.0.7
MarkupSafe 1.0
matplotlib 2.1.0
mkl 2017.0.3 0
multipledispatch 0.4.9
networkx 2.0
numexpr 2.6.4
numpy 1.13.1 py27_0
openssl 1.0.2l 0
pandas 0.19.2
pandas-datareader 0.5.0
patsy 0.4.1
pip 9.0.1 py27_1
pudb 2017.1.4
Pygments 2.2.0
pyparsing 2.2.0
python 2.7.13 0
python-dateutil 2.6.1
python-editor 1.0.3
pytz 2017.2
readline 6.2 2
requests 2.18.4
requests-file 1.4.2
requests-ftp 0.3.1
scipy 0.19.1 np113py27_0
setuptools 36.4.0 py27_1
six 1.11.0
sortedcontainers 1.5.7
SQLAlchemy 1.1.14
sqlite 3.13.0 0
statsmodels 0.8.0
subprocess32 3.2.7
TA-Lib 0.4.10
tables 3.4.2
tk 8.5.18 0
toolz 0.8.2
urllib3 1.22
urwid 1.3.1
wheel 0.29.0 py27_0
zlib 1.2.11 0
Now that you know a little about me, let me tell you about the issue I am
having:
When running the algorithm on a minute basis I've noticed that it can take quite some time until a placed order gets executed:
Here you can see that even though my limit is set 10% above the current price, it still takes quite some time until the order is being executed (up to >1h):
finally executed:
Some code parts for creating the logs, in case its good for anything.
So if I set an order with such a high buy price I’d expect it to be executed quite quickly.
Is this expected behaviour or am I doing something wrong?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.