The pylivetrader from alpacahq

Seemingly hanging after pylivetrader run -f algo.py

Ubuntu 16.04 Via VMWare on Mac.

I set my env variables accordingly and run:

pylivetrader run -f algo.py

Thereafter this is the readout:

[2019-06-11 08:24:47.971239] INFO: Algorithm: livetrader start running with backend = alpaca data-frequency = minute

It then hangs here for a long time; hours.
What seems to be the issue here?

For reference the code in algo.py is the example pylivetrader algorithm:

from pylivetrader.api import *


def initialize(context):
    context.i = 0
    context.asset = symbol('AAPL')

def handle_data(context, data):
    # Compute averages
    # data.history() has to be called with the same params
    # from above and returns a pandas dataframe.
    short_mavg = data.history(context.asset, 'price', bar_count=100, frequency="1m").mean()
    long_mavg = data.history(context.asset, 'price', bar_count=300, frequency="1m").mean()`

    # Trading logic
    if short_mavg > long_mavg:
        # order_target orders as many shares as needed to
        # achieve the desired number of shares.
        order_target(context.asset, 100)
    elif short_mavg < long_mavg:
        order_target(context.asset, 0)

history() vs data.current() discrepancies fix

See code below to reproduce and verify when fixed using pylivetrader on AP.

An estimated 94% of stocks on Alpaca (AP) show differences in history() vs current() or other discrepancies.
Expected 0 issues like on Quantopian (Q).

Last summaries when stopped (see Definitions)

     AP LOW VOLUME                    AP HIGH VOLUME                  QUANTOPIAN ALL
737 odd of 744 stocks all time   663 odd of 743 stocks all time   0 odd of 917 stocks all time
hst_mismatch_cls 1               hst_mismatch_cls 0               hst_mismatch_cls 0
hst_mismatch_prc 719             hst_mismatch_prc 661             hst_mismatch_prc 0
     hst_prc_nan 1                    hst_prc_nan 3                    hst_prc_nan 0
         hst_old 43                       hst_old 11                       hst_old 0
    prc_int_zero 0                   prc_int_zero 0                   prc_int_zero 0
     prc_integer 13                   prc_integer 13                   prc_integer 0

(on AP, 1487 total stocks examined since two runs, low and high volume separated)

Percent of inconsistencies (any type)

99% in low volume stocks (737/744)
89% in high volume stocks (663/743)

Worst prevalence: `hst_mismatch_prc` problem isolated

Difference between history() and data.current(stock, 'price')
96% in low volume stocks (719/744)
89% in high volume stocks (661/743)
2% average percent diff in hst_mismatch_prc

Second-worst problem: `hst_old` problem isolated

History not up to current minute
3.6% in all stocks examined (54/1487)

Outputs from history tail(3) and data.current() at the timestamp using get_datetime().

`hst_mismatch_prc` example instance

Most extreme history() vs current 'price' mismatch, way off, by 33% ...

2019-01-02 09:31:00-05:00 get_datetime()    hst_mismatch_prc   2.75  2.06   (33%)
   AVCO history  price   tail():
2018-12-31 10:09:00-05:00    2.75
2018-12-31 10:10:00-05:00    2.75
2018-12-31 10:11:00-05:00    2.75        <== 2.75
   AVCO history  close   tail():
2018-12-31 10:09:00-05:00     NaN
2018-12-31 10:10:00-05:00     NaN
2018-12-31 10:11:00-05:00    2.75
   data.current(AVCO,  'price'): 2.06    <== 2.06
   data.current(AVCO,  'close'): 2.75
   data.current(AVCO, 'volume'): 400.0

`hst_mismatch_cls` example instance

The latest value from history() doesn't match current 'close'. How?

2019-01-02 09:31:00-05:00 get_datetime()    hst_mismatch_cls   1.73  1.72
   TUES history  price   tail():
2019-01-02 09:53:00-05:00    1.72
2019-01-02 09:54:00-05:00    1.74
2019-01-02 09:55:00-05:00    1.73
   TUES history  close   tail():
2019-01-02 09:53:00-05:00    1.72
2019-01-02 09:54:00-05:00    1.74
2019-01-02 09:55:00-05:00    1.73        <== 1.73
   data.current(TUES,  'price'): 1.73
   data.current(TUES,  'close'): 1.72    <== 1.72 (rare oddity)
   data.current(TUES, 'volume'): 400.0

`hst_old` example instance

2018-12-28 is from some time window in the past here when run on 2019-01-02.
This happens quit a bit, even on higher volume stocks.
(Earlier code when run on 12-24 found an instance nearly a month old)

2019-01-02 09:31:00-05:00 get_datetime()    hst_old prc
   TMSR history  price   tail():
2018-12-28 14:31:00-05:00    1.53
2018-12-28 14:32:00-05:00    1.53
2018-12-28 14:33:00-05:00    1.53        <== 2018-12-28
   TMSR history  close   tail():
2018-12-28 14:31:00-05:00     NaN
2018-12-28 14:32:00-05:00     NaN
2018-12-28 14:33:00-05:00    1.53
   data.current(TMSR,  'price'): 1.53
   data.current(TMSR,  'close'): 1.53
   data.current(TMSR, 'volume'): 400.0

`hst_prc_nan` example instance

How can 'price' in history() (since it is always supposed to be forward-filled) ever be NaN?

2019-01-02 09:31:00-05:00 get_datetime()    hst_prc_nan   nan  2.13
   IDN history  price   tail():
2018-12-31 15:58:00-05:00   NaN
2018-12-31 15:59:00-05:00   NaN
2018-12-31 16:00:00-05:00   NaN          <== NaN
   IDN history  close   tail():
2018-12-31 15:58:00-05:00   NaN
2018-12-31 15:59:00-05:00   NaN
2018-12-31 16:00:00-05:00   NaN
   data.current(IDN,  'price'): 2.13     <== float
   data.current(IDN,  'close'): nan
   data.current(IDN, 'volume'): nan

Also note, when volume is nan, should be set as 0 to match Q

`prc_integer` example instance

Why is current price sometimes an integer?

2019-01-02 10:55:00-05:00 get_datetime()    prc_integer   1.98  2
   ONCY history  price   tail():
2019-01-02 10:54:00-05:00    1.98
2019-01-02 10:55:00-05:00    1.98
2019-01-02 10:56:00-05:00    1.98        <== float
   ONCY history  close   tail():
2019-01-02 10:54:00-05:00     NaN
2019-01-02 10:55:00-05:00     NaN
2019-01-02 10:56:00-05:00    1.98
   data.current(ONCY,  'price'): 2       <== integer
   data.current(ONCY,  'close'): 1.98
   data.current(ONCY, 'volume'): 800.0

Definitions

prc              data.current(stock, 'price')
cls              data.current(stock, 'close')
hst_mismatch_prc data.history(stock, 'price' ...) vs data.current(stock, 'price')
hst_mismatch_cls data.history(stock, 'close' ...) vs data.current(stock, 'close')
hst_old          Mismatch in history latest date for either prc or cls
hst_prc_nan      data.history(stock, 'price' ...) returns NaN
prc_int_zero     data.current(stock, 'price')     returns 0 [saw this once]
prc_integer      data.current(stock, 'price')     returns an integer instead of float

For interoperability, history() should always be up to the current minute.
They are currently sometimes varying time periods in the past.

Q backtest summaries

2018-12-21 05:45 before_trading_start:87 INFO 750 stocks today
2018-12-21 12:59 log_counts:190 INFO 0 odd of 750 stocks all time
2018-12-21 12:59 log_counts:193 INFO hst_mismatch_cls 0
2018-12-21 12:59 log_counts:193 INFO hst_mismatch_prc 0
2018-12-21 12:59 log_counts:193 INFO          hst_old 0
2018-12-21 12:59 log_counts:193 INFO      hst_prc_nan 0
2018-12-21 12:59 log_counts:193 INFO     prc_int_zero 0
2018-12-21 12:59 log_counts:193 INFO      prc_integer 0
2018-12-24 05:45 before_trading_start:87 INFO 750 stocks today
2018-12-24 09:59 log_counts:190 INFO 0 odd of 846 stocks all time
2018-12-24 09:59 log_counts:193 INFO hst_mismatch_cls 0
2018-12-24 09:59 log_counts:193 INFO hst_mismatch_prc 0
2018-12-24 09:59 log_counts:193 INFO          hst_old 0
2018-12-24 09:59 log_counts:193 INFO      hst_prc_nan 0
2018-12-24 09:59 log_counts:193 INFO     prc_int_zero 0
2018-12-24 09:59 log_counts:193 INFO      prc_integer 0
2018-12-26 05:45 before_trading_start:87 INFO 750 stocks today
2018-12-26 12:59 log_counts:190 INFO 0 odd of 917 stocks all time
2018-12-26 12:59 log_counts:193 INFO hst_mismatch_cls 0
2018-12-26 12:59 log_counts:193 INFO hst_mismatch_prc 0
2018-12-26 12:59 log_counts:193 INFO          hst_old 0
2018-12-26 12:59 log_counts:193 INFO      hst_prc_nan 0
2018-12-26 12:59 log_counts:193 INFO     prc_int_zero 0
2018-12-26 12:59 log_counts:193 INFO      prc_integer 0

Algorithm code used on both AP and Q

See attached algo_code_for_history_vs_current_bug_report.txt link below ...

Running that, replacing all 'close' with 'high' and others in OHLCV could possibly provide clues too.

Need to know which is more trustworthy (history() or data.current()) in the interim as soon as known.

algo_code_for_history_vs_current_bug_report.txt

KeyError on pylivetrader restart

It was reported that when pylivetrader's execution is interrupted (E.g. with a Ctrl-C) and restarted, it can encounter a KeyError:

C:\ap>pylivetrader run -f paper50.py 2>>&1 | tee output50.txt

[2018-12-11 09:59:12.829549-0500] INFO: : AP

[2018-12-11 09:59:12.845149-0500] INFO: Algorithm: livetrader start running with backend = alpaca data-frequency = minute

Traceback (most recent call last):

  File "c:\python36\lib\runpy.py", line 193, in _run_module_as_main

    "__main__", mod_spec)

  File "c:\python36\lib\runpy.py", line 85, in _run_code

    exec(code, run_globals)

  File "C:\Python36\Scripts\pylivetrader.exe\__main__.py", line 9, in <module>

  File "c:\python36\lib\site-packages\click\core.py", line 764, in __call__

    return self.main(*args, **kwargs)

  File "c:\python36\lib\site-packages\click\core.py", line 717, in main

    rv = self.invoke(ctx)

  File "c:\python36\lib\site-packages\click\core.py", line 1137, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "c:\python36\lib\site-packages\click\core.py", line 956, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "c:\python36\lib\site-packages\click\core.py", line 555, in invoke

    return callback(*args, **kwargs)

  File "c:\python36\lib\site-packages\click\decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "c:\python36\lib\site-packages\pylivetrader\__main__.py", line 144, in run

    algorithm.run(retry=ctx.retry)

  File "c:\python36\lib\site-packages\pylivetrader\algorithm.py", line 242, in run

    return self.executor.run(retry=retry)

  File "c:\python36\lib\site-packages\pylivetrader\executor\executor.py", line 120, in run

    algo.before_trading_start(self.current_data)

  File "c:\python36\lib\site-packages\pylivetrader\algorithm.py", line 215, in before_trading_start

    self._before_trading_start(self, data)

  File "paper50.py", line 236, in before_trading_start

    c.pipe = pipeline_output('pipe')

  File "c:\python36\lib\site-packages\pylivetrader\misc\api_context.py", line 62, in wrapped

    return getattr(algorithm, f.__name__)(*args, **kwargs)

  File "c:\python36\lib\site-packages\pylivetrader\algorithm.py", line 1049, in pipeline_output

    output = eng.run_pipeline(self._pipelines[name])

KeyError: 'pipe'

The error goes away with state file deletion. We should find the root cause of the issue and fix it.

Reducing Huge Memory Footprint For Heroku

This is of course not a bug, but I'm wondering if there's a any way to make the memory footprint of a running algo smaller. Currently, I've tried running my algo on Heroku against the paper trading api in Alpaca, and I'm seeing that Heroku is flagging the memory usage. From what I can tell, the app is hitting something just north of 800MB, and Heroku limits ram to 512MB for anything that you don't have to shell out the bucks for. Now, granted, I do have a couple of factors that are acting on 6 month windows of daily data, so that might have something to do with it, but is there any way to make things more efficient?

Allow algo's to influence client_order_id for algo grouping

It is useful to use client_order_id to help group different algos. If there was a way to define a order_prefix for example it could be used in alpaca.Backend._new_order_id for users to change.

get_order() doesn't return orders already filled

On Alpaca, orders are deleted when fully filled, can no longer do get_order() on their id.
So https://www.quantopian.com/posts/track-orders can't work then.
That's a bad thing because such code can be used for both diagnostics and active things like determining when to change a stop and/or limit (a [partial] fill occurred).
Quite powerful in my view, giving one an edge.
Some folks coming from Quantopian (Q) might be unpleasantly surprised in migrating to Alpaca, using that code or in their own order retrieval purposes.
I myself use that heavily as a core component of my strategy.

The workaround involves extra code less easy to follow, manually saving all order objects with get_order() immediately after every order (a slowdown that risks a missed minute by taking too much time), and rechecking them later to see whether they have disappeared, for the logging of a most recent partial fill, also involving my more extensive diagnostics I have found so useful.

This post has a backtest that demonstrates Q's order objects are always available:
https://www.quantopian.com/posts/error-execution-timeout-1#5bd8be2e7eb72d1bc7e07c1a

The web API can return the full history of orders, but I would say configurable (2 days or so by default) limit (time-based or number-based) would be nice.

Issue with pylivetrader for NAN values

My app is stuck in fetch with following warning.
/app/.heroku/python/lib/python3.6/site-packages/zipline/pipeline/filters/filter.py:382: RuntimeWarning: invalid value encountered in less_equal
2018-11-16T19:42:48.281114+00:00 app[worker.1]: return (lower_bounds <= data) & (data <= upper_bounds)
2018-11-16T19:42:52.683746+00:00 app[worker.1]: 0

Total PL Calculation is calculated incorrectly

Numerous reports of total account balance being up but calculation showing a negative PL.

Update docket container to latest version

Hello can you update the docker containers in pylivetrader to most recent versions? Looks like it’s still on 0.25

https://hub.docker.com/r/alpacamarkets/pylivetrader

API access error for paper trading

Running the code for example q01 (using config.yaml file rather than exporting environment variables) throws the following error:

(myvirtualenv) ➜  src pylivetrader run -f algo.py --backend-config ../relative/path/to/config.yaml 

[2018-12-17 02:13:35.133648] INFO: Algorithm: livetrader start running with backend = alpaca data-frequency = minute
Traceback (most recent call last):
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/rest.py", line 113, in _one_request
    resp.raise_for_status()
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.alpaca.markets//v1/assets?assert_class=us_equity

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/python-virtualenvs/myvirtualenv/bin/pylivetrader", line 11, in <module>
    sys.exit(main())
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/__main__.py", line 144, in run
    algorithm.run(retry=ctx.retry)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 232, in run
    self.asset_finder.retrieve_all(self.asset_finder.sids)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/assets/finder.py", line 192, in sids
    for sid, _ in self._asset_cache.items()
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/assets/finder.py", line 38, in _asset_cache
    for asset in self.backend.get_equities()
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/backend/alpaca.py", line 130, in get_equities
    raw_assets = self._api.list_assets(asset_class='us_equity')
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/rest.py", line 218, in list_assets
    resp = self.get('/assets', params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/rest.py", line 129, in get
    return self._request('GET', path, data)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/rest.py", line 92, in _request
    return self._one_request(method, url, opts, retry)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/rest.py", line 121, in _one_request
    raise APIError(error, http_error)
alpaca_trade_api.rest.APIError: access key verification failed : access key not found (Code = 40110000) (Code = 40110000)

Am using the public and secret API keys from Alpaca account for paper trading, eg.

(myvirtualenv) ➜  src cat ../relative/path/to/config.yaml
key_id: XXXXXX
secret: XXXXXX/XXXXXX/XXXXXX
base_url: https://api.alpaca.markets/

Copying the same format as here (https://github.com/alpacahq/pylivetrader#simple-usage) (using base_url: https://api.alpaca.markets without trailing / does not help either, nor does removing the base_url peoperty from the .yaml). Exporting the values to environment as in the original example does not work either.

--storage-engine is not an option

(pylivetraderenv) >pylivetrader run -l DEBUG -f bardata.py --backend-config config_pylivetrader.yml --storage-engine redis
Usage: pylivetrader run [OPTIONS] [ALGOFILE]...
Try "pylivetrader run --help" for help.

Error: no such option: --storage-engine

Polygon API hits 500 upon handle_data call

Hey there,

This seems to be a problem with the Polygon API as run through pylivetrader. In my handle_data function of my algorithm, I run the following command:

iwv_close = data.history(context.iwv, "price", context.lookback , "1d")

This causes the following 500 HTTP error from Polygon:

2019-02-19T18:28:00.858854+00:00 app[worker.1]: [2019-02-19 18:28:00.858001] ERROR: Executor: 500 Server Error: Internal Server Error for url: https://api.polygon.io/v1/historic/agg/day/IWV?limit=180&apiKey={API_KEY_OMITTED}
2019-02-19T18:28:00.858867+00:00 app[worker.1]: Traceback (most recent call last):
2019-02-19T18:28:00.858870+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/executor/executor.py", line 67, in wrapper
2019-02-19T18:28:00.858872+00:00 app[worker.1]:     func(*args, **kwargs)
2019-02-19T18:28:00.858874+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/executor/executor.py", line 88, in every_bar
2019-02-19T18:28:00.858875+00:00 app[worker.1]:     handle_data(algo, current_data, dt_to_use)
2019-02-19T18:28:00.858877+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/misc/events.py", line 218, in handle_data
2019-02-19T18:28:00.858879+00:00 app[worker.1]:     dt,
2019-02-19T18:28:00.858880+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/misc/events.py", line 237, in handle_data
2019-02-19T18:28:00.858882+00:00 app[worker.1]:     self.callback(context, data)
2019-02-19T18:28:00.858887+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/algorithm.py", line 203, in handle_data
2019-02-19T18:28:00.858889+00:00 app[worker.1]:     self._handle_data(self, data)
2019-02-19T18:28:00.858890+00:00 app[worker.1]:   File "main.py", line 15, in handle_data
2019-02-19T18:28:00.858892+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/data/bardata.py", line 180, in history
2019-02-19T18:28:00.858894+00:00 app[worker.1]:     self.data_frequency,
2019-02-19T18:28:00.858895+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/data/data_portal.py", line 75, in get_history_window
2019-02-19T18:28:00.858897+00:00 app[worker.1]:     end_dt=end_dt).swaplevel(
2019-02-19T18:28:00.858899+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/data/data_portal.py", line 51, in _get_realtime_bars
2019-02-19T18:28:00.858900+00:00 app[worker.1]:     assets, frequency, bar_count=bar_count)
2019-02-19T18:28:00.858902+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 405, in get_bars
2019-02-19T18:28:00.858904+00:00 app[worker.1]:     symbols, 'day' if is_daily else 'minute', limit=bar_count)
2019-02-19T18:28:00.858905+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 493, in _symbol_bars
2019-02-19T18:28:00.858907+00:00 app[worker.1]:     return parallelize(fetch, workers=25)(symbols)
2019-02-19T18:28:00.858909+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 106, in wrapper
2019-02-19T18:28:00.858910+00:00 app[worker.1]:     task_result = task.result()
2019-02-19T18:28:00.858912+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2019-02-19T18:28:00.858914+00:00 app[worker.1]:     return self.__get_result()
2019-02-19T18:28:00.858916+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2019-02-19T18:28:00.858918+00:00 app[worker.1]:     raise self._exception
2019-02-19T18:28:00.858919+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/concurrent/futures/thread.py", line 56, in run
2019-02-19T18:28:00.858921+00:00 app[worker.1]:     result = self.fn(*self.args, **self.kwargs)
2019-02-19T18:28:00.858922+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 76, in wrapper
2019-02-19T18:28:00.858924+00:00 app[worker.1]:     return func(*args, **kwargs)
2019-02-19T18:28:00.858926+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 476, in fetch
2019-02-19T18:28:00.858927+00:00 app[worker.1]:     size, symbol, _from, to, query_limit).df
2019-02-19T18:28:00.858929+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/alpaca_trade_api/polygon/rest.py", line 75, in historic_agg
2019-02-19T18:28:00.858930+00:00 app[worker.1]:     raw = self.get(path, params)
2019-02-19T18:28:00.858932+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/alpaca_trade_api/polygon/rest.py", line 33, in get
2019-02-19T18:28:00.858933+00:00 app[worker.1]:     return self._request('GET', path, params=params)
2019-02-19T18:28:00.858935+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/alpaca_trade_api/polygon/rest.py", line 29, in _request
2019-02-19T18:28:00.858936+00:00 app[worker.1]:     resp.raise_for_status()
2019-02-19T18:28:00.858938+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
2019-02-19T18:28:00.858940+00:00 app[worker.1]:     raise HTTPError(http_error_msg, response=self)
2019-02-19T18:28:00.858942+00:00 app[worker.1]: requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.polygon.io/v1/historic/agg/day/IWV?limit=180&apiKey={API_KEY_OMITTED}
2019-02-19T18:28:00.859037+00:00 app[worker.1]: [2019-02-19 18:28:00.858903] WARNING: Executor: Continuing execution

Upon hitting the same URL from Polygon's API through a GET request, I get the following 500 error:

Server failure during read query at consistency LOCAL_ONE (1 responses were required but only 0 replicas responded, 2 failed)

But this error only happens on first request if manually requested.

This seems to be a timeout error thrown by Cassandra. As such, I believe a solution may be to increase Cassandra's tombstone failure threshold, or to handle via pylivetrader by catching this exception and trying a second time (seems like it may work after hitting twice in a row, at least this happens in manual requesting).

When outside market minutes, BarData's current(...) fails.

Reproduceable by calling order_target_percent(...) in an algorithm's before_trading_start() function. Not really an expected use case, but current() could be hit from other functions as well.

Stack trace:

Traceback (most recent call last):
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/bin/pylivetrader", line 11, in <module>
    load_entry_point('pylivetrader', 'console_scripts', 'pylivetrader')()
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/tthackston/dev/alpaca/quantopian-fundamentals-examples/venv/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/__main__.py", line 134, in run
    algorithm.run(retry=ctx.retry)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 240, in run
    return self.executor.run(retry=retry)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/executor/executor.py", line 120, in run
    algo.before_trading_start(self.current_data)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 213, in before_trading_start
    self._before_trading_start(self, data)
  File "iex_quote_test.py", line 79, in before_trading_start
    rebalance(context)
  File "iex_quote_test.py", line 103, in rebalance
    order_target_percent(symbol(stock), weight)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/misc/api_context.py", line 62, in wrapped
    return getattr(algorithm, f.__name__)(*args, **kwargs)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 587, in order_target_percent
    amount = self._calculate_order_target_percent_amount(asset, target)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 804, in _calculate_order_target_percent_amount
    target_amount = self._calculate_order_percent_amount(asset, target)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 794, in _calculate_order_percent_amount
    return self._calculate_order_value_amount(asset, value)
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/algorithm.py", line 772, in _calculate_order_value_amount
    self.executor.current_data.current(asset, "price")
  File "/Users/tthackston/dev/alpaca/pylivetrader/pylivetrader/data/bardata.py", line 77, in current
    self.simulation_dt_func(),
AttributeError: 'BarData' object has no attribute 'simulation_dt_func'

data.history throwing error due to improper timestamp to datetime conversion

Every data.history call in my algorithm throws errors due to pylivetrader/backend/alpaca.py attempting to convert timestamps to datetimes for the dataframe index, but the dataframe's index is already in datetime format. This is located

# convert timestamps to datetimes
# astype is necessary to deal with empty result
df.index = pd.to_datetime(
    df.index.astype('int64') * 1000000,
    utc=True,
).tz_convert('America/New_York')
df.index.name = 'timestamp'

Using IEX instead of Alpaca

Has anyone successfully used IEX instead of Alpaca?

I see various references to it in the code and have failed in getting it to run myself.

My goal is to run pylivetrader locally and display trading information via pandas locally, as opposed to going to Alpaca overview.

I'm quite stuck on this, any help would be greatly appreciated.

ModuleNotFoundError: No module named '_Interface'

Running the code for example q01, I get the following error:

C:\Users\owner\Desktop\College_Stuff\Quantopian\pylivetrader-master\examples\q01>pylivetrader run -f ./algo.py Traceback (most recent call last): File "c:\users\owner\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "c:\users\owner\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Scripts\pylivetrader.exe\__main__.py", line 9, in <module> File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\core.py", line 722, in __call__ return self.main(*args, **kwargs) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\core.py", line 697, in main rv = self.invoke(ctx) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\core.py", line 535, in invoke return callback(*args, **kwargs) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\click\decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\pylivetrader\__main__.py", line 141, in run ctx = process_algo_params(ctx, **kwargs) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\pylivetrader\__main__.py", line 115, in process_algo_params algomodule = get_algomodule_by_path(algofile) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\pylivetrader\loader.py", line 50, in get_algomodule_by_path return get_algomodule(file, filename) File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\pylivetrader\loader.py", line 63, in get_algomodule exec_(code, ns) File "algo.py", line 12, in <module> from pipeline_live.data.iex.pricing import USEquityPricing File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\pipeline_live\data\iex\pricing.py", line 1, in <module> from zipline.pipeline.data.dataset import Column, DataSet File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\__init__.py", line 29, in <module> from .utils.run_algo import run_algorithm File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\utils\run_algo.py", line 22, in <module> from zipline.pipeline.data import USEquityPricing File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\pipeline\__init__.py", line 3, in <module> from .classifiers import Classifier, CustomClassifier File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\pipeline\classifiers\__init__.py", line 1, in <module> from .classifier import ( File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\pipeline\classifiers\classifier.py", line 22, in <module> from zipline.pipeline.term import ComputableTerm File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\pipeline\term.py", line 47, in <module> from .domain import Domain, GENERIC, infer_domain File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\zipline\pipeline\domain.py", line 20, in <module> from interface import default, implements, Interface File "C:\Users\owner\AppData\Local\Programs\Python\Python36\Lib\site-packages\interface\__init__.py", line 75, in <module> from _Interface import Interface ModuleNotFoundError: No module named '_Interface'

I'm running on Windows 10, Python 3.6.5. This is my pip freeze:
absl-py==0.2.2 aiodns==1.1.1 aiohttp==3.3.2 alembic==1.0.5 alpaca-trade-api==0.22 arrow==0.12.1 astor==0.6.2 astroid==1.6.5 async-timeout==3.0.0 asyncio-nats-client==0.8.2 atomicwrites==1.1.5 attrs==18.1.0 autopep8==1.3.5 awscli==1.15.59 backcall==0.1.0 backports-datetime-fromisoformat==0.0.1 bcolz==0.12.1 bintrees==2.0.7 bleach==1.5.0 boto==2.49.0 boto3==1.7.58 botocore==1.10.58 Bottleneck==1.2.1 bz2file==0.98 cchardet==2.1.1 ccxt==1.16.11 certifi==2018.4.16 chardet==3.0.4 click==6.7 cma==2.6.0 colorama==0.3.9 contextlib2==0.5.5 cupy==4.3.0+cuda92148 cycler==0.10.0 cyordereddict==1.0.0 Cython==0.29.1 decorator==4.3.0 deuces==0.2 Django==2.1.1 docutils==0.14 ed25519==1.4 empyrical==0.5.0 fastrlock==0.3 Flask==0.12.2 Flask-Cors==3.0.3 future==0.16.0 gast==0.2.0 gdax==1.0.6 gensim==3.5.0 GPy==1.8.4 graphviz==0.8.4 grpcio==1.12.1 gym==0.10.5 h5py==2.8.0 html5lib==0.9999999 ibapi==9.73.7 idna==2.6 idna-ssl==1.0.1 iexfinance==0.3.5 imageio==2.3.0 inflection==0.3.1 Interface==2.11.1 intervaltree==2.1.0 ipython==7.2.0 ipython-genutils==0.2.0 iso3166==0.9 isort==4.3.4 itsdangerous==0.24 jedi==0.13.1 Jinja2==2.10 jmespath==0.9.3 jsonfield==2.0.2 jsonpickle==0.9.6 Keras==2.2.0 Keras-Applications==1.0.2 Keras-Preprocessing==1.0.1 keras-rl==0.4.2 kiwisolver==1.0.1 lazy-object-proxy==1.3.1 Logbook==1.4.1 lru-dict==1.1.6 lxml==4.2.5 Mako==1.0.7 Markdown==2.6.11 MarkupSafe==1.0 matplotlib==2.2.2 mccabe==0.6.1 more-itertools==4.2.0 mss==3.2.1 multidict==4.3.1 multipledispatch==0.6.0 mysql-connector==2.1.6 mysqlclient==1.3.13 networkx==1.11 nltk==3.3 numexpr==2.6.8 numpy==1.14.6 opencv-python==3.4.1.15 pandas==0.20.3 pandas-datareader==0.7.0 paramz==0.9.2 parso==0.3.1 pathlib==1.0.1 patsy==0.5.1 pickleshare==0.7.5 Pillow==5.2.0 pipeline-live==0.1.4 pluggy==0.6.0 pokereval==0.1.2 prompt-toolkit==2.0.7 protobuf==3.6.0 psycopg2==2.7.3 py==1.5.4 pyasn1==0.4.3 pycares==2.3.0 pycodestyle==2.4.0 pyfiglet==0.7.5 pyglet==1.3.2 Pygments==2.3.0 PyJWT==1.6.4 pylint==1.9.2 pylivetrader==0.0.20 pynput==1.4 pyparsing==2.2.0 PyQt5==5.11.2 PyQt5-sip==4.19.12 PySocks==1.6.8 pytest==3.6.3 python-box==3.1.1 python-dateutil==2.7.3 python-editor==1.0.3 python-interface==1.5.0 pytz==2018.7 PyYAML==3.12 Quandl==3.4.5 requests==2.20.1 requests-file==1.4.3 rsa==3.4.2 s3transfer==0.1.13 scikit-learn==0.19.1 scikit-optimize==0.5.2 scipy==1.1.0 seaborn==0.8.1 six==1.10.0 sklearn==0.0 smart-open==1.6.0 sortedcontainers==2.1.0 SQLAlchemy==1.1.12 statsmodels==0.9.0 TA-Lib==0.4.17 tables==3.4.4 tensorboard==1.9.0 tensorflow==1.9.0 tensorflow-tensorboard==1.5.1 termcolor==1.1.0 toolz==0.9.0 tornado==5.1.1 tradersbot==0.2.5 trading-calendars==1.6.0 traitlets==4.3.2 twilio==6.16.4 urllib3==1.22 wcwidth==0.1.7 websocket-client==0.40.0 websockets==7.0 Werkzeug==0.14.1 wrapt==1.10.11 xgboost==0.72 yarl==1.1.0 zipline==1.3.0+217.g0975a2e5 zope.event==4.4 zope.interface==4.6.0 zope.schema==4.9.3

Preserve NaN for less liquid stock prices

In order to comply Quantopian behavior, the data should be padded with NaN and let the algo decide what to do with it.

Q translater

Having the Quantopian migration step in documentation is great, but many of the processes can be automated. There is a partial code that tries to translate the import paths, but this breaks the mapping from the original code when it runs, and the better way to do it is to provide another step (CLI) to explicitly translate the source code.

Order filled value sign to match Quantopian

For Alpaca pylivetrader & Quantopian zipline interoperability,
order.filled should be negative on sell.

This backtest can be used to see the difference and verify the fix.
For a few minutes it does buying and selling, showing the order fills.
Code that can run on both Alpaca and Quantopian to verify

With at least 30k starting capital ...

Run that code as a backtest in the IDE on Quantopian, just a couple of days.
DPZ filled -38/-38 status 1 id e3a1 held 0
Run the code in paper trading on Alpaca.
TSLA filled 31/-31 status 1 id 3011 held 0

Those are sell order examples of current output.
A minus sign means sell or sold.
-38/-38 means 38 shares were sold on the order to sell 38 shares.

TSLA 31/-31 indicates 31 bought on an order to sell 31 shares. That's an error.

Until this is fixed, workaround requires determining whether on Alpaca
and multiplying filled by -1 but just on sell.

401 Client Error: Unauthorized for url, example q01

Attempting to run example q01 for paper trading and encountering error

(myvirtualenv) ➜  src bash ../credentials/config.sh; pylivetrader run -f algo.py -l 'DEBUG' 
[2018-12-18 17:17:17.226250] INFO: Algorithm: livetrader start running with backend = alpaca data-frequency = minute
0
[2018-12-18 17:17:19.468488] INFO: algo: 
        Algorithm initialized variables:
            context.MaxCandidates 100
            LowVar 6
            HighVar 40
        
[2018-12-18 17:17:19.472587] INFO: algo: 
        Algorithm selected stocks NumExprFilter(expr='x_0 <= (100.0)', bindings={'x_0': Rank(NumExprFactor(expr='(x_0 - x_1) / (x_1)', bindings={'x_1': SimpleMovingAverage([USEquityPricing.close], 45), 'x_0': SimpleMovingAverage([USEquityPricing.close], 3)}), method='ordinal', mask=AssetExists())})
        
/home/me/.zipline/data/dailycache/iex_company.pkl: digest mismatch 1ecedfa98cc42ea86945c4ddb7eaaffc != b82e657814f0790f98496814b11fb538, reloading
/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/iexfinance/__init__.py:45: UserWarning: Stock is moved to iexfinance.stocks. This function will in be deprecated in v0.4.0
  warnings.warn(WNG_MSG % ("Stock", "stocks"))
[2018-12-18 17:17:21.607837] DEBUG: pipeline_live.data.sources.util: fetch: 10.18% completed
[2018-12-18 17:17:22.294798] DEBUG: pipeline_live.data.sources.util: fetch: 20.35% completed
[2018-12-18 17:17:22.996512] DEBUG: pipeline_live.data.sources.util: fetch: 30.53% completed
[2018-12-18 17:17:23.692973] DEBUG: pipeline_live.data.sources.util: fetch: 40.70% completed
[2018-12-18 17:17:24.354522] DEBUG: pipeline_live.data.sources.util: fetch: 50.88% completed
[2018-12-18 17:17:25.018852] DEBUG: pipeline_live.data.sources.util: fetch: 61.06% completed
[2018-12-18 17:17:25.693408] DEBUG: pipeline_live.data.sources.util: fetch: 70.10% completed
[2018-12-18 17:17:26.351974] DEBUG: pipeline_live.data.sources.util: fetch: 80.28% completed
[2018-12-18 17:17:26.844290] DEBUG: pipeline_live.data.sources.util: fetch: 90.45% completed
[2018-12-18 17:17:27.475785] DEBUG: pipeline_live.data.sources.util: fetch: 100.00% completed
[2018-12-18 17:17:27.573776] INFO: pipeline_live.data.iex.pricing_loader: chart_range=3m
/home/me/.zipline/data/dailycache/iex_chart_3m: digest mismatch 1ecedfa98cc42ea86945c4ddb7eaaffc != b82e657814f0790f98496814b11fb538, reloading
[2018-12-18 17:17:36.361035] DEBUG: pipeline_live.data.sources.util: fetch: 10.18% completed
[2018-12-18 17:17:40.283283] DEBUG: pipeline_live.data.sources.util: fetch: 20.35% completed
[2018-12-18 17:17:43.810842] DEBUG: pipeline_live.data.sources.util: fetch: 30.53% completed
[2018-12-18 17:17:46.952893] DEBUG: pipeline_live.data.sources.util: fetch: 40.70% completed
[2018-12-18 17:17:49.735045] DEBUG: pipeline_live.data.sources.util: fetch: 50.88% completed
[2018-12-18 17:17:51.555943] DEBUG: pipeline_live.data.sources.util: fetch: 61.06% completed
[2018-12-18 17:17:52.823105] DEBUG: pipeline_live.data.sources.util: fetch: 70.10% completed
[2018-12-18 17:17:54.573683] DEBUG: pipeline_live.data.sources.util: fetch: 80.28% completed
[2018-12-18 17:17:56.343674] DEBUG: pipeline_live.data.sources.util: fetch: 90.45% completed
[2018-12-18 17:17:57.761533] DEBUG: pipeline_live.data.sources.util: fetch: 100.00% completed
[2018-12-18 17:18:01.062911] INFO: pipeline_live.data.iex.pricing_loader: chart_range=3m
/home/me/.zipline/data/dailycache/iex_key_stats.pkl: digest mismatch 1ecedfa98cc42ea86945c4ddb7eaaffc != b82e657814f0790f98496814b11fb538, reloading
[2018-12-18 17:18:09.170879] DEBUG: pipeline_live.data.sources.util: fetch: 10.18% completed
[2018-12-18 17:18:10.233156] DEBUG: pipeline_live.data.sources.util: fetch: 20.35% completed
[2018-12-18 17:18:10.678750] DEBUG: pipeline_live.data.sources.util: fetch: 30.53% completed
[2018-12-18 17:18:11.445735] DEBUG: pipeline_live.data.sources.util: fetch: 40.70% completed
[2018-12-18 17:18:12.105574] DEBUG: pipeline_live.data.sources.util: fetch: 50.88% completed
[2018-12-18 17:18:12.747900] DEBUG: pipeline_live.data.sources.util: fetch: 61.06% completed
[2018-12-18 17:18:13.421420] DEBUG: pipeline_live.data.sources.util: fetch: 70.10% completed
[2018-12-18 17:18:14.081844] DEBUG: pipeline_live.data.sources.util: fetch: 80.28% completed
[2018-12-18 17:18:14.631220] DEBUG: pipeline_live.data.sources.util: fetch: 90.45% completed
[2018-12-18 17:18:15.420115] DEBUG: pipeline_live.data.sources.util: fetch: 100.00% completed
Traceback (most recent call last):
  File "/home/me/python-virtualenvs/myvirtualenv/bin/pylivetrader", line 11, in <module>
    sys.exit(main())
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/__main__.py", line 144, in run
    algorithm.run(retry=ctx.retry)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 242, in run
    return self.executor.run(retry=retry)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/executor/executor.py", line 120, in run
    algo.before_trading_start(self.current_data)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 215, in before_trading_start
    self._before_trading_start(self, data)
  File "algo.py", line 204, in before_trading_start
    
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/misc/api_context.py", line 62, in wrapped
    return getattr(algorithm, f.__name__)(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 1049, in pipeline_output
    output = eng.run_pipeline(self._pipelines[name])
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/engine.py", line 77, in run_pipeline
    initial_workspace,
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/engine.py", line 252, in compute_chunk
    mask,
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/zipline/pipeline/mixins.py", line 214, in _compute
    compute(date, masked_assets, out_row, *inputs, **params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/polygon/filters.py", line 12, in compute
    company = polygon.company()
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/sources/polygon.py", line 17, in company
    return _company(all_symbols)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/sources/util.py", line 68, in wrapper
    body = func(*args, **kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/sources/polygon.py", line 32, in _company
    return parallelize(fetch, workers=25, splitlen=50)(all_symbols)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/sources/util.py", line 30, in wrapper
    task_result = task.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/pipeline_live/data/sources/polygon.py", line 27, in fetch
    response = api.polygon.get('/meta/symbols/company', params=params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/polygon/rest.py", line 33, in get
    return self._request('GET', path, params=params)
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/alpaca_trade_api/polygon/rest.py", line 29, in _request
    resp.raise_for_status()
  File "/home/me/python-virtualenvs/myvirtualenv/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
r    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.polygon.io/v1/meta/symbols/company?symbols=BMRA%2CBWZ%2CMSBF%2CECT%2CARKG%2CMIDU%2CESGU%2CMYOS%2CCTHR%2CJLS%2CNBO%2CSOIL%2CMAV%2CPPT%2CUIVM%2CFORK%2CNMM%2CCDXC%2CSYPR%2CNVFY%2CFEUZ%2CFMB%2CNVUS%2CPME%2CJHMI%2CIGN%2CWMCR%2CLOAC%2CTHD%2CKDMN%2CWWR%2CBTO%2CVIIX%2CPFI%2CDSE%2CTPYP%2CHYXU%2CSRF%2CFLJH%2CDVYA%2CCFA%2CDWT%2CFEMB%2CDTEA%2CCHAD%2CFAMI%2CEEB%2CNMS%2CDHCP%2CHNP&apiKey=AKTV6DOG63CG9DWK91G1

environment variables look like

(myvirtualenv) ➜  src env
...
APCA_API_KEY_ID=AKTV6DOG63CG9DWK91G1
APCA_API_SECRET_KEY=xxxxxx/xxxxxx/xxxxxx
APCA_API_BASE_URL=https://paper-api.alpaca.markets

Note, have note changed the code for example q01 and am simply trying to test it as is on paper trading when this error is being thrown.

get_open_orders() issue

get_open_orders() doesn't seem to find pending sell orders when pending sell orders are pushed off of the order history list on the alpaca account dashboard. Not sure if this happens with buy orders also but it's possible. Orders get pushed off the list when there are more than the maximum displayed pending or canceled orders in the list as illustrated in the image..

.

ImportError: cannot import name 'attach_pipeline'

The import I'm using:

from pylivetrader.algorithm import attach_pipeline, pipeline_output

Even when I install pipeline-live in my Dockerfile, it seems to throw this error:

/usr/local/lib/python3.6/site-packages/IPython/frontend.py:21: ShimWarning: The top-level `frontend` package has been deprecated since IPython 1.0. All its subpackages have been moved to the top `IPython` level.
  "All its subpackages have been moved to the top `IPython` level.", ShimWarning)
Traceback (most recent call last):
  File "/usr/local/bin/pylivetrader", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/__main__.py", line 131, in run
    ctx = process_algo_params(ctx, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/__main__.py", line 106, in process_algo_params
    algomodule = get_algomodule_by_path(algofile)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/loader.py", line 50, in get_algomodule_by_path
    return get_algomodule(file, filename)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/loader.py", line 63, in get_algomodule
    exec_(code, ns)
  File "algo.py", line 40, in <module>
    from pylivetrader.algorithm import attach_pipeline, pipeline_output
ImportError: cannot import name 'attach_pipeline'

(Running in Docker: alpacamarkets/pylivetrader:0.0.13)

Here's my Dockerfile:

FROM alpacamarkets/pylivetrader:0.0.13

RUN mkdir /app
COPY . /app
WORKDIR /app

RUN pip install pipeline-live

CMD pylivetrader run -f algo.py

Cannot find reference '' in api.py

The code from pylivetrader.api import ( order_percent, get_datetime, get_open_orders, cancel_order, schedule_function, order_target_percent, date_rules, time_rules, symbol) does not work as half of the imports are not found. Any suggestions as I see most people's code using this library still uses this format.

Provide reference configuration using Docker and tutorial for it

Despite of recent improvements around pipenv and virtualenv, it is still challenging to many algo traders to set up the environment correctly. While docker is a techie solution to it, if we document how to do it in each scenario, this should become much easier for non developers.

context namespace

I ran into an issue with trying to set context.symbol = 'TIK', due to how this is handled in api_context.api_method() this isn't supported, and resulted in a problem that was hard to trace down.

Setting context.symbol obscured the symbol() lookup due to the way the namespaces are munged in api_method local redirection. e.g. it resulted in errors attempting to call a string as a function.

The context.symbol ended up being stored in the pkl file and trying to troubleshoot the issue by changing my code didn't work as the errant context was reloaded.

Docker permission denied

Followed accepted answer at https://askubuntu.com/questions/938700/how-do-i-install-docker-on-ubuntu-16-04-lts to get set up.

I'm not familiar with Docker, but am trying to use it as I was having library trouble.

When I run docker run -v $PWD:/work -w /work alpacamarkets/pylivetrader pylivetrader run -f algo.py, I get

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.

I also get this error when I run docker pull alpacamarkets/pylivetrader.

Pickling instances of user-defined classes with StateStore fails

I have a class called JPosition defined in my algo.py file so I can more comfortably track certain things about the positions I take.

class JPostition:
    def __init__(self,
                 asset,
                 entry_price=None,
                 averaged_down=False,
                 complete=False):
        self.asset = asset
        self.entry_price = entry_price 
        self.averaged_down = averaged_down
        self.complete = complete

    def percent_change(self, current_price):
        return (current_price - self.entry_price) / current_price

I store instances of JPosition in context within a dictionary where Assets are keys.

def before_trading_start(context, data):
    context.j_positions = {}

    tickers = ["SNAP", "MU", "AMD"]
    for ticker in tickers:
        s = symbol(ticker)
        context.j_positions[s] = JPostition(s)

Every time pylivetrader attempts to pickle context via its state management functionality, an exception is raised.

Traceback (most recent call last):
  File "/usr/local/bin/pylivetrader", line 10, in <module>                                                                
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__                                      
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main                                          
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke                                       
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                               
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke                                        
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke                                        
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func                                 
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/__main__.py", line 153, in run                                
    algorithm.run(retry=ctx.retry)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/algorithm.py", line 252, in run                               
    return self.executor.run(retry=retry)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/executor/executor.py", line 120, in run                       
    algo.before_trading_start(self.current_data)
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/algorithm.py", line 227, in before_trading_start              
    self, self._algoname, self._context_persistence_excludes)                                                             
  File "/usr/local/lib/python3.6/site-packages/pylivetrader/statestore/__init__.py", line 42, in save                     
    pickle.dump(state, f)
_pickle.PicklingError: Can't pickle <class 'JPostition'>: attribute lookup JPostition on builtins failed

Here's what state is just before the exception is raised.

{
    'j_positions': {
        Asset(83e52ac1-bb18-4e9f-b68d-dda5a8af3ec0, symbol=SNAP, asset_name=SNAP, exchange=NYSE): <JPostition object at 0x7f0f111b7fd0>,
        Asset(1ef18995-5094-4f62-a5c5-e00168611152, symbol=MU, asset_name=MU, exchange=NASDAQ): <JPostition object at 0x7f0f14e95a90>,
        Asset(03fb07bb-5db1-4077-8dea-5d711b272625, symbol=AMD, asset_name=AMD, exchange=NASDAQ): <JPostition objectat 0x7f0f111b7f28>
    },
    'datetime': Timestamp('2019-02-07 14:06:00+0000', tz='UTC'),
    '__state_checksum': 'algo'
}

I'd really like to continue using the built-in state management, but couldn't figure out why this was happening. I just added j_positions to the list of exclusions in the meantime.

def initialize(context):
    log.info("Initializing")
    # Pickling instances of user-defined classes fails.
    context._context_persistence_excludes.append("j_positions")

Improve querying of recent orders by adding local lookup by ID

v0.0.18 changed order querying to go to the server, but as @garyha pointed out in issue #40, some users of Quantopian will prefer to be able to configure how many days of orders they want to store locally and look them up that way for a performance boost.

Support python 3.6 and 3.7

The newer versions are well supported today, but we are not running tests in CI.

get_open_orders(asset) isn't Quantopian compatible

With get_open_orders for a specific stock, Q returns the list of orders while AP is returning the same dictionary as if the security id input had not been supplied.

      for s in sids:               # s is the security id
        oos = get_open_orders(s)   # oos stands for open orders
        for o in oos:

            print('---------------')
            print(s)

                # Equity(bd952854-09b0-4ef9-b87e-6aefe4d642b7 [UQM])

            print(o)

                # Equity(bd952854-09b0-4ef9-b87e-6aefe4d642b7 [UQM])

               #  [expected to be looping thru a list of orders here, instead am looping thru the keys of a dictionary]

            print(get_open_orders())

                # {Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AMEX): [Event({'id': '8fc15d9587c84556a8329851b00ff3f7', 'dt': Timestamp('2018-10-2
                # 3 17:20:18.154396+0000', tz='UTC'), 'reason': None, 'created': Timestamp('2018-10-23 17:20:18.154396+0000', tz='UTC'), 'amount': -1, 'filled': 0, 'commission': 0, 's
                # top': 1.24, 'limit': None, 'stop_reached': False, 'limit_reached': False, 'sid': Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AME
                # X), 'status': <ORDER_STATUS.OPEN: 0>})]}

            print('---------------')

Above, raw from the shell:

Equity(bd952854-09b0-4ef9-b87e-6aefe4d642b7 [UQM])
Equity(bd952854-09b0-4ef9-b87e-6aefe4d642b7 [UQM])
{Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AMEX): [Event({'id': '8fc15d9587c84556a8329851b00ff3f7', 'dt': Timestamp('2018-10-2
3 17:20:18.154396+0000', tz='UTC'), 'reason': None, 'created': Timestamp('2018-10-23 17:20:18.154396+0000', tz='UTC'), 'amount': -1, 'filled': 0, 'commission': 0, 's
top': 1.24, 'limit': None, 'stop_reached': False, 'limit_reached': False, 'sid': Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AME
X), 'status': <ORDER_STATUS.OPEN: 0>})]}

Here's some test code instead:

def trade(context, data):
    for s in context.out:
        order(s, 1)
        print(  get_open_orders(s)  )
        print(  get_open_orders()  )
        break
    assert(0)

Q

get_open_orders(s)    a list
2017-10-18 06:31  PRINT [Event({'status': 0, 'created': Timestamp('2017-10-18 13:31:00+0000', tz='UTC'), 'limit_reached': False, 'stop': None, 'reason': None, 'stop_reached': False, 'commission': 0, 'amount': 1, 'limit': None, 'sid': Equity(117 [AEY]), 'dt': Timestamp('2017-10-18 13:31:00+0000', tz='UTC'), 'id': '9db921cae8384c60b1ee59d916c554b6', 'filled': 0})]
get_open_orders()     a dictionary
2017-10-18 06:31  PRINT {Equity(117 [AEY]): [Event({'status': 0, 'created': Timestamp('2017-10-18 13:31:00+0000', tz='UTC'), 'limit_reached': False, 'stop': None, 'reason': None, 'stop_reached': False, 'commission': 0, 'amount': 1, 'limit': None, 'sid': Equity(117 [AEY]), 'dt': Timestamp('2017-10-18 13:31:00+0000', tz='UTC'), 'id': '9db921cae8384c60b1ee59d916c554b6', 'filled': 0})]}

AP

get_open_orders(s)     a dictionary
{Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AMEX): [Event({'id': '8fc15d9587c84556a8329851b00ff3f7', 'dt': Timestamp('2018-10-2
3 17:20:18.154396+0000', tz='UTC'), 'reason': None, 'created': Timestamp('2018-10-23 17:20:18.154396+0000', tz='UTC'), 'amount': -1, 'filled': 0, 'commission': 0, 's
top': 1.24, 'limit': None, 'stop_reached': False, 'limit_reached': False, 'sid': Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AME
X), 'status': <ORDER_STATUS.OPEN: 0>})]}
get_open_orders()     a dictionary
{Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AMEX): [Event({'id': '8fc15d9587c84556a8329851b00ff3f7', 'dt': Timestamp('2018-10-2
3 17:20:18.154396+0000', tz='UTC'), 'reason': None, 'created': Timestamp('2018-10-23 17:20:18.154396+0000', tz='UTC'), 'amount': -1, 'filled': 0, 'commission': 0, 's
top': 1.24, 'limit': None, 'stop_reached': False, 'limit_reached': False, 'sid': Asset(bd952854-09b0-4ef9-b87e-6aefe4d642b7, symbol=UQM, asset_name=UQM, exchange=AME
X), 'status': <ORDER_STATUS.OPEN: 0>})]}

Conda and/or windows bdist package

There are quite a bit of users on Windows and one of the hardest thing is the setup, especially with pipeline-live which has deep dependency with binary build requirements. As noted in https://docs.alpaca.markets/libraries/pylivetrader-windows/ lru-dict requires Visual Studio, which is not very easy to handle.

I'd like to make it as easy as pip install or conda install. This may need some research on that. bdist_wheel may be enough if it's built on windows?

Alpaca/polygon client doesn't obey broker config

Setting keys in env vars works, but config values are ignored.

ValueError: Key ID must be given to access Alpaca trade API

Backend for Interactive Brokers

Backend class is pluggable extension and can hook into another broker. Interactive Brokers is one of the most popular

ImportError: cannot import name 'Timedelta'

I've been trying to figure this out.

whether I run pylivetrader through Docker, install it via pip, use Conda or Jupyter notebooks I am dealt this error:

from pandas._libs.tslibs import Timedelta, conversion, is_null_datetimelike ImportError: cannot import name 'Timedelta'

I've followed every solution I can find online and nothing seems to work.

Any help will be greatly appreciated.

pylivetrader: shorting a stock fails and v2 is not enabled

Attempted to post a message on slack couple or more times. So creating an issue for this as I did not receive a response.

What does it entail to enable shorting and v2 api in pylivetrader? I already have new account setup ( not legacy one with 3AP) and live account linked with funds and active.

GET account calls shows shorting is enabled on my a/c when I do plain rest call. I know there were recent changes to make v2 as default in pylivetrader but when I am attempting to do shorting, I get below error. wondering how to debug from pylivetrader perspective
order(sid,-10)
order for symbol W is rejected insufficient qty (0 < 10)
additional info: tried with and without APCA_API_VERSION=v2 still no luck. Additionally, not sure if this is related. my algo stops working after hours may be have to do with v2 not being enabled???

running algo inside latest docker image

AttributeError: 'DataFrame' object has no attribute 'current_ratio'

Link to code: https://github.com/alpacahq/pylivetrader/blob/master/examples/graham-fundamentals/GrahamFundamentals.py
When I run the code, I get this error. I have not modified the code in any way, so I do not know why this problem occurs.
AttributeError: 'DataFrame' object has no attribute 'current_ratio'

ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

When attempting to run the q01 example the following error occurs:

ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

Traceback (most recent call last):
File "/home/[REMOVED]/.local/share/virtualenvs/q01-9tU7-qwB/bin/pylivetrader", line 7, in
from pylivetrader.main import main
File "/home/[REMOVED]/.local/share/virtualenvs/q01-9tU7-qwB/lib/python3.6/site-packages/pylivetrader/init.py", line 16, in
from pylivetrader import api # noqa
File "/home/[REMOVED]/.local/share/virtualenvs/q01-9tU7-qwB/lib/python3.6/site-packages/pylivetrader/api.py", line 24, in
from pylivetrader.finance import execution, commission, slippage, cancel_policy
File "/home/[REMOVED]/.local/share/virtualenvs/q01-9tU7-qwB/lib/python3.6/site-packages/pylivetrader/finance/execution.py", line 22, in
import pylivetrader.misc.math_utils as zp_math
File "/home/[REMOVED]/.local/share/virtualenvs/q01-9tU7-qwB/lib/python3.6/site-packages/pylivetrader/misc/math_utils.py", line 53, in
nanmean = bn.nanmean
AttributeError: module 'bottleneck' has no attribute 'nanmean'

This is in a fresh pipenv after running pipenv install inside the example directory.

This could be due to the version of numpy (1.14.6) required not containing this module. However, official documentation on what _multiarray_umath actually is or when it was added is very sparse.

Testing numpy versions myself, I can import this module when using numpy 1.16.0rc2. Which dependency requires this version of numpy?

Has anyone run into this and resolved it?

Log order submissions and improve error logging

WARNING: Alpaca: order is rejected insufficient qty (0 < 24) does not give much clue without symbol name.

More broadly, it is better to log any order submissions (with optionally opt out) to help understand the issue.

pylivetrader: command not found

I'm using Ubuntu 18.04 on Google Cloud Platform. The commands I use, in order, are:

sudo apt update
sudo apt -y upgrade
sudo apt install -y python3-pip
python3 -m pip install pylivetrader
python3 -m pip install pipeline-live
pylivetrader run -f algo.py --backend-config config.yaml

On the last command, I get pylivetrader: command not found

Any ideas?

fyi - Supported platforms in readme should specify python3

I'm dinking around with swift + python and it only works with python2.

otherwise - I hit this error.
https://github.com/nats-io/asyncio-nats

Should be compatible with at least Python +3.5.1 using gnatsd as the server.

(gymai) ➜ ~ pip install asyncio-nats-client
Collecting asyncio-nats-client
Using cached https://files.pythonhosted.org/packages/f6/a0/3e9a55cfe262699a2ce98714e14a7381bc674112f567af80457d16ea9b2f/asyncio-nats-client-0.8.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/_g/9f_q1hw907s4pdbctf_lj9n19x5s_w/T/pip-install-m4AmzP/asyncio-nats-client/setup.py", line 2, in
from nats.aio.client import version
File "nats/init.py", line 22
yield from nc.connect(**options)
^
SyntaxError: invalid syntax

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/_g/9f_q1hw907s4pdbctf_lj9n19x5s_w/T/pip-install-m4AmzP/asyncio-nats-client/

Unit test framework

Each algo code should be easily unit tested. The concept of unit test is not very known to the type of users here. Rather than asking them to build their own unit test, we should provide some high level template that does:

provide local fake backend
provide fast replay executor
use real (past) market data
with a pre-defined small set of symbols, simulate a (constant) day using the historical data quickly, basically smoke test to catch any issues like typo.

Add logger level configuration and change default

Currently, log level defaults to DEBUG. We should change this to INFO, and we should also allow people to specify their preferred level via console command.

q01 code deployed to heroku works with does not complete buy/sell live trade

Logging timestamp timezone

Should be able to configure and/or based on the system timezone.

Retry in the main loop

Feature request to implement retry in AlgorithmExecutor main loop.

This is supposed to be more like a cron scheduler, and it should not stop with one failure of the scheduled functions.

On top of that, it would be nice to dump local variables and other useful information in the except block, since debugging in live trading is often stressful missing the enough debug information.

finder.py not finding sids

I'm trying to run some different, and notably more complex, example algorithms from Quantopian.

When I do, I always receive this same error:

Traceback (most recent call last):
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/assets/finder.py", line 102, in retrieve_asset
    asset = self._asset_cache[sid]
KeyError: 19920

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/XX-XXX/Desktop/venv/bin/pylivetrader", line 10, in <module>
    sys.exit(main())
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/__main__.py", line 161, in run
    algorithm.run(retry=ctx.retry)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 264, in run
    self.initialize()
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 224, in initialize
    self._initialize(self, *args, **kwargs)
  File "algo1.py", line 14, in initialize
    context.stocks = [sid(19920),sid(32265)] # QQQ & PSQ
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/misc/api_context.py", line 62, in wrapped
    return getattr(algorithm, f.__name__)(*args, **kwargs)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/algorithm.py", line 473, in sid
    return self.asset_finder.retrieve_asset(sid)
  File "/home/XX-XXX/Desktop/venv/lib/python3.5/site-packages/pylivetrader/assets/finder.py", line 107, in retrieve_asset
    raise SidsNotFound(sids=[sid])
pylivetrader.errors.SidsNotFound: No asset found for sid: 19920.

I convert the algorithm files using !2to3 -w X.py with no issues.

Why is it that finder.py is not finding the sids which emerge from these example algorithms?

The source for the example algorithm is here.

Many thanks!

Address warning from IPython

frontend.py:21: ShimWarning: The top-level `frontend` package has been deprecated since IPython 1.0. All its subpackages have b
een moved to the top `IPython` level.
  "All its subpackages have been moved to the top `IPython` level.", ShimWarning)

I believe we just need to delete .frontend in the import

"Error " Sector name: Basic Materials

I have this error:
raise ValueError("Invalid sector name: {}".format(sector))
ValueError: Invalid sector name: Basic Materials
[2019-07-08 17:30:05.511926] WARNING: Executor: Continuing execution

data.current efficiency

data.current API takes multiple assets as a parameter. However, it is split into each asset and calls backend's get_spot_value for each asset, even though the backend can query the current value a lot more efficiently if list of assets are given at a time. This is a heritage from zipline and should be addressed.

Handle Polygon 404s for portfolio items more gracefully

Per antonovo on slack:

If the portfolio has an item that polygon returns 404 for, then context._backend.portfolio property is going to break
context.portfolio.cash File "lib/python3.6/site-packages/pylivetrader/algorithm.py", line 475, in portfolio self._portfolio = self._backend.portfolio File "lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 192, in portfolio z_portfolio.positions = self.positions File "lib/python3.6/site-packages/pylivetrader/backend/alpaca.py", line 180, in positions price = trade.price AttributeError: 'NoneType' object has no attribute 'price'
And this seems to be caused by this: WARNING: Alpaca: 404 Client Error: Not Found for url: https://api.polygon.io/v1/last/stocks/SYRS?apiKey=xxxx

We should add some handling for this, whether that means displaying a 0 value for the not found asset or something else.

alpacahq / pylivetrader Goto Github PK

pylivetrader's People

Stargazers

Watchers

Forkers

pylivetrader's Issues

Last summaries when stopped (see Definitions)

Percent of inconsistencies (any type)

Worst prevalence: hst_mismatch_prc problem isolated

Second-worst problem: hst_old problem isolated

hst_mismatch_prc example instance

hst_mismatch_cls example instance

hst_old example instance

hst_prc_nan example instance

prc_integer example instance