beancount/beanprice

Daily price quotes fetching library for plain-text accounting

beanprice: Price quotes fetcher for Beancount

Description

A script to fetch market data prices from various sources on the internet and render them for plain text accounting price syntax (and Beancount).

This used to be located within Beancount itself (at v2) under beancount.prices. This repo will contain all future updates to that script and to those price sources.

Documentation

Some documentation is still part of Beancount. More about how to use this can be found on that mailing-list. Otherwise read the source.

Quick start

To install beanprice, run:

pip install git+https://github.com/beancount/beanprice.git

You can fetch the latest price of a stock by running:

bean-price -e 'USD:yahoo/AAPL'

To fetch the latest prices from your beancount file, first ensure that commodities have price metadata, e.g.

2000-01-01 commodity AAPL
  price: "USD:yahoo/AAPL"

Then run:

bean-price ledger.beancount

To update prices up to the present day, run:

bean-price --update ledger.beancount

For more detailed guide for price fetching, read https://beancount.github.io/docs/fetching_prices_in_beancount.html.

Price source info

The following price sources are available:

Name	Module	Provides prices for	Base currency	Latest price?	Historical price?
Alphavantage	beanprice.alphavantage	Stocks, FX, Crypto	Many currencies	✓	✕
Coinbase	beanprice.coinbase	Most common (crypto)currencies	Many currencies	✓	✓
Coincap	beanprice.coincap	Most common (crypto)currencies	USD	✓	✓
Coinmarketcap	beanprice.coinmarketcap	Most common (crypto)currencies	Many Currencies	✓	✕
IEX	beanprice.iex	Trading symbols	USD	✓	🚧 (Not yet!)
OANDA	beanprice.oanda	Many currencies	Many currencies	✓	✓
Quandl	beanprice.quandl	Various datasets	Various datasets	✓	✓
Rates API	beanprice.ratesapi	Many currencies	Many currencies	✓	✓
Thrift Savings Plan	beanprice.tsp	TSP Funds	USD	✓	✓
Yahoo	beanprice.yahoo	Many currencies	Many currencies	✓	✓
EastMoneyFund(天天基金)	beanprice.eastmoneyfund	Chinese Funds	CNY	✓	✓

Testing

Run tests:

pytest beanprice

Lint:

pylint beanprice

Type checker:

mypy beanprice --ignore-missing-imports

Copyright and License

Copyright (C) 2007-2020 Martin Blais. All Rights Reserved.

This code is distributed under the terms of the "GNU GPLv2 only". See COPYING file for details.

beanprice's Issues

Bug with fetching using Yahoo price source

Original report by Martin Blais (Bitbucket: blais, GitHub: blais).

---

On Thu, Feb 18, 2016 at 6:01 AM, Daniël Bos [email protected] wrote:
I'm currently trying to set up price fetching for the different
currencies I'm using. It appears that the Yahoo source currently
doesn't work for currencies.

When I request the latest price, I get the following error:

Could not fetch for job: DatedPrice(base='CNYUSD', quote='USD',
date=None, sources=[PriceSource(module=<module
'beancount.prices.sources.yahoo' from
'XXX/beancount/src/python/beancount/prices/sources/yahoo.py'>,
symbol='CNYUSD', invert=False)])

You have to find the symbol in Yahoo's symbology that will work.
Not sure which it is.

This works:

$ bean-price -e 'USD:google/CNYUSD'
2016-02-18 price CNYUSD 0.15340953 USD

When I request historical prices, I get the following error:

This looks like a bug that needs looking into.

DEBUG : Reading
http://ichart.yahoo.com/table.csv?a=0&b=27&c=2016&d=1&e=1&f=2016&g=d&ignore=.csv&s=CNYUSD
Traceback (most recent call last):
File "XXX/beancount/bin/bean-price", line 2, in
from beancount.prices.price import main; main()
File "XXX/beancount/src/python/beancount/prices/price.py", line 322, in main
functools.partial(fetch_price, swap_inverted=args.swap_inverted), jobs)))
File "/usr/lib/python3.4/concurrent/futures/_base.py", line 549, in
result_iterator
yield future.result()
File "/usr/lib/python3.4/concurrent/futures/_base.py", line 402, in result
return self.__get_result()
File "/usr/lib/python3.4/concurrent/futures/_base.py", line 354, in
__get_result
raise self._exception
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "XXX/beancount/src/python/beancount/prices/price.py", line 131,
in fetch_price
srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
File "XXX/beancount/src/python/beancount/prices/price.py", line 86,
in fetch_cached_price
result = source.get_historical_price(symbol, date)
File "XXX/beancount/src/python/beancount/prices/sources/yahoo.py",
line 203, in get_historical_price
data = response.read().decode('utf-8').strip()
AttributeError: 'NoneType' object has no attribute 'read'

--

Can't fetch from Google

Running the command bean-price -e USD:google/NASDAQ:FB outputs the value below:

$ bean-price -e USD:google/NASDAQ:FB
ERROR   : Could not fetch for job: DatedPrice(base='NASDAQ:FB', quote='USD', date=None, sources=[PriceSource(module=<module 'google' (namespace)>, symbol='NASDAQ:FB', invert=False)])

price from postings

Is it possible to have bean-price fetch undeclared commodity prices from postings, for example, having this posting below:

2021-10-28 * "APPLE.COM/AU              SYDNEY  " ^airpods-3
  Liabilities:CC  -279 AUD
  Assets:AAP3    1 APPLE_AIR_PODS3 @ 279.00 AUD

and with bean-price to "fetch" from the .bean file and generate the price entry for APPLE_AIR_PODS3?

Setup type checker in CI

Track fixing existing issues and setting up type checker in CI.

I will use pytype as suggested in https://beancount.github.io/docs/beancount_v3_dependencies.html

Add option to set default price source for bean-price

Original report by Martin Blais (Bitbucket: blais, GitHub: blais).

---

• Define the new options in options.py
• Pass options_map to get_price_jobs_at_date
• Fetch the option in there and use it instead of default_source
• Remove default_source

This matters now because Yahoo / Google are broken.

Create a new tag/version

Do you plan to create a new tag/version? The last one, 1.2.0, is almost three years old.

Yahoo source in v2 fails when Yahoo! cannot quote

Both BTE and SPAQ tickers returned no results (their Web site, however, does show prices).

I added the asserts below. The error checking is not sufficient, it appears. Sometimes result is flat out empty.

Traceback (most recent call last):
  File "/usr/bin/bean-price", line 11, in <module>
    load_entry_point('beancount==2.3.3', 'console_scripts', 'bean-price')()
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 650, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 439, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 347, in fetch_cached_price
    result = (source.get_latest_price(symbol)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/sources/yahoo.py", line 90, in get_latest_price
    result = parse_response(response)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/sources/yahoo.py", line 51, in parse_response
    assert 0, content
AssertionError: {'result': [], 'error': None}

Traceback (most recent call last):
  File "/usr/bin/bean-price", line 11, in <module>
    load_entry_point('beancount==2.3.3', 'console_scripts', 'bean-price')()
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 650, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 439, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/price.py", line 347, in fetch_cached_price
    result = (source.get_latest_price(symbol)
  File "/usr/lib64/python3.8/site-packages/beancount/prices/sources/yahoo.py", line 90, in get_latest_price
    assert 0, (response, ticker)
AssertionError: (<Response [200]>, 'BTE')

Iex price source not working any more ?

Original report by Timothée GROS (Bitbucket: grostim, GitHub: grostim).

---

I am getting “Forbidden” error for all stocks.

‌

bean-price -e 'USD:iex/MSFT'
ERROR : Error fetching MSFT: Invalid response (403): Forbidden
ERROR : Could not fetch for job: DatedPrice(base='MSFT', quote='USD', date=None, sources=[PriceSource(module=<module 'beancount.prices.sources.iex' from '/usr/local/lib/python3.7/site-packages/beancount/prices/sources/iex.py'>, symbol='MSFT', invert=False)])

Looks like the way to stoks data has changed.

Regular expression in parse_single_source is too restrictive

Original report by Stephan Müller (Bitbucket: pg3-stephan).

---

The regular expression in parse_single_source here uses match = re.match(r'([a-zA-Z]+[a-zA-Z0-9\.]+)/(\^?)([a-zA-Z0-9:=_\-\.]+)$', source)

This is very restrictive. For example. it excludes the common case of module names with underscores (PEP8). A simple fix would be to include underscores: match = re.match(r'([a-zA-Z_]+[a-zA-Z0-9_\.]+)/(\^?)([a-zA-Z0-9:=_\-\.]+)$', source)

Another approach would be to allow any module name and just try the import. Untested code below:

#!python

def parse_single_source(source):
    """Parse a single source string.

    Source specifications follow the syntax:

      <module>/[^]<ticker>

    The <module> is resolved against the Python path, but first looked up
    under the package where the default price extractors lie.

    Args:
      source: A single source string specification.
    Returns:
      A PriceSource tuple, or
    Raises:
      ValueError: If invalid.
    """
    match = re.match(r'([^/]+)/(\^?)([a-zA-Z0-9:=_\-\.]+)$', source)
    if not match:
        raise ValueError('Invalid source name: "{}"'.format(source))
    short_module_name, invert, symbol = match.groups()
    module = import_source(short_module_name)
    return PriceSource(module, symbol, bool(invert))

CoinCap test is broken

https://github.com/beancount/beanprice/blob/master/beanprice/sources/coincap_test.py#L119

Support non-unit currency price

For some of the commodities, quotes may not be on 1 unit of a currency. For example, this ETF on Yahoo is quoted in 0.01 GBP. If I put price: "GBP:yahoo/XSDR.L" in beancount file, the price fetched would be 100x larger than the actual price.

I've also considered to introduce a separate unit for it, e.g. CGBP, and assume it is 0.01 GBP, but the problem then becomes that there is no easy way to fetch the exchange rate for this artificial currency.

It would probably be better to extend the syntax and support non-unit currency price, for example, price: "0.01GBP:yahoo/XSDR.L", and convert the price accordingly.

bean-price --update has no tests

Needs unit testing coverage.
get_price_jobs_up_to_date() is currently untested.

Intermittent issue with Yahoo datasource

When trying to update from Yahoo I get intermittent errors

Traceback (most recent call last):
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: '236efef3aaeade6596ac1f98e86215e1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 508, in fetch_cached_price
    timestamp_created, result_naive = _CACHE[key]
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\dbm\dumb.py", line 147, in __getitem__
    pos, siz = self._index[key]     # may raise KeyError
KeyError: b'236efef3aaeade6596ac1f98e86215e1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\robwa\AppData\Local\Programs\Python\Python39\Scripts\bean-price.exe\__main__.py", line 7, in <module>
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 854, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 592, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 526, in fetch_cached_price
    source.get_historical_price(symbol, time))
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\sources\yahoo.py", line 146, in get_historical_price
    series, currency = get_price_series(ticker, time - timedelta(days=5), time)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\sources\yahoo.py", line 102, in get_price_series
    series = [(datetime.fromtimestamp(timestamp, tz=tzone), Decimal(price))
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\sources\yahoo.py", line 102, in <listcomp>
    series = [(datetime.fromtimestamp(timestamp, tz=tzone), Decimal(price))
TypeError: conversion from NoneType to Decimal is not supported

However, this appears to corrupt the cache, since subsequent attempts to update return

Traceback (most recent call last):
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: 'e9bbf1545efc11c7ce26136f938dff22'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\robwa\AppData\Local\Programs\Python\Python39\Scripts\bean-price.exe\__main__.py", line 7, in <module>
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 854, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 608, in result_iterator
    yield fs.pop().result()
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 438, in result
    return self.__get_result()
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 390, in __get_result
    raise self._exception
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 592, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\site-packages\beanprice\price.py", line 508, in fetch_cached_price
    timestamp_created, result_naive = _CACHE[key]
  File "c:\users\robwa\appdata\local\programs\python\python39\lib\shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
EOFError: Ran out of input

The only way forward is to delete the cache.

Is it possible to treat this as a soft error for that data point and simply skip it?

IEX source no longer works

IEX API 1.0 (https://api.iextrading.com/1.0/...)

Now returns 403.

bean-price -e 'USD:iex/AAPL'

ERROR   : Error fetching AAPL: Invalid response (403): <html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

ERROR   : Could not fetch for job: DatedPrice(base='AAPL', quote='USD', date=None, sources=[PriceSource(module=<module 'beanprice.sources.iex' from '/home/chris/.local/pipx/venvs/beanprice/lib/python3.10/site-packages/beanprice/sources/iex.py'>, symbol='AAPL', invert=False)])

The old API seems to have been shut down. The IEX-API repo in IEX's Github account is gone.

IEX Historical Data from IEX Exchange is now only in the format of 3-6 GB/symbol/day pcap data dumps.

IEX Cloud

Now there's IEX Cloud for API access to financial data.

A free account gets "50k credits on a limited, but free plan", which would allow about 5.5k API requests, and is estimated to be enough for "Fifty days of open, high, low, and close (OHLC) for the top 500 U.S. stocks".

Paid plans starts at $19/month or $108/year, for personal use.

The API docs look good. Relevant endpoints include:

• Time Series Endpoint
• Stocks / Equities > Historical Prices
• Forex / Currencies > Historical Daily
• Commodities

ratesapi fails as host is no longer available

Currently the ratesapi fails with connection errors as the host is no longer available.

Apply same fixed to Yahoo as in Beancount

beancount/beancount#668
beancount/beancount#669

Make iter_dates include end date.

Original report by Vivek G (Bitbucket: seltzered, GitHub: seltzered).

---

This is partly a question about conventions in beancount, but I'm thinking that the iter_dates function (in utils/dateutils.py) should include the end_date rather than omitting it. See suggested commits in https://bitbucket.org/seltzered/beancount/branch/iter_end_dates .

(reason I'm thinking about this is I'm trying to add a --date-range option into bean-price).

Cache causes error: terminated by signal SIGSEGV (Address boundary error)

I ran bean-price /Users/williamdavies/my-beancount/my-ledger-test.beancount --update --inactive --date 2019-01-01 -vv > prices.beancount

Output:

INFO    : Using price cache at "/var/folders/gl/hvd855yx0v73w7z0_wv_sx900000gp/T/bean-price.cache" (with indefinite expiration)
INFO    : Processing at date: 2019-01-01
INFO    : Loading "/Users/williamdavies/my-beancount/my-ledger-test.beancount"

<some failed balance assertions>

DEBUG   : Ignoring currency with no metadata: USD
DEBUG   : Ignoring currency with no metadata: JPY
DEBUG   : -------- Currencies in primary list:
DEBUG   :                           GBP /JPY
INFO    : Fetching: GBPJPY=X (time: 2018-01-01 16:00:00+00:00)
INFO    : Fetching: GBPJPY=X (time: 2018-01-02 16:00:00+00:00)
DEBUG   : Starting new HTTPS connection (1): query1.finance.yahoo.com:443
DEBUG   : Starting new HTTPS connection (1): query1.finance.yahoo.com:443
DEBUG   : https://query1.finance.yahoo.com:443 "GET /v8/finance/chart/GBPJPY=X?period1=1514476800&period2=1514908800&interval=1d&lang=en-US&corsDomain=finance.yahoo.com&.tsrc=finance HTTP/1.1" 200 1396
INFO    : Fetching: GBPJPY=X (time: 2018-01-03 16:00:00+00:00)
DEBUG   : https://query1.finance.yahoo.com:443 "GET /v8/finance/chart/GBPJPY=X?period1=1514390400&period2=1514822400&interval=1d&lang=en-US&corsDomain=finance.yahoo.com&.tsrc=finance HTTP/1.1" 200 1394
fish: Job 2, 'bean-price /Users/williamdavies…' terminated by signal SIGSEGV (Address boundary error)

If I use the -n flag the dry run succeeds. When I use --no-cache the command succeeds. I am fetching 261 prices for a single currency pair (GBP /JPY). I am using this source: "JPY:yahoo/GBPJPY=X".

Wanted to share my workaround/solution and bring attention to the bug (if it is a bug!).

Thank you!

beancount.prices.sources.google.Source.get_historical_price() is no longer working as expected

Original report by Zhuoyun Wei (Bitbucket: wzyboy, GitHub: wzyboy).

---

Hi,

I found a bug where bean-price always returns today's price no matter what date I provided. I root the cause here: https://bitbucket.org/blais/beancount/src/621cec5ed38bcd128a3502a3b5c367f283deffe2/beancount/prices/sources/google.py?at=default&fileviewer=file-view-default#google.py-181:184

The Google Finance API now (since when? I have no idea) always returns a full year's data (from ~365 days ago to today), ignoring startdate and enddate parameters. So fetching the most recent record (line 1) is always to get today's price.

Bean-price: support fetch over range of dates

Original report by Vivek G (Bitbucket: seltzered, GitHub: seltzered).

---

I've added initial support for bean-price to fetch over a range of dates in https://bitbucket.org/seltzered/beancount/commits/branch/bean-price-daterange .

Currently it attempts to fetch over dates that may be invalid (e.g. weekends) and will filter out redundant results.

Yahoo historical prices unavailable

Original report by Hugo Ideler (Bitbucket: HugoId, GitHub: hugoideler).

---

It seems that fetching historical Yahoo prices is no longer supported using ichart.yahoo.com.

Example:

#!sh

$ bean-price --no-cache -e "EUR:yahoo/MSFT" --date=2017-07-06
ERROR   : Could not fetch for job: DatedPrice(base='MSFT', quote='EUR', date=datetime.date(2017, 7, 6), sources=[PriceSource(module=<module 'beancount.prices.sources.yahoo' from '/usr/lib/python3.6/site-packages/beancount/prices/sources/yahoo.py'>, symbol='MSFT', invert=False)])

Current prices still work:

#!sh

$ bean-price --no-cache -e "EUR:yahoo/MSFT" 
2017-07-07 price MSFT                                69.46 EUR

This seems confirmed at the Yahoo forums: https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503 .

Question: How to create a python package for a price source?

Original report by vonpupp (Bitbucket: vonpupp, GitHub: vonpupp).

---

Hi,

I have created a custom price source that seems to work fine, however I would like to be able to create a package for it so it can be easily distributed and maintained without messing with beancount source code.

Currently I have the following package skeleton (I used cookicutter)

.
├── beancount_cryptocompare
│   ├── cryptocompareusd.py
│   └── __pycache__
│       ├── cryptocompareusd.cpython-36.pyc
│       └── __init__.cpython-36.pyc
├── beancount_cryptocompare.egg-info
│   ├── dependency_links.txt
│   ├── not-zip-safe
│   ├── PKG-INFO
│   ├── requires.txt
│   ├── SOURCES.txt
│   └── top_level.txt
├── ...
├── setup.cfg
├── setup.py

When I run a python shell and import beancount_cryptocompare.cryptocompareusd it doesn't complain, however when I use the namespace on the price attribute it doesn't work, example:

(.env3) > $ bean-price -e USD:cryptocompareusd/ETH --no-cache                                                                                                                     
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 147, in import_source
    __import__(default_name)
ModuleNotFoundError: No module named 'beancount.prices.sources.cryptocompareusd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 151, in import_source
    __import__(module_name)
  File "/home/av/repos/crypto/beancount_cryptocompare/beancount_cryptocompare/cryptocompareusd.py", line 10, in <module>
    import cryptocompare
ModuleNotFoundError: No module named 'cryptocompare'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/bean-price", line 2, in <module>
    from beancount.prices.price import main; main()
  File "/usr/lib/python3.6/site-packages/beancount/prices/price.py", line 310, in main
    args, jobs, entries = process_args()
  File "/usr/lib/python3.6/site-packages/beancount/prices/price.py", line 281, in process_args
    psource_map = find_prices.parse_source_map(source_str)
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 99, in parse_source_map
    for source_str in source_strs.split(','))
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 99, in <genexpr>
    for source_str in source_strs.split(','))
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 124, in parse_single_source
    module = import_source(short_module_name)
  File "/usr/lib/python3.6/site-packages/beancount/prices/find_prices.py", line 155, in import_source
    module_name, exc))
ImportError: Could not find price source module "cryptocompareusd": No module named 'cryptocompare'

If I try with the full namespace, it doesn't work either, example:

av@sa4 ~/repos/crypto/beancount_cryptocompare                                                                                                                           [20:54:15] 
(.env3) > $ bean-price -e USD:beancount_cryptocompare.cryptocompareusd/ETH --no-cache                                                                                             
usage: bean-price [-h] [-e] [-v] [-d DATE] [-i] [-u] [-c] [-a] [-s] [-n]
                  [--cache CACHE_FILENAME] [--no-cache] [--clear-cache]
                  sources [sources ...]

From what I understood the find_prices works as some sort of plugin importer or similar, is that right?

IMHO it would be very desirable not to mess with PYTHONPATH nor beancount source. Would that be possible?

Could you please help me?

Thank you very much.

test suite fails with Python 3.5: SyntaxError: invalid syntax f"..."

Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).

---

I don't know if you care (if not, feel free to close) but one test fails with Python 3.5:

ERROR collecting beancount/prices/sources/quandl_test.py
beancount/prices/sources/quandl_test.py:12: in <module>
    from beancount.prices.sources import quandl
E     File "/home/tbm/scratch/cvs/beancount/beancount/prices/sources/quandl.py", line 56
E       url = f"https://www.quandl.com/api/v3/datasets/{database}/{dataset}.json"
E                                                                               ^
E   SyntaxError: invalid syntax

Bean-price --update today's price

What are the criteria for fetching a price? I have noticed that today's price is never fetched. This could be intentional, but then it means that a price fetcher that only provides the current price will never be updated. Is this a bug?

Set up CI

I see there is a .travis.yml file, but I believe travis is not actually running tests / lints on this project. Could we set it up, to ensure that all PRs do not break the code?

Add support for Financial Times and Coinbase as datasources for bean-price

Original report by Raj Krishnan (Bitbucket: rajk04, GitHub: rajk04).

---

Mike has created a standalone script to fetch prices from coinbase and financial times. It would be useful to integrate these as additional data sources for bean-price, as it would offer out of the box conversion for cryptocurrencies and a lot of UK funds.

(Mike has given his approval for adapting his script for beancount)

I would be interested in implementing it, in case this is a feature we would want in beancount.

Publish beanprice to PyPI

I've made a new tag: https://github.com/beancount/beanprice/releases/tag/v1.0.1

ratesapi source no longer works

Looks like exchangerate.host no longer works - they require an API key now and also a the response structure changed. I think it's a similar thing that happened to ratesapi.io back in 2021. Anybody knows an alternative?

Use Source.get_prices_series when fetching many prices

Instead of sending one HTTP request per requested date, we could use existing Source.get_prices_series method to fetch price for multiple days using one HTTP.

I tried fetching the prices of one ticker for the last 8 years (based on my ledger), and after few minutes the tool failed. I locally hacked the tool to use Source.get_prices_series and I got all prices in ~2s. I will try to clean up my code and get it in good enough shape to send a PR.

Yahoo price source stacktrace for renamed/missing companies

I'm getting the following stacktrace while running bean-price:

Traceback (most recent call last):
  File "/usr/local/bin/bean-price", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/beancount/prices/price.py", line 650, in main
    price_entries = sorted(price_entries, key=lambda e: e.currency)
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/beancount/prices/price.py", line 439, in fetch_price
    srcprice = fetch_cached_price(source, psource.symbol, dprice.date)
  File "/usr/local/lib/python3.8/site-packages/beancount/prices/price.py", line 347, in fetch_cached_price
    result = (source.get_latest_price(symbol)
  File "/usr/local/lib/python3.8/site-packages/beancount/prices/sources/yahoo.py", line 89, in get_latest_price
    result = parse_response(response)
  File "/usr/local/lib/python3.8/site-packages/beancount/prices/sources/yahoo.py", line 49, in parse_response
    return content['result'][0]
IndexError: list index out of range

Tracking this down, it seems the offending query is for USD:yahoo/CELG (CELG has been renamed to CELG-RI):

Payload of the request: {'symbols': 'CELG', 'fields': 'symbol,regularMarketPrice,regularMarketTime', 'exchange': 'NYSE', 'lang': 'en-US', 'corsDomain': 'finance.yahoo.com', '.tsrc': 'finance'}

Response 200: {'quoteResponse': {'result': [], 'error': None}}

It seems there's a condition check missing from parse_response.

Implement price source for IEX

Original report by Martin Blais (Bitbucket: blais, GitHub: blais).

---

https://iextrading.com/developer/docs/#stocks

Should we use beanprice v2 or v3

First, I am sorry for using issues to start a discussion (what do you think about enabling Github Discussions to have a place where questions related to beanprice development can be asked?).

Coming back to the question. AFAICT beanprice has been extracted from beancount v3 repo (beancount v2 still has bean-price binary), but beanprice repo depends on beancount v2. This means that when someone install beanprice, there is a conflict between the two binaries (I am actually not sure which bean-price is installed at that point, and it might be confusing to the user).

I was wondering if we should switch beanprice repo to use beancount v3 instead. What do you think?

bean-price -a is broken (regression after #210)

Original report by Mikhail Gusarov (GitHub: misha-ridge).

---

bean-price -a is now broken, as it sets undeclared to True, and in beancount/beancount#210 undeclared was changed to accept a name of default source:

Traceback (most recent call last):
  File "/Users/dottedmag/w/fava/lib/python3.7/site-packages/beancount/prices/find_prices.py", line 147, in import_source
    __import__(default_name)
ModuleNotFoundError: No module named 'beancount.prices.sources.True'

Unable to install

The instructions say:

To install beanprice, run:

pip install git+https://github.com/beancount/beanprice.git

I tried this. It starts doing some work, up until it gets to this line:

Running setup.py install for beancount ... error

It then prints out the following error text and fails:

ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jwqrifcb/beancount/setup.py'"'"'; file='"'"'/tmp/pip-install-jwqrifcb/beancount/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-k63ptjfe/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /var/home/chris/.local/include/python3.9/beancount
cwd: /tmp/pip-install-jwqrifcb/beancount/
Complete output (352 lines):
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Addition of MUTF_IN to google.py

Original report by Bala Ramadurai (Bitbucket: nikimonikado, GitHub: balaramadurai).

---

Hello,
I love beancount's simplicity and sophistication :) It has made a lot of things easy for me in data entry including the bean-price.

One niggle though - MUTF_IN is not working for me. I have now added this to google.py manually alongside "MUTF_CA" and it seems to be working fine. But, I thought I'd raise this as an issue here. I am not knowledgeable enough in python to raise a PR for this change , so here goes 😄

Thanks

OANDA source no longer works

It seems that the API point returns 502 now. Probably they have removed that API point, as the document now shows a new revision.

YahooError: Status 401: Invalid Cookie

Yesterday I started getting the following error when trying to use bean-price:

ERROR   : Error fetching EURUSD=X: Status 401: {'code': 'Unauthorized', 'description': 'Invalid Cookie'}

Command line:

bean-price --no-cache --clobber --inactive budget.beancount >> prices.beancount

When I remove --no-cache, the issue is almost gone, but not always.
I suppose it's related with rate limiting or other measures on Yahoo side.

Question: Use custom source without modifying bean-price package

Image I have

my_pricer_sources/
|-- source1.py

Can I refer to this source in the bean-price command line without put this source1.py inside bean-price package?

commodity directive introducing syntax errors

Original report by Cameron Hurd (Bitbucket: cam5, GitHub: cam5).

---

If I have a file test.beancount, like below, and I run bean-check ./test.beancount on it, I receive some unexpected output:

test.beancount:14:      Invalid token: 'Anything'
test.beancount:14:      Invalid token: 'Else"'

; test.beancount
2019-01-01 open Assets:TFSA:Cash  USD, CAD
2019-01-01 open Assets:TFSA:FB    FB
2019-01-01 open Expenses:Test     CAD

2019-01-01 * "Buying Shares"
  Assets:TFSA:FB            10.00 FB   {250.00 USD}; 294.21 CAD @ 25.00 ea, doesn't include commission fees
  Assets:TFSA:Cash

2019-01-01 commodity FB
  name: "Facebook Shares"
  asset-class: "stock"
  price: "USD:google/NASDAQ:FB”

2019-03-23 * "Anything Else"
  Assets:TFSA:Cash         -10.00 CAD
  Expenses:Test

I read the section "Prices from a Beancount Input File" from Prices in Beancount, but I'm not sure how to proceed in debugging!

Support implicit prices for explicitly specified pairs

Original report by Martin Blais (Bitbucket: blais, GitHub: blais).

---

Re. your first email: That's not great behavior from Beancount.
Ideally at the very least it should issue a warning, or try to resolve the rates.

In general, replacing the currency conversion with an unconditional graph search is too dangerous IMO, it could start getting really expensive without us noticing. Besides, if we did that I would consider redesigning the price map from scratch, avoiding filling it in with inverses in the first place. A better approach to solve the networth report issue might be to have a function which fills in missing pairs in the existing price map for particular desired currencies (i.e., the operating currencies in this case) once and then use that. What do you think?

I'll comment further on your PR.
Thanks,

On Thursday, January 12, 2017 at 2:56:34 AM UTC-5, Justus Pendleton wrote:
As a follow up to myself...to try to understand what is going on I reduced it down so I only have one operating_currency -- AUD.

One surprise is the way currency conversions work in holdings_report.py:convert_to_currency. What I mean is:

I have this...

2017-01-11 price AUD 0.7379 USD
2017-01-11 price USD 22,568.27 VND

But because I don't have an explicit AUD <-> VND price the call to prices.get_latest_price(price_map, base_quote) returns (None, None). This results in it throwing out anything I have with a cost_currency of VND.

If I add another entry, so it looks like

2017-01-11 price AUD 0.7379 USD
2017-01-11 price AUD 16,856.94 VND
2017-01-11 price USD 22,568.27 VND

then things work as expected. Is it even possible to have bean-price fetch two different prices for AUD like that? One in VND and one in USD?

It seems like currency/commodity conversion should be using a graph rather than just a dictionary lookup. I have a patch that does this that I can submit so long as I'm not misunderstanding how all this is supposed to work. When I change it to use a graph lookup then the problem I talked about in my original post goes away.

Unclear how to install this / Poor documentation

Hello.

I'm trying to install this so I can retrieve price history automatically, instead of doing it manually. I use Fava and would prefer something integrated with that, but as far as I can tell beanprice is all there is.

The installation instructions say:

Install using the usual pip incantation.

What is this supposed to mean? I've tried guessing and running pip install beanprice and pip install bean-price, and both of these commands give me the following error output:

ERROR: Could not find a version that satisfies the requirement beanprice (from versions: none)
ERROR: No matching distribution found for beanprice

So now I'm at a loss. There also doesn't seem to be any documentation on how to actually use the script, as the readme just points to the entire Beancount Google Groups forum.

I'd appreciate any help. Thanks.

bean-price --update no longer exists

I am starting to use bean-price and one thing that I noted is that there is no --update option.

usage: bean-price [-h] [--version] [-e] [-v] [-d DATE] [-i] [-u UNDECLARED] [-c] [-a] [-s] [-n]
[--cache CACHE_FILENAME] [--no-cache] [--clear-cache]
sources [sources ...]

Fetch prices from the internet and output them as Beancount price directives.

positional arguments:
sources A list of filenames (or source "module/symbol", if -e is specified) from which
to create a list of jobs.

options:
-h, --help show this help message and exit
--version, -V show program's version number and exit
-e, --expressions, --expression
Interpret the arguments as "module/symbol" source strings.
-v, --verbose Print out progress log. Specify twice for debugging info.
-d DATE, --date DATE Specify the date for which to fetch the prices.
-i, --inactive Select all commodities from input files, not just the ones active on the date
-u UNDECLARED, --undeclared UNDECLARED
Include commodities viewed in the file even without a corresponding Commodity
directive, from this default source. The currency name itself is used as the
lookup symbol in this default source.
-c, --clobber Do not skip prices which are already present in input files; fetch them anyway.
-a, --all A shorthand for --inactive, --undeclared, --clobber.
-s, --swap-inverted For inverted sources, swap currencies instead of inverting the rate. For
example, if fetching the rate for CAD from 'USD:google/^CURRENCY:USDCAD'
results in 1.25, by default we would output "price CAD 0.8000 USD". Using this
option we would instead output " price USD 1.2500 CAD".
-n, --dry-run Don't actually fetch the prices, just print the list of the ones to be fetched.

cache:
--cache CACHE_FILENAME
Enable the cache and with the given cache name.
--no-cache Disable the price cache.
--clear-cache Clear the cache prior to startup

By the way this is the version I am using: Beancount 2.3.6 (git:d77540c4; 2023-10-05)

Implement IEX historical price fetches

IEX looks like it now supports historical prices:
https://iextrading.com/developers/docs/#hist

Our price source does not:
https://github.com/beancount/beancount/blob/master/beancount/prices/sources/iex.py#L60

That source would be useful for many I thin.

--update doesn't work for commodities not held at cost

Input file:

plugin "beancount.plugins.auto_accounts"

2021-04-01 commodity EUR
  price: "USD:yahoo/EURUSD=X"

2021-04-01 *
  Assets:Cash 100.00 EUR @ 150.00 USD
  Equity:Opening-Balances

this works:

> bean-price test.beancount
2021-04-27 price EUR                                  1.21 USD

this does nothing:

> bean-price test.beancount --update

If I add {...} to the transaction, both commands work as expected.

I believe this bug has been previously pointed out in #26 (cc @doriath)

Price rounding

I expect beanprice shouldn't rounding any price.
Due to beanprice writes prices with dcontext the rounding may happen.

$ bean-price -e 'RUB:yahoo/VTBR.ME'
2021-11-10 price VTBR.ME                           0.05053 RUB

$ bean-price current.beancount | grep VTBR
2021-11-10 price VTBR                                 0.05 RUB

current.beancount has commodity definition:

1970-01-01 commodity VTBR
  price: "RUB:yahoo/VTBR.ME"

$ bean-doctor display_context current.beancount
...
RUB             : sign=1   integer_max=5   fractional_common=2   fractional_max=2   "-00000.00" "-00000.00"
...
__default__     : sign=0   integer_max=1   fractional_common=_   fractional_max=_   "0.*" "0.*"

Pipe price sources

I would like to have some way of combining price sources. For example, say there is a source that denotes the price of Asset X in USD, I would then like to combine that with a source for USD denoted in EUR to get the price of asset X in EUR.

I tried to write this as just another price source, but the result will be quite ugly with all the character escapes that are needed. What do you think would be the best course of action?

Something like this could work with minimal changes to the base source parsing, but the syntax becomes quite convoluted. By adding more string escaping it could work without changes to the current parsing at all.

2000-01-01 commodity AAPL
  price: "USD:pipe/yahoo//AAPL>coinbase//USD-EUR"

Using `bean-price --all` causes an error

Original report by lucassz (Bitbucket: lucassz, GitHub: lucassz).

---

When running a command such as bean-price --all main.bean, I get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/beancount/prices/find_prices.py", line 147, in import_source
    __import__(default_name)
ModuleNotFoundError: No module named 'beancount.prices.sources.True'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/bean-price", line 4, in <module>
    from beancount.prices.price import main; main()
  File "/usr/local/lib/python3.7/site-packages/beancount/prices/price.py", line 354, in main
    args, jobs, entries, dcontext = process_args()
  File "/usr/local/lib/python3.7/site-packages/beancount/prices/price.py", line 347, in process_args
    entries, args.date, args.inactive, args.undeclared))
  File "/usr/local/lib/python3.7/site-packages/beancount/prices/find_prices.py", line 366, in get_price_jobs_at_date
    default_source = import_source(undeclared_source)
  File "/usr/local/lib/python3.7/site-packages/beancount/prices/find_prices.py", line 151, in import_source
    __import__(module_name)
TypeError: __import__() argument 1 must be str, not bool

Based on my debugging, it seems that this is because the --undeclared argument is being set to True, whereas it needs to specify a default source as indicated in the documentation for the --undeclared option.

Yahoo test is broken

https://github.com/beancount/beanprice/blame/master/beanprice/sources/yahoo_test.py#L54