Batch fetch operations

[ccxt]

Batch fetch operations

[nmiculinic]

Regading asycio, I have a suggestion. Reading your code, the only function doing HTTP IO is fetch. This one could be made async using python's aiohttp; examples of which are many online. Consequently, every other function depending on it becomes a coroutine from load_products to everything else. Thus instead of having

a = fetch(....)

You'd have

a = await fetch(..)

which doesn't seem to hard to integrate into the transpiler. Maybe you could create new module aioccxt for users who'd like asyncio capabilities with python3.5+ API (async/await)

[Nilotaviano]

First of all, sorry if I'm posting this in the wrong place. I've seen you implemented fetchTickers for some exchanges, I'm just wondering if you are planning to do the same for bittrex and poloniex. I may be able to do it at night, if you take pull requests.

[Nilotaviano]

Wow, that was fast! Thank you. I was looking forward to what would be my first pull request though hahaha. I'll be testing ccxt in the next few days, so I will definitelly be participating in the development.

[kroitor]

Hi!

Currently:

1. we have support for fetchTickers where applicable
2. also have support for async/await
3. and have usage examples for asyncio

Having the above, the task of making batch_fetch_* with respect to the exchanges' rateLimit is as trivial as writing the following short snippet (you can wrap it in a function easily):

# -*- coding: utf-8 -*-

import asyncio
import ccxt.async as ccxt

#------------------------------------------------------------------------------

async def poll(tickers):
    i = 0
    kraken = ccxt.kraken()
    while True:
        symbol = tickers[i % len(tickers)]
        yield (symbol, await kraken.fetch_ticker(symbol))
        i += 1
        await asyncio.sleep(kraken.rateLimit / 1000)

#------------------------------------------------------------------------------

async def main():
    async for (symbol, ticker) in poll(['BTC/USD', 'ETH/BTC', 'BTC/EUR']):
        print(symbol, ticker)

#------------------------------------------------------------------------------

asyncio.get_event_loop().run_until_complete(main())

Basically, this is how you batch_fetch sequentially from the same exchange.
You can also do it in parallel in one go for multiple exchanges.

We don't want to restrict the polling rate, so to keep everything explicit we leave it up to the user to design his event loop. That is at least until we implement an internal poller inside the library (hopefully soon). Until then, I think like closing this issue and maybe we will return to it later to unify it even more (if needed).

[kroitor]

Yeah, that's actually not so hard to implement, I will do that. The only caveat here is staying below the exchange rate limit with concurrent requests (this might require a custom dispatch queue). I would also like to know your opinion whether you think remaining py2-compatible is still worth it? Afaik, the concurrent.futures is a py3-only feature, so a fallback might be necessary for py2 as well...

[nmiculinic]

I would also like to know your opinion whether you think remaining py2-compatible is still worth it? Afaik, the concurrent.futures is a py3-only feature, so a fallback might be necessary for py2 as well...

Python2 is already in deprecation cycle, and almost all libraries today are either py3 only or py2-3 compatible with six, or the like. In my opinion, most users don't care for py2 and are not dependent on legacy code. Python3.5+ has cool async/await syntax and many other features.

Yeah, that's actually not so hard to implement, I will do that. The only caveat here is staying below the exchange rate limit with concurrent requests (this might require a custom dispatch queue).

How familiar are you with asyncio and coroutines?
Asyncio has become standard way to use concurrent programming in Python (notice I use concurrent, not parallel, there's a distinction, but I digress). aiohttps is http library for it. http://aiohttp.readthedocs.io/en/stable/

The async/await syntax is python3.5+ and it's quite cool and useful.

Thus your class functions can become coroutines (You'll have async def instead of def). For rate limiting, you can check if you can make this call, and if you should rate limit, you put on hold via asyncio.sleep(<some time in seconds.. 1 for example). This way you're yielding control back to the scheduler who controls all coroutines.

Something like:

async def x(self, arg...):
    while not self.within_rate_limit():
          await asyncio.sleep(1)

Fallback could be then if market API doesn't support it:

async def batchFetchTicker(self, symbols):
    return await asyncio.gather([self.FetchTicker(symbol) for symbol in symbols])

[kroitor]

How familiar are you with asyncio and coroutines?

I think I know asyncio ) I specialize in network and p2p-programming, and I heavily use async/await, Promises et al in JavaScript. I have a lot of experience with programming concurrency in web and c/c++, and I know and understand all that stuff pretty well. Races, waiting groups, finalisations, exclusions, etc... I haven't tried this syntax in Python yet, because it's a new feature and Python is not my "language number one", but if you look at async/await in JS, you'll definitely notice they're identical (or very close).

// JavaScript

// example
async fetchTicker (product) {
    let response = await this.publicGetStats ({
        'currency': this.productId (product),
    });
    let ticker = response['stats'];
    let timestamp = this.milliseconds ();
    return {
        'timestamp': timestamp,
        'datetime': this.iso8601 (timestamp),
        'high': parseFloat (ticker['max']),
        'low': parseFloat (ticker['min']),
        // ...
    };
}

// another example
let sleep = async ms => await new Promise (resolve => setTimeout (resolve, ms))
let testMarket = market => new Promise (async resolve => {
    let delay = market.rateLimit
    let products  = await market.loadProducts ()
    let symbols = Object.keys (products)
    console.log (market.id , symbols.length, 'symbols', symbols.join (', '))
    for (let s in symbols) {
        let symbol = symbols[s]
        await sleep (delay)
        console.log (market.id, symbol, 'ticker', await market.fetchTicker (symbol))
        await sleep (delay) 
        console.log (market.id, symbol, 'order book', await market.fetchOrderBook (symbol))
    }
})

The only reason for not using those new features in Python is backward-compatibility with older Python versions (< 3).

Thanks for answering, I will take all you said into account dealing with this issue.

[kroitor]

By the way, speaking of PEP-style, from #9

ex.fetchTicker("BTC/ETH"), ex.fetchTicker("DSH/BTC")

You can write that as ex.fetch_ticker ("BTC/ETH"), ex.fetch_ticker ("DSH/BTC"). Python and PHP versions support underscore notation for all unified methods, that's a little step closer to PEP ) In fact, both notations are supported for all unified methods in all three languages from the very beginning of this library. I couldn't decide if I should stick to one or the other notation here and there, so I decided I will support both of them everywhere. )

[nmiculinic]

Yeah, you're right, by PEP8 I should have written it with an underscore.

[kroitor]

An update: we will implement batch fetching very soon (in several days) for markets that support that, and afterwards we will make a workaround for those exchanges that don't expose such an endpoint.

[kroitor]

@nmiculinic yep, we're working on this right now and hope to deliver soon (probably, in several days... we have to do it in a seamless and portable way... but don't worry, we will integrate async mode for sure ))

[kroitor]

@Nilotaviano, sure! We had plans to implement that method for all exchanges, but if you do it first - we welcome all your pull requests! Just read the https://github.com/kroitor/ccxt/blob/master/CONTRIBUTING.md doc to avoid difficulties ) If you have any questions, don't hesitate to ask! It's not very difficult to implement, you just decouple a parseTicker function from existing parsing code in fetchTicker, like it's done with other exchanges, and voila, you now have fetchTickers! )
Thank you! )

[kroitor]

@Nilotaviano , you know, just don't bother, I've added fetchTickers to Poloniex and Bittrex already )

Note, that Bittrex returns 261 markets, but 265 tickers. There are four market ids, that are not listed by their API:

BTC-ANS = ANS/BTC
BTC-SEC = SEC/BTC // does SEC have its own crypto? %)
ETH-ANS = ANS/ETH
USDT-ANS = ANS/USDT

We would be happy if you participate in developing other aspects of the library as well ) Thanks!

[kroitor]

@Nilotaviano there's still plenty of room for contributing ) Thx!

[kroitor]

@nmiculinic hello! We're almost done with async/await support in python, and the last thing left is:
how you want to link against it? There are options:

1. Python 2 sync by default, Python 3.5+ async by default
2. Python 2 sync, Python 3 sync by default, async version imported with something like import ccxt.async as ccxt (for backward-compatibility)
3. Some other way?

We need your advice here to make it a better design for the library.

Opinions from other participants are also welcome!

[nmiculinic]

Thank you, awesome work!

As for linking, I'd suggest we follow the python zen: Explicit is better than implicit.

And go with the second option

import ccxt.async as ccxt

[kroitor]

@nmiculinic Great! It will be available in ccxt really soon! We will also update you on this here as well.

[kroitor]

Ok, good news everyone! )
We have added initial support for async/await in Python 3.5+! All unified methods are async if ccxt is linked against with import ccxt.async as ccxt! It's still undocumented, but it should be trivial to switch to. We will update the docs very soon (today). Thx for all your feedback, continuing our work on this!

[kroitor]

Also, your bug reports are very welcome!

[kroitor]

There may still be problems with installing it from PyPI, we're working on it now! Sorry for flooding your inboxes! Will update you on this soon!

[kroitor]

Initial async/await support is available in 1.4.43! Thx for your feedback! Working on...

[kroitor]

A usage example (py3.5+):
https://github.com/kroitor/ccxt/blob/master/examples/py/async.py

[kroitor]

Added this to the docs: https://github.com/kroitor/ccxt/wiki/Manual#synchronous-vs-asynchronous-calls

[macd2]

Hey there,
how can one fetch two correctly with asyncio
i need orderbook and ticker.

I used the boilerplate fro python and asyncio from the examples but it wont work. Is there an example how to do this?

thanks