ryanwang520/aiothrift

AsyncIO (PEP 3156) Thrift support

aiothrift

Asyncio implementation for thrift protocol, which is heavily based on thriftpy2.

Documentation: https://aiothrift.readthedocs.org/

Installation

$ pip install aiothrift

Usage example

Thrift file

service PingPong {
    string ping(),
    i64 add(1:i32 a, 2:i64 b),
}

Server

import asyncio
import aiothrift

pingpong_thrift = aiothrift.load('pingpong.thrift', module_name='pingpong_thrift')

class Dispatcher:
    def ping(self):
        return "pong"

    async def add(self, a, b):
        await asyncio.sleep(1)
        return a + b

async def main():
  server = await aiothrift.create_server(pingpong_thrift.PingPong, Dispatcher()))
  async with server:
      await server.serve_forever()

asyncio.run(main())

Client

import asyncio
import aiothrift

pingpong_thrift = aiothrift.load('pingpong.thrift', module_name='pingpong_thrift')

async def go():
    conn = await aiothrift.create_connection(pingpong_thrift.PingPong)
    print(await conn.ping())
    print(await conn.add(5, 6))
    conn.close()

asyncio.run(go())

Or use ConnectionPool

import asyncio
import aiothrift

pingpong_thrift = aiothrift.load('pingpong.thrift', module_name='pingpong_thrift')

async def go():
    client = await aiothrift.create_pool(pingpong_thrift.PingPong)
    print(await client.ping())
    print(await client.add(5, 6))
    client.close()
    await client.wait_closed()

asyncio.run(go())

It's just that simple to begin with aiothrift, and you are not forced to use aiothrift on both server and client side. So if you already have a normal thrift server setup, feel free to create an async thrift client to communicate with that server.

Requirements

• Python >= 3.7.0
• async-timeout
• thriftpy2

LICENSE

aiothrift is offered under the MIT license.

aiothrift's Issues

await for map result

• version : 0.2.7
• problem : for method map<i32,i32> test(1: list<i32> ids) , would cause error RuntimeWarning: coroutine 'read_map_begin' was never awaited
• reason : in file protocol.py , line 362, no keyword await for k_type, v_type, sz = read_map_begin(reader)

Request for sharper error messages when invalid thrift calls are made

A mistake I make every now and then is to either (1) supply the incorrect Service identifier to a create_pool command or (2) supply the incorrect Service identifier when I instantiate a thrift server. The resulting error messages do not give much insight into the cause of the problem.

Over time I have written a long checklist of possible causes of problems that I can work through to find the cause of the error. Still, someone in this position would suffer less loss-of-momentum if the error messages were more descriptive. This ticket is a request to improve error handling.

In a situation just now, I incorrectly instantiated the thrift server, by supplying the wrong Service as an argument to aiothrift.create_server. For this situation, the stack trace given on the client side is helpful, because it allows me to see which call is not being dealt with correctly.

The logging on the incorrectly setup-up server side is unhelpful,

{path}/venv/lib/python3.7/site-packages/aiothrift/protocol.py:466: RuntimeWarning: coroutine 'skip' was never awaited
  skip(self.trans, ttype)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "{path}{/venv/lib/python3.7/site-packages/aiothrift/server.py", line 31, in __call__
    await self.processor.process(iproto, oproto)
  File "{path}/venv/lib/python3.7/site-packages/aiothrift/processor.py", line 68, in process
    api, seqid, result, call = await self.process_in(iprot)
  File "{path}/venv/lib/python3.7/site-packages/aiothrift/processor.py", line 16, in process_in
    await iprot.skip(TType.STRUCT)
TypeError: object NoneType can't be used in 'await' expression

Could the client side check for a None situation, and give an error message instead of this stack trace above? As an example of the error message, "There is no exposed method {blah} on service interface {name of incorrectly instantiated Service}"?

Exceptions aren't propagated from server to client

If the server raises an exception then those exceptions aren't propagated to the client. The normal thrift library automatically handles this by sending a ThriftApplicationException from the server to the client as part of the response message.

How can I propagate exceptions from the server to the client using aiothrift? Any help would be appreciated.

Datagram endpoint support

Are there any plans to support UDP communications?

I came across this library while trying to create a asyncio compatible client for the Jaeger OpenTracing collector. The collector accepts Thrift binary format trace information by listening on a datagram interface. The existing Python jaeger client is a Python 2.7 library which is not suitable for use in my Python 3 asyncio based applications.

Would the change roughly be adding a create_datagram_endpoint function (similar to the existing open_connection) and then creating something like a ThriftDatagram that takes a asyncio.DatagramProtocol in a similar way to the existing ThriftConnection?

Timeout not working as expected in create_pool

Python 3.7.3 / aiothrift 0.2.3 / thrifypy2 0.4.14

Aiothrift create_pool is not respecting my timeout param,

            rpc_to_worker = await aiothrift.create_pool(
                service=service,
                address=(ip, 6985),
                timeout=2)

It takes around a minute to timeout.

This line is in aiothrift connection.py,

reader, writer = await asyncio.open_connection(host, port, **kw)

Timeout behaviour seems to work when the above line is changed to this,

    fut = asyncio.open_connection(host, port, **kw)
    reader, writer = await asyncio.wait_for(fut, timeout=timeout)

Adapted from here, https://stackoverflow.com/questions/29756507/how-can-i-add-a-connection-timeout-with-asyncio.

Does the library support FramedTransport?

seems not working with TNonblockingServer, whose communication always be with header of 4bytes message length

Expecting a release (SSL support)

Hi,

create_server does not pass kw to asyncio.start_server at v0.1, so it's hardly support SSL.

BUT,
The latest aiothrift/server.py has already add kw.

I am expecting a release.

Using aiothrift server with a non-aiothrift client

I am trying to get a thriftpy server to use aiothrift without changing the clients. When I send a request to the server I am getting the following exception:

2017-12-05 17:36:59,860 aiothrift[53]: CRITICAL unhandled app exception
Traceback (most recent call last):
  File "/home/thesamet/dev/tbcode/.pants.d/pyprep/requirements/CPython-3.5.3/3df7c8a19a9b0f11a6528cd277dcbb5b2e6c3adb/.deps/aiothrift-0.1-py3-none-any.whl/aiothrift/server.py", line 23, in __call__
    yield from self.processor.process(iproto, oproto)
  File "/home/thesamet/dev/tbcode/.pants.d/pyprep/requirements/CPython-3.5.3/3df7c8a19a9b0f11a6528cd277dcbb5b2e6c3adb/.deps/aiothrift-0.1-py3-none-any.whl/aiothrift/processor.py", line 71, in process
    api, seqid, result, call = yield from self.process_in(iprot)
  File "/home/thesamet/dev/tbcode/.pants.d/pyprep/requirements/CPython-3.5.3/3df7c8a19a9b0f11a6528cd277dcbb5b2e6c3adb/.deps/aiothrift-0.1-py3-none-any.whl/aiothrift/processor.py", line 18, in process_in
    api, type, seqid = yield from iprot.read_message_begin()
  File "/home/thesamet/dev/tbcode/.pants.d/pyprep/requirements/CPython-3.5.3/3df7c8a19a9b0f11a6528cd277dcbb5b2e6c3adb/.deps/aiothrift-0.1-py3-none-any.whl/aiothrift/protocol.py", line 425, in read_message_begin
    self.trans, strict=self.strict_read)
  File "/home/thesamet/dev/tbcode/.pants.d/pyprep/requirements/CPython-3.5.3/3df7c8a19a9b0f11a6528cd277dcbb5b2e6c3adb/.deps/aiothrift-0.1-py3-none-any.whl/aiothrift/protocol.py", line 176, in read_message_begin
    message='No protocol version header')
thriftpy.protocol.exc.TProtocolException: TProtocolException(type=4)

I see that the expectation of getting a version can be controlled by a variable named strict_read, but I don't see a way to control it as a user through create_server().

I am not too familiar with the thrift wire format - is my client code being non-conforming if it doesn't send version information?

Skip not awaited

I am using 0.2.5, installed via pip.

Logging shows this error, from TBinaryProtocol::skip,

/home/admin/pydist/lib/aiothrift/protocol.py:466: RuntimeWarning: coroutine 'skip' was never awaited
  skip(self.trans, ttype)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback