thriftpy/thriftpy

Thriftpy has been deprecated, please migrate to https://github.com/Thriftpy/thriftpy2

ThriftPy

Thriftpy has been deprecated, please migrate to thriftpy2

ThriftPy is a pure python implementation of Apache Thrift in a pythonic way.

Documentation: https://thriftpy.readthedocs.org/

Installation

Install with pip.

$ pip install thriftpy

You may also install cython first to build cython extension locally.

$ pip install cython thriftpy

Code Demo

ThriftPy make it super easy to write server/client code with thrift. Let's checkout this simple pingpong service demo.

We need a 'pingpong.thrift' file:

service PingPong {
    string ping(),
}

Then we can make a server:

import thriftpy
pingpong_thrift = thriftpy.load("pingpong.thrift", module_name="pingpong_thrift")

from thriftpy.rpc import make_server

class Dispatcher(object):
    def ping(self):
        return "pong"

server = make_server(pingpong_thrift.PingPong, Dispatcher(), '127.0.0.1', 6000)
server.serve()

And a client:

import thriftpy
pingpong_thrift = thriftpy.load("pingpong.thrift", module_name="pingpong_thrift")

from thriftpy.rpc import make_client

client = make_client(pingpong_thrift.PingPong, '127.0.0.1', 6000)
print(client.ping())

See, it's that easy!

You can refer to 'examples' and 'tests' directory in source code for more usage examples.

Features

Currently ThriftPy have these features (also advantages over the upstream python lib):

• Supports Python 2.7, Python 3.4+, PyPy and PyPy3.
• Pure python implementation. No longer need to compile & install the 'thrift' package. All you need is thriftpy and thrift file.

• Compatible with Apache Thrift. You can use ThriftPy together with the official implementation servers and clients, such as a upstream server with a thriftpy client or the opposite.

  Currently implemented protocols and transports:

  • binary protocol (python and cython)
  • compact protocol (python and cython)
  • json protocol
  • buffered transport (python & cython)
  • framed transport
  • tornado server and client (with tornado 4.0)
  • http server and client

• Can directly load thrift file as module, the sdk code will be generated on the fly.

  For example, pingpong_thrift = thriftpy.load("pingpong.thrift", module_name="pingpong_thrift") will load 'pingpong.thrift' as 'pingpong_thrift' module.

  Or, when import hook enabled by thriftpy.install_import_hook(), you can directly use import pingpong_thrift to import the 'pingpong.thrift' file as module, you may also use from pingpong_thrift import PingService to import specific object from the thrift module.

• Easy RPC server/client setup.

Contribute

1. Fork the repo and make changes.
2. Write a test which shows a bug was fixed or the feature works as expected.
3. Make sure travis-ci or tox tests succeed.
4. Send pull request.

Contributors

https://github.com/eleme/thriftpy/graphs/contributors

Changelog

https://github.com/eleme/thriftpy/blob/master/CHANGES.rst

thriftpy's Issues

Java ProcessBuilder.inheritIO not working with thriftpy

I posted originaly description of my problem here: http://stackoverflow.com/questions/33769080/processbuilder-inheritio-not-working-when-python-with-thread-setdaemon

I am running python application from Java. I established two-way communication between Java and Python.
I wrote in my Java code pb.inheritIO() to redirect python prints to standard output. And redirection is not working.

I made a workaround by overloading handle method of TThreadedServer.
In while loop I forced flushing streams:

    while True:
        self.processor.process(iprot, oprot)
        sys.stdout.flush()
        sys.stderr.flush()

Could it be a valid solution to be integrated into thriftpy?

C framed transport should possibly not flush?

We're seeing a problem where a Thrift server (using upstream Apache's implementation, I believe in Java) hangs when we try to send a request bigger than 4096 bytes.

I tracked the problem down to this line in TCyBufferedTransport. We're using framing, and this line causes a single struct to be broken into two frames, because TFramedTransport creates a new frame on every flush. Presumably the server thinks it should be getting more data, but never does. I don't have access to the server, so I'm not quite sure why.

If I remove that line, everything works fine. (The call to grow would need fixing, but in my case it happens to work.) If I use TBufferedTransport instead, which doesn't need to grow a fixed-size buffer, everything works fine. Either way, the whole message is sent as a single frame.

I'm not sure who's actually at fault here — there doesn't seem to be any documentation of any of the Thrift wire formats, so I have no idea when it's legal to start a new frame. I do notice that both Apache and thriftpy will read incorrectly if they try to read a single contiguous block that spans multiple frames, but that's not what's happening here.

Library is unusable on Windows since cython usage dropped at installation, but not dropped at protocol\__init__.py file

protocol__init__.py file

from thriftpy._compat import PYPY, CYTHON
if not PYPY:
    # enable cython binary by default for CPython.
    if CYTHON:
        from .cybin import TCyBinaryProtocol, TCyBinaryProtocolFactory
        TBinaryProtocol = TCyBinaryProtocol  # noqa
        TBinaryProtocolFactory = TCyBinaryProtocolFactory  # noqa
else:
    # disable cython binary protocol for PYPY since it's slower.
    TCyBinaryProtocol = TBinaryProtocol
    TCyBinaryProtocolFactory = TBinaryProtocolFactory

in setup.py

UNIX = platform.system() in ("Linux", "Darwin")
# only build ext in CPython with UNIX platform
if UNIX and not PYPY:
# rebuild .c files if cython available
if CYTHON:
cythonize("thriftpy/transport/cybase.pyx")
cythonize("thriftpy/transport/**/*.pyx")
cythonize("thriftpy/protocol/cybin/cybin.pyx")
ext_modules.append(Extension("thriftpy.transport.cybase",
["thriftpy/transport/cybase.c"]))
ext_modules.append(Extension("thriftpy.transport.buffered.cybuffered",
["thriftpy/transport/buffered/cybuffered.c"]))
ext_modules.append(Extension("thriftpy.transport.memory.cymemory",
["thriftpy/transport/memory/cymemory.c"]))
ext_modules.append(Extension("thriftpy.transport.framed.cyframed",
["thriftpy/transport/framed/cyframed.c"]))
ext_modules.append(Extension("thriftpy.protocol.cybin",
["thriftpy/protocol/cybin/cybin.c"]))

So platform is checked on installation, but it is not checked at runtime configuration

Need faster parser

Hi~ The parser is fucking too slow now, which takes several minutes for a large thrift file, would you try to do any efforts to improve this? Thanks.

i am currently using make_client method.but i do not find a way to do transport.close()

i am currently using make_client method.but i do not find a way to do transport.close()

JSON Protocol cannot serialise nested structs/enums

Thrift file:

struct Food {
}

struct Fridge {
    1: Food food
}

import thriftpy
from thriftpy.protocol import json
from thriftpy import utils

food = thriftpy.load("test.thrift", "test_thrift")

fridge = food.Fridge(food.Food())

print (utils.serialize(fridge)) # works
print (utils.serialize(fridge, json.TJSONProtocolFactory())) #fails

  File "/Users/progs/test/test.py", line 12, in <module>
    print (utils.serialize(fridge, json.TJSONProtocolFactory()))
  File "/Library/Python/2.7/site-packages/thriftpy/utils.py", line 14, in serialize
    thrift_object.write(protocol)
  File "/Library/Python/2.7/site-packages/thriftpy/thrift.py", line 94, in write
    oprot.write_struct(self)
  File "/Library/Python/2.7/site-packages/thriftpy/protocol/json.py", line 203, in write_struct
    "payload": struct_to_json(obj)
  File "/Library/Python/2.7/site-packages/thriftpy/protocol/json.py", line 120, in struct_to_json
    field_type, field_name, field_type_spec = field_spec
ValueError: too many values to unpack

Lack of support of type set

The title could be wrong, but based on my investigation of the code with pdb it looks like thriftpy does not have support for set type?

Here's an error:
Traceback (most recent call last):
File "./convert.py", line 16, in
vehicle = thriftpy.load("vehicle.thrift", module_name="vehicle_thrift")
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 79, in load
for path in result["includes"]
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 79, in
for path in result["includes"]
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 149, in load
thrift_spec[m["id"]] = _ttype_spec(m["type"], m["name"])
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 109, in _ttype_spec
ttype = _ttype(ttype)
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 104, in _ttype
return TType.MAP, (_ttype(t[1]), _ttype(t[2]))
File "/home/dkulinski/Code/git-area/bulk-loading/mine/py2/lib/python2.7/site-packages/thriftpy/parser/init.py", line 106, in _ttype
raise Exception("ttype parse error: {0}".format(t))
Exception: ttype parse error: ['set', 'string']

The code where it fails:
def _ttype(t, module=None):
module = module or thrift_schema
if isinstance(t, str):
if "." in t:
include, field = t.split(".", 1)
return _ttype(field, module=module._includes[include])
elif t in module._struct_names:
return TType.STRUCT, getattr(module, t)
elif t in result["enums"]:
return TType.I32, getattr(module, t)
elif t in result["typedefs"]:
return _ttype(result["typedefs"][t])
else:
return getattr(TType, t.upper())
elif t[0] == "list":
return TType.LIST, _ttype(t[1])
elif t[0] == "map":
return TType.MAP, (_ttype(t[1]), _ttype(t[2]))
else:
raise Exception("ttype parse error: {0}".format(t))

Upstream compatiable Multiplexing protocol

Original discussed here at #88

The Apache Thrift implementation is different from the current implementation of thriftpy: https://issues.apache.org/jira/browse/THRIFT-563

TCyBinaryProtocol breaks streaming

After every read_struct, the protocol calls self.trans.clean(), which discards whatever else was left in the buffer. The non-cythonized protocol doesn't do this. Ex:

import thriftpy
from thriftpy.transport import TMemoryBuffer
from thriftpy.transport import TCyBufferedTransportFactory
from thriftpy.protocol import TCyBinaryProtocolFactory
from thriftpy.protocol import TBinaryProtocolFactory

demo_thrift = thriftpy.load('demo.thrift')  # struct Event { 1: i32 id }

buf = TMemoryBuffer()
trans = TCyBufferedTransportFactory().get_transport(buf)
proto = TCyBinaryProtocolFactory().get_protocol(trans)

proto.write_struct(demo_thrift.Event(123))
proto.write_struct(demo_thrift.Event(456))
trans.flush()

stream = buf.getvalue()

buf = TMemoryBuffer(stream)
trans = TCyBufferedTransportFactory().get_transport(buf)
proto = TBinaryProtocolFactory().get_protocol(trans)

event1 = demo_thrift.Event()
event2 = demo_thrift.Event()
proto.read_struct(event1)
print(event1)
proto.read_struct(event2)
print(event2)

buf = TMemoryBuffer(stream)
trans = TCyBufferedTransportFactory().get_transport(buf)
proto = TCyBinaryProtocolFactory().get_protocol(trans)

event1 = demo_thrift.Event()
event2 = demo_thrift.Event()
proto.read_struct(event1)
print(event1)
proto.read_struct(event2)
print(event2)

Pure-Python version works; Cython doesn't. (I don't know if this is a correct use of Thrift, but we're doing it, so I'm stuck with it. ;))

I'd send a PR but I'm not sure why this clause exists or what the intended behavior is. Maybe it was meant to be called when there's an exception, like in write_struct?

New thriftpy release with latest commits/PR's?

Hi @hit9 and @lxyu, I was wondering if you were planning for a new thriftpy release soon with the latest commits/PR's?

In particular, we would like to start relying upon the load_fp API added in #153, but since we plan to include functionality for this in another open source module, we'd like to depend upon a specific released version of thriftpy. Thanks for all your help!

Thriftpy and cython issue

Hi
I'm trying to install ibis on Ubuntu with anaconda 3.5 but when I do "import ibis"
i get this error

   from ibis.impala.client import (ImpalaConnection,  # noqa
  File "/home/pino/anaconda3/lib/python3.5/site-packages/ibis/impala/client.py", line 34, in <module>
    from ibis.impala.compat import impyla, ImpylaError, HS2Error
  File "/home/pino/anaconda3/lib/python3.5/site-packages/ibis/impala/compat.py", line 17, in <module>
    import impala.dbapi as impyla  # noqa
  File "/home/pino/anaconda3/lib/python3.5/site-packages/impala/dbapi.py", line 28, in <module>
    import impala.hiveserver2 as hs2
  File "/home/pino/anaconda3/lib/python3.5/site-packages/impala/hiveserver2.py", line 32, in <module>
    from impala._thrift_api import (
  File "/home/pino/anaconda3/lib/python3.5/site-packages/impala/_thrift_api.py", line 62, in <module>
    from thriftpy.protocol.binary import TBinaryProtocol  # noqa
  File "/home/pino/anaconda3/lib/python3.5/site-packages/thriftpy/protocol/__init__.py", line 13, in <module>
    from .cybin import TCyBinaryProtocol, TCyBinaryProtocolFactory
ImportError: No module named 'thriftpy.protocol.cybin'

Any idea?

Thanks

parser fails on the dead include checking with python < 3 on Windows

In the parser module, when running the dead include check:

for thrift in thrift_stack:
    if os.path.samefile(path, thrift.__thrift_file__):

the os.path.samefile method is only implemented on Windows in Python 3

UnicodeEncodeError using TJSONProtocol with unicode string (python2.7)

Hi all! This is the traceback:

Traceback (most recent call last):
  File "foo.py", line 14, in <module>
    foo.write(proto.TJSONProtocol(buff))
  File "/Users/gustavofonseca/prj/envs/thrift/lib/python2.7/site-packages/thriftpy/thrift.py", line 93, in write
    oprot.write_struct(self)
  File "/Users/gustavofonseca/prj/envs/thrift/lib/python2.7/site-packages/thriftpy/protocol/json.py", line 206, in write_struct
    "payload": struct_to_json(obj)
  File "/Users/gustavofonseca/prj/envs/thrift/lib/python2.7/site-packages/thriftpy/protocol/json.py", line 129, in struct_to_json
    outobj[field_name] = json_value(field_type, v, field_type_spec)
  File "/Users/gustavofonseca/prj/envs/thrift/lib/python2.7/site-packages/thriftpy/protocol/json.py", line 23, in json_value
    return str(val)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 1: ordinal not in range(128)

foo.thrift:

struct Foo {
    1: optional string name;
}

foo.py:

# coding: utf-8
import io

import thriftpy
import thriftpy.protocol.json as proto

thrift_spec = thriftpy.load('foo.thrift', module_name='foo_thrift')

foo = thrift_spec.Foo(name=u'pão de açúcar')
#print proto.struct_to_json(foo) 


buff = io.BytesIO()
foo.write(proto.TJSONProtocol(buff))

print buff.getvalue()

Import error on protocol package

The relevant section of the stacktrace is:

    ...
    from thriftpy.rpc import make_server
File "/var/www/jmanager_scielo_org/jmanager-env/lib/python2.7/site-packages/thriftpy/rpc.py", line 7, in <module>
    from thriftpy.protocol import TBinaryProtocolFactory
  File "/var/www/jmanager_scielo_org/jmanager-env/lib/python2.7/site-packages/thriftpy/protocol/__init__.py", line 15, in <module>
    from .cybin import TCyBinaryProtocol, TCyBinaryProtocolFactory
ImportError: /var/www/jmanager_scielo_org/jmanager-env/lib/python2.7/site-packages/thriftpy/protocol/cybin.so: undefined symbol: be64toh

I've installed thriftpy with $ pip install cython thriftpy on a Centos 5.10 64bit machine.

Am I missing something?

Recursive structs no longer supported

Thrift structures referencing themselves are no longer supported.

struct RecursiveType { 1: string data, 2: list<RecursiveType> tree }

used to work with Thriftpy 0.1.15 but do no longer work with 0.3.1.
This is a regression. Sadly the official Python thrift bindings also don't support this. Other language bindings do though.

.c files missing from repository

I am trying to build on Windows and files such as transport/cybase.c are missing from the repository.

Install requires PLY 3.4

Is there a reason why PLY 3.4 is required i.e. 3.6 won't suffice?

Missing validate method

Structs that get defined seem to be missing the validate method that is added by thrift during the gen process. Considering that I am planning to use my structs in multiple contexts I use the validate method when using the objects outside of the RPC context.

https://github.com/apache/thrift/blob/01a77ab01e7459d96059a2b49d9885d14a360ef1/compiler/cpp/src/generate/t_py_generator.cc#L908

enum in Included .thrift can not load correctly

ex:

common.thrift

enum TReturnCode {
    OK = 0,                        // no error
    INTERNAL_ERROR = 1,            // internal error
    INVALID_PARAMETER  = 2,        // invalid parameter name
    INVALID_PARAMETER_VALUE  = 3,  // invalid parameter value
    INVALID_HTTP_AUTH_HEADER = 4,  // invalid authorization
    INVALID_URI = 5,               // invalide URI
    PARTIAL_ERROR = 6,             // partial error for batch operations
}

service.thrift

include "common.thrift"

struct Selector {
    1: optional string user,        // user identifier
    2: optional string scope,       // scope
    3: optional list<common.TReturnCode> returnCodes,   // return codes list
}

service Protocol {
    void getReturnCode(1: Selector metricsMetaSelector)
}

Such two thrift files can not load. Got a error: AttributeError: type object 'TType' has no attribute 'TRETURNCODE'

Recursive union failes at startup

union Dynamic {
  1: bool boolean;
  2: i64 integer;
  3: double doubl;
  4: string str;
  5: list<Dynamic> arr;
  6: map<string, Dynamic> object;
}

thriftpy.parser.exc.ThriftParserError: No type found: 'Dynamic', at line 6

taken from: https://github.com/facebook/fbthrift/blob/master/thrift/lib/thrift/dynamic.thrift

Parsing thrift with non-named enums

So i have a thrift file with this spec inside it. (from the hiveserver2 CLI service).

enum TTypeId {
  BOOLEAN_TYPE,
  TINYINT_TYPE,
  SMALLINT_TYPE,
  INT_TYPE,
  BIGINT_TYPE,
  FLOAT_TYPE,
  DOUBLE_TYPE,
  STRING_TYPE,
  TIMESTAMP_TYPE,
  BINARY_TYPE,
  ARRAY_TYPE,
  MAP_TYPE,
  STRUCT_TYPE,
  UNION_TYPE,
  USER_DEFINED_TYPE,
  DECIMAL_TYPE,
  NULL_TYPE,
  DATE_TYPE,
  VARCHAR_TYPE,
  CHAR_TYPE
}

const set<TTypeId> PRIMITIVE_TYPES = [
  TTypeId.BOOLEAN_TYPE,
  TTypeId.TINYINT_TYPE,
  TTypeId.SMALLINT_TYPE,
  TTypeId.INT_TYPE,
  TTypeId.BIGINT_TYPE,
  TTypeId.FLOAT_TYPE,
  TTypeId.DOUBLE_TYPE,
  TTypeId.STRING_TYPE,
  TTypeId.TIMESTAMP_TYPE,
  TTypeId.BINARY_TYPE,
  TTypeId.DECIMAL_TYPE,
  TTypeId.NULL_TYPE,
  TTypeId.DATE_TYPE,
  TTypeId.VARCHAR_TYPE,
  TTypeId.CHAR_TYPE
]

When trying to parse this i get a

ThriftParserError: No named enum value found named 'TTypeId.BOOLEAN_TYPE'

It seems that values do not need to be names explicitly?

typedef on included types: Grammer error

common.thrift

typedef string UID

over.thrift

include "common.thrift"

typedef string Base
typedef Base Base1 // <-- OK
typedef common.UID Over // <-- Fail

struct Any {
1: common.UID id // <-- OK
}

We have lots of this error

I don't know why, maybe client has something wrong with closing the socket?

But the server always raise this exception.

Need a release support service multiplexing

hello

I do need service multiplexing support, I found the commit Added service multiplexing support. is what I need, any plan?

Thanks

Tornado docs & examples.

We should add documentation for using the tornado client & server - potentially even a working example to the examples directory.

Support annotations

Logger naming

Would you be open to a pull requests that uses named loggers instead of using the root Logger? My suggestion is to add:

import logging
logger = logging.getLogger(__name__)`
logger.info('hi there')

instead of

import logging
logging.info('hi there')

This allows any logs emitted by the module to be tracked back to the module; additionally we can suppress logs easily on a per-module basis. This matches the apache/thrift implementation as well.

High availability?

Is it possible to support multi server address to achieve high availability? thanks.

Can't install on windows

Tries to compile some c files and fails. Thought it was pure python?!

> c:\Python34\Scripts\virtualenv.exe venv-3.4
> venv-3.4\Scripts\activate
> pip install thriftpy

Collecting thriftpy
  Downloading thriftpy-0.2.0.tar.gz (177kB)
    100% |################################| 180kB 911kB/s
Collecting ply==3.4 (from thriftpy)
  Downloading ply-3.4.tar.gz (138kB)
    100% |################################| 139kB 1.2MB/s
Installing collected packages: ply, thriftpy
  Running setup.py install for ply
  Running setup.py install for thriftpy
    building 'thriftpy.transport.cybase' extension
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/cybase.c /Fobuild\temp.win3
2-3.4\Release\thriftpy/transport/cybase.obj
    cybase.c
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cybase build\temp.win32-3.4\Release\thriftpy/transport/cybase.obj /OUT:build\lib.win32-3.4\thriftpy\transport\cybase.pyd /IMPLIB:build\temp.win32-3.4\Release\thriftpy/transport\cybase
.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport\cybase.pyd.manifest
       Creating library build\temp.win32-3.4\Release\thriftpy/transport\cybase.lib and object build\temp.win32-3.4\Release\thriftpy/transport\cybase.exp
    building 'thriftpy.transport.buffered.cybuffered' extension
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/buffered/cybuffered.c /Fobu
ild\temp.win32-3.4\Release\thriftpy/transport/buffered/cybuffered.obj
    cybuffered.c
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cybuffered build\temp.win32-3.4\Release\thriftpy/transport/buffered/cybuffered.obj /OUT:build\lib.win32-3.4\thriftpy\transport\buffered\cybuffered.pyd /IMPLIB:build\temp.win32-3.4\Rel
ease\thriftpy/transport/buffered\cybuffered.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.pyd.manifest
       Creating library build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.lib and object build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.exp
    building 'thriftpy.transport.memory.cymemory' extension
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/memory/cymemory.c /Fobuild\
temp.win32-3.4\Release\thriftpy/transport/memory/cymemory.obj
    cymemory.c
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cymemory build\temp.win32-3.4\Release\thriftpy/transport/memory/cymemory.obj /OUT:build\lib.win32-3.4\thriftpy\transport\memory\cymemory.pyd /IMPLIB:build\temp.win32-3.4\Release\thrif
tpy/transport/memory\cymemory.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.pyd.manifest
       Creating library build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.lib and object build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.exp
    building 'thriftpy.transport.framed.cyframed' extension
    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/framed/cyframed.c /Fobuild\
temp.win32-3.4\Release\thriftpy/transport/framed/cyframed.obj
    cyframed.c
    c:\users\sesa334283\appdata\local\temp\pip-build-saq5mh71\thriftpy\thriftpy\transport\framed\../../protocol/cybin/endian_port.h(34) : fatal error C1083: Cannot open include file: 'endian.h': No su
ch file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2
    Complete output from command D:\pcs\trunk\venv-3.4\Scripts\python.exe -c "import setuptools, tokenize;__file__='C:\\Users\\SESA33~1\\AppData\\Local\\Temp\\pip-build-saq5mh71\\thriftpy\\setup.py';e
xec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\SESA33~1\AppData\Local\Temp\pip-w3af_y5n-record\install-record.txt --s
ingle-version-externally-managed --compile --install-headers D:\pcs\trunk\venv-3.4\include\site\python3.4:
    running install

    running build

    running build_py

    creating build

    creating build\lib.win32-3.4

    creating build\lib.win32-3.4\thriftpy

    copying thriftpy\hook.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\rpc.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\server.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\thrift.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\tornado.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\utils.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\_compat.py -> build\lib.win32-3.4\thriftpy

    copying thriftpy\__init__.py -> build\lib.win32-3.4\thriftpy

    creating build\lib.win32-3.4\thriftpy\parser

    copying thriftpy\parser\exc.py -> build\lib.win32-3.4\thriftpy\parser

    copying thriftpy\parser\lexer.py -> build\lib.win32-3.4\thriftpy\parser

    copying thriftpy\parser\parser.py -> build\lib.win32-3.4\thriftpy\parser

    copying thriftpy\parser\__init__.py -> build\lib.win32-3.4\thriftpy\parser

    creating build\lib.win32-3.4\thriftpy\protocol

    copying thriftpy\protocol\binary.py -> build\lib.win32-3.4\thriftpy\protocol

    copying thriftpy\protocol\exc.py -> build\lib.win32-3.4\thriftpy\protocol

    copying thriftpy\protocol\json.py -> build\lib.win32-3.4\thriftpy\protocol

    copying thriftpy\protocol\__init__.py -> build\lib.win32-3.4\thriftpy\protocol

    creating build\lib.win32-3.4\thriftpy\transport

    copying thriftpy\transport\socket.py -> build\lib.win32-3.4\thriftpy\transport

    copying thriftpy\transport\__init__.py -> build\lib.win32-3.4\thriftpy\transport

    creating build\lib.win32-3.4\thriftpy\transport\buffered

    copying thriftpy\transport\buffered\__init__.py -> build\lib.win32-3.4\thriftpy\transport\buffered

    creating build\lib.win32-3.4\thriftpy\transport\framed

    copying thriftpy\transport\framed\__init__.py -> build\lib.win32-3.4\thriftpy\transport\framed

    creating build\lib.win32-3.4\thriftpy\transport\memory

    copying thriftpy\transport\memory\__init__.py -> build\lib.win32-3.4\thriftpy\transport\memory

    running build_ext

    building 'thriftpy.transport.cybase' extension

    creating build\temp.win32-3.4

    creating build\temp.win32-3.4\Release

    creating build\temp.win32-3.4\Release\thriftpy

    creating build\temp.win32-3.4\Release\thriftpy\transport

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/cybase.c /Fobuild\temp.win3
2-3.4\Release\thriftpy/transport/cybase.obj

    cybase.c

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cybase build\temp.win32-3.4\Release\thriftpy/transport/cybase.obj /OUT:build\lib.win32-3.4\thriftpy\transport\cybase.pyd /IMPLIB:build\temp.win32-3.4\Release\thriftpy/transport\cybase
.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport\cybase.pyd.manifest

       Creating library build\temp.win32-3.4\Release\thriftpy/transport\cybase.lib and object build\temp.win32-3.4\Release\thriftpy/transport\cybase.exp

    building 'thriftpy.transport.buffered.cybuffered' extension

    creating build\temp.win32-3.4\Release\thriftpy\transport\buffered

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/buffered/cybuffered.c /Fobu
ild\temp.win32-3.4\Release\thriftpy/transport/buffered/cybuffered.obj

    cybuffered.c

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cybuffered build\temp.win32-3.4\Release\thriftpy/transport/buffered/cybuffered.obj /OUT:build\lib.win32-3.4\thriftpy\transport\buffered\cybuffered.pyd /IMPLIB:build\temp.win32-3.4\Rel
ease\thriftpy/transport/buffered\cybuffered.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.pyd.manifest

       Creating library build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.lib and object build\temp.win32-3.4\Release\thriftpy/transport/buffered\cybuffered.exp

    building 'thriftpy.transport.memory.cymemory' extension

    creating build\temp.win32-3.4\Release\thriftpy\transport\memory

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/memory/cymemory.c /Fobuild\
temp.win32-3.4\Release\thriftpy/transport/memory/cymemory.obj

    cymemory.c

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\Libs /LIBPATH:D:\pcs\trunk\venv-3.4\libs /LIBPATH:D:\pcs\trunk\venv-3.4\PCbuil
d /EXPORT:PyInit_cymemory build\temp.win32-3.4\Release\thriftpy/transport/memory/cymemory.obj /OUT:build\lib.win32-3.4\thriftpy\transport\memory\cymemory.pyd /IMPLIB:build\temp.win32-3.4\Release\thrif
tpy/transport/memory\cymemory.lib /MANIFESTFILE:build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.pyd.manifest

       Creating library build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.lib and object build\temp.win32-3.4\Release\thriftpy/transport/memory\cymemory.exp

    building 'thriftpy.transport.framed.cyframed' extension

    creating build\temp.win32-3.4\Release\thriftpy\transport\framed

    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python34\include -IC:\Python34\include /Tcthriftpy/transport/framed/cyframed.c /Fobuild\
temp.win32-3.4\Release\thriftpy/transport/framed/cyframed.obj

    cyframed.c

    c:\users\sesa334283\appdata\local\temp\pip-build-saq5mh71\thriftpy\thriftpy\transport\framed\../../protocol/cybin/endian_port.h(34) : fatal error C1083: Cannot open include file: 'endian.h': No su
ch file or directory

    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2

    ----------------------------------------
    Command "D:\pcs\trunk\venv-3.4\Scripts\python.exe -c "import setuptools, tokenize;__file__='C:\\Users\\SESA33~1\\AppData\\Local\\Temp\\pip-build-saq5mh71\\thriftpy\\setup.py';exec(compile(getattr(
tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\SESA33~1\AppData\Local\Temp\pip-w3af_y5n-record\install-record.txt --single-version-extern
ally-managed --compile --install-headers D:\pcs\trunk\venv-3.4\include\site\python3.4" failed with error code 1 in C:\Users\SESA33~1\AppData\Local\Temp\pip-build-saq5mh71\thriftpy

HTTP Server Example

Hello. I want to use thriftpy as a thrift server for JS thrift client with HTTP transport. I found example for standard thrift library, but cannot find anything similar for thriftpy.

Tornado Example Error

I'm trying to run the simplest thiftypy 0.2.0 / tornado 4.1 example I can find. So, I came across the docstring example in thiftpy/tornado.py

I set up a pingpong server, and although I can get an rpc client working fine, a tornado client as shown in the docstring will always fail with a TTransportException. So the following script:

import logging

import thriftpy.rpc
import thriftpy.tornado

from tornado import ioloop

pingpong_thrift = thriftpy.load("pingpong.thrift", module_name="pingpong_thrift")

def rpc_test():
    client = thriftpy.rpc.make_client(pingpong_thrift.PingPong, '127.0.0.1', 6000)

    return client.ping()

def tornado_test():
    client = ioloop.IOLoop.current().run_sync(
        lambda: thriftpy.tornado.make_client(pingpong_thrift.PingPong, '127.0.0.1', 6000))

    return ioloop.IOLoop.current().run_sync(client.ping)

if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)

    print rpc_test()
    print tornado_test()

will yield:

pong
DEBUG:root:socket connecting
Traceback (most recent call last):
  File "/home/bamiller/workspace/DmHbase/thriftpy_test_client.py", line 25, in <module>
    print tornado_test()
  File "/home/bamiller/workspace/DmHbase/thriftpy_test_client.py", line 19, in tornado_test
    return ioloop.IOLoop.current().run_sync(client.ping)
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/ioloop.py", line 421, in run_sync
    return future_cell[0].result()
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/thriftpy/tornado.py", line 195, in _recv
    frame = yield self._oprot.trans.read_frame()
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/thriftpy/tornado.py", line 124, in read_frame
    raise gen.Return(frame)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/bamiller/opt/python/local/lib/python2.7/site-packages/thriftpy/tornado.py", line 100, in io_exception_context
    message=str(e))
thriftpy.transport.TTransportException: TTransportException(message='', type=4)

I can run the test case in tests/test_tornado.py so I'm guessing there's nothing wrong with the version of thriftypy / tornado I'm using, but if there was a little bit more in the way of examples / docs that would be really helpful.

Thanks!

Thrift parser requires definition before a type is referenced

Recent versions of the Apache thrift compiler allow out-of-order definitions within a file. For example:

struct A {
    1: optional B b;
}

struct B {
    1: optional i32 val;
}

However, the thriftpy parser, as of 0.2.0, does not allow this. In isolation this restriction isn't too annoying, but this breaks compatibility with thrift definitions which previously worked and it also is incompatible with thrift definitions that work with the Apache compiler.

Add camelCase methods to support drop-in replacement of thrift

In an effort to support both python 2 and 3, I am trying to use both the Apache Thrift python package and thriftpy. One thing that makes it more difficult is that method names have been changed from the Apache camelCase to the more pythonic snake_case. However, it would be convenient to have camelCase versions available so that I don't have to check the python version each time I want to invoke a method.

For example, thriftpy.transport.buffered.cybuffered.TCyBuffered.is_open while the Apache version would generate an isOpen method.

Thoughts?

Cannot load Apache Storm thrift file

I was hoping to use your library to make Apache Storm work with Python 3, and it appears that it cannot load their storm.thrift file. The exception makes it look like it's a problem with how you're handling exception definitions, but it's actually that the parse() function didn't successfully parse any struct or exception blocks at all.

In [16]: thriftpy.load("storm.thrift")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-16-4e7920d7bd1c> in <module>()
----> 1 thriftpy.load("storm.thrift")

/Users/dblanchard/anaconda/envs/e-rater_dev/lib/python3.3/site-packages/thriftpy/parser.py in load(thrift_file)
    205                 for t in api.throws:
    206                     result_thrift_spec[int(t.id)] = _ttype_spec(t.ttype,
--> 207                                                                 t.name)
    208 
    209             result_attrs["thrift_spec"] = result_thrift_spec

/Users/dblanchard/anaconda/envs/e-rater_dev/lib/python3.3/site-packages/thriftpy/parser.py in _ttype_spec(ttype, name)
    124 
    125     def _ttype_spec(ttype, name):
--> 126         ttype = _ttype(ttype)
    127         if isinstance(ttype, int):
    128             return ttype, name

/Users/dblanchard/anaconda/envs/e-rater_dev/lib/python3.3/site-packages/thriftpy/parser.py in _ttype(t)
    114                 return _ttype(result["typedefs"][t])
    115             else:
--> 116                 return getattr(TType, t.upper())
    117 
    118         if t.ttype == "list":

AttributeError: type object 'TType' has no attribute 'DRPCEXECUTIONEXCEPTION'
> /Users/dblanchard/anaconda/envs/e-rater_dev/lib/python3.3/site-packages/thriftpy/parser.py(116)_ttype()
    115             else:
--> 116                 return getattr(TType, t.upper())
    117 

But if you look at result you can see what's actually going on is a larger parse failure.

ipdb> pp result
{'consts': {},
 'enums': {'TopologyInitialStatus': (['enum', 'TopologyInitialStatus', ([(['ACTIVE', 1], {'name': [('ACTIVE', 0)], 'value': [(1, 1)]}), (['INACTIVE', 2], {'name': [('INACTIVE', 0)], 'value': [(2, 1)]})], {})], {'members': [(([(['ACTIVE', 1], {'name': [('ACTIVE', 0)], 'value': [(1, 1)]}), (['INACTIVE', 2], {'name': [('INACTIVE', 0)], 'value': [(2, 1)]})], {}), 2)], 'name': [('TopologyInitialStatus', 1)]})},
 'exceptions': [],
 'services': [(['service', 'DistributedRPC', ([(['string', 'execute', ([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]}), ([2, 'string', 'funcArgs'], {'ttype': [('string', 1)], 'name': [('funcArgs', 2)], 'id': [(2, 0)]})], {}), 'throws', ([([1, 'DRPCExecutionException', 'e'], {'ttype': [('DRPCExecutionException', 1)], 'name': [('e', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('string', 0)], 'params': [(([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]}), ([2, 'string', 'funcArgs'], {'ttype': [('string', 1)], 'name': [('funcArgs', 2)], 'id': [(2, 0)]})], {}), 2)], 'name': [('execute', 1)], 'throws': [(([([1, 'DRPCExecutionException', 'e'], {'ttype': [('DRPCExecutionException', 1)], 'name': [('e', 2)], 'id': [(1, 0)]})], {}), 4)]})], {})], {'name': [('DistributedRPC', 1)], 'apis': [(([(['string', 'execute', ([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]}), ([2, 'string', 'funcArgs'], {'ttype': [('string', 1)], 'name': [('funcArgs', 2)], 'id': [(2, 0)]})], {}), 'throws', ([([1, 'DRPCExecutionException', 'e'], {'ttype': [('DRPCExecutionException', 1)], 'name': [('e', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('string', 0)], 'params': [(([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]}), ([2, 'string', 'funcArgs'], {'ttype': [('string', 1)], 'name': [('funcArgs', 2)], 'id': [(2, 0)]})], {}), 2)], 'name': [('execute', 1)], 'throws': [(([([1, 'DRPCExecutionException', 'e'], {'ttype': [('DRPCExecutionException', 1)], 'name': [('e', 2)], 'id': [(1, 0)]})], {}), 4)]})], {}), 2)]}),
              (['service', 'DistributedRPCInvocations', ([(['void', 'result', ([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]}), ([2, 'string', 'result'], {'ttype': [('string', 1)], 'name': [('result', 2)], 'id': [(2, 0)]})], {})], {'ttype': [('void', 0)], 'params': [(([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]}), ([2, 'string', 'result'], {'ttype': [('string', 1)], 'name': [('result', 2)], 'id': [(2, 0)]})], {}), 2)], 'name': [('result', 1)]}), (['DRPCRequest', 'fetchRequest', ([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('DRPCRequest', 0)], 'params': [(([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]})], {}), 2)], 'name': [('fetchRequest', 1)]}), (['void', 'failRequest', ([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('void', 0)], 'params': [(([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]})], {}), 2)], 'name': [('failRequest', 1)]})], {})], {'name': [('DistributedRPCInvocations', 1)], 'apis': [(([(['void', 'result', ([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]}), ([2, 'string', 'result'], {'ttype': [('string', 1)], 'name': [('result', 2)], 'id': [(2, 0)]})], {})], {'ttype': [('void', 0)], 'params': [(([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]}), ([2, 'string', 'result'], {'ttype': [('string', 1)], 'name': [('result', 2)], 'id': [(2, 0)]})], {}), 2)], 'name': [('result', 1)]}), (['DRPCRequest', 'fetchRequest', ([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('DRPCRequest', 0)], 'params': [(([([1, 'string', 'functionName'], {'ttype': [('string', 1)], 'name': [('functionName', 2)], 'id': [(1, 0)]})], {}), 2)], 'name': [('fetchRequest', 1)]}), (['void', 'failRequest', ([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]})], {})], {'ttype': [('void', 0)], 'params': [(([([1, 'string', 'id'], {'ttype': [('string', 1)], 'name': [('id', 2)], 'id': [(1, 0)]})], {}), 2)], 'name': [('failRequest', 1)]})], {}), 2)]})],
 'structs': [],
 'typedefs': {}}

You can see that in their thrift file they define a few union blocks, which it does not look like your parser supports, so maybe that is the root of this issue.

Support TSSLSocket

Currently supported by thrift but not thriftpy.

No license

There's no LICENSE file and no license specified in the code or docs, which means that strictly speaking, this is not open source code and no one except you is allowed to use or modify it. :)

Would you mind adding a LICENSE to appease my corporate overlords?

edit: I see the pypi metadata claims MIT, but the license text doesn't appear anywhere.

Apache parser reserved keywords

https://github.com/apache/thrift/blob/master/compiler/cpp/src/thriftl.ll#L193

Different types with the same name should cause `Duplicate typename` error

E.g. the following thrift should cause duplicate typename error

struct foo {}
enum foo {}

But this wasn't handled in thriftpy.

Possibility to make servers that run against already opened sockets

This would be great for deployment scenarios where both app processes and sockets are managed by something like Mozilla Circus.

thriftpy requires a file path to the '.thrift' file, rather than supporting a file-like object or string

If you look at the core code in thriftpy's parser and entrypoint function, thriftpy.load (which calls thriftpy.parser.parse), you'll notice that it requires a string file path (expected to be a ".thrift" file) rather than supporting a file-like object.

Although this is OK for convenience, this is bad for a number of deployment scenarios.

a) if a thriftpy-dependent codebase is deployed as part of a Python egg, then it is likely the '.thrift' file will be part of the code egg; in this case, it should be accessed via the pkg_resources API, and in this case, the ideal functions to use are resource_string or resource_stream. As the pkg_resources docs say, "Instead of manipulating __file__, you simply pass a module name or package name to resource_string, resource_stream, or resource_filename, along with the name of the resource. Normally, you should try to use resource_string or resource_stream, unless you are interfacing with code you don't control (especially C code) that absolutely must have a filename. The reason is that if you ask for a filename, and your package is packed into a zipfile, then the resource must be extracted to a temporary directory, which is a more costly operation than just returning a string or file-like object."

b) if a thriftpy-dependent codebase is deployed as part of a Python wheel, then it is likely that the '.thrift' file will be part of the wheel. In this case, the pkgutil.get_data API is the right one to use. Again, we run into a problem with requiring file paths. As the docs state, "The function returns a binary string that is the contents of the specified resource." No file paths available. Note that the Wheel PEP also indicates that Wheels may optionally have a .data directory for storing files, but it isn't clear how to access them via an API.

Thankfully, the code that needs to change here is relatively straightforward. The problematic code is in the parser here, where open() is called directly.

https://github.com/eleme/thriftpy/blob/develop/thriftpy/parser/parser.py#L436-L451

If we let thriftpy.load() accept a file-like object by inspecting for .read() via hasattr, and, in that case, allowing us to simply call it instead of trying to resolve a '.thrift' file on-disk, we will satisfy the requirements. If you agree with this change, I'll go ahead and draft a PR for it. cc @lxyu

Pull out parser

I'd love to reuse just the parse without forking; what do you guys think?

Perhaps we can move it to a package called thriftpy-parser.

binary type in python3 transtered as b'\x..', which cannot be interpreted as other languages

I transfer a binary from python to java, the python side use thriftpy with python3.4, when i get with java, the bianry it's the b'...' format.
How could it fixed for python3?

Support for HTTP transport.

Thoughts?

support TCompactProtocol

java server use TCompactProtocol, client use thriftpy default Protocol:

client error:

  File "/home/yangtao04/.virtualenvs/mc-callback/lib/python2.7/site-packages/thriftpy/transport/transport.py", line 167, in read
    self.read_frame()
  File "/home/yangtao04/.virtualenvs/mc-callback/lib/python2.7/site-packages/thriftpy/transport/transport.py", line 173, in read_frame
    self.__rbuf = BytesIO(self.__trans.read(sz))
  File "/home/yangtao04/.virtualenvs/mc-callback/lib/python2.7/site-packages/thriftpy/transport/socket.py", line 102, in read
    message='TSocket read 0 bytes')
thriftpy.transport.transport.TTransportException: TTransportException(message='TSocket read 0 bytes', type=4)

server_error:

2015-01-28 00:53:36,710 [WARN](Thread-14:AbstractNonblockingServer.java:internalRead(520)) - Got an IOException in internalRead!
java.io.IOException: Connection reset by peer

But use official thrift-py with TCompactProtocol is OK.

Docs do not reflect a working version of the pingpong demo

references to thrift.rpc ..

the pingpong eg in the master of this repo works fine

"const map<EnumType, ?>" fails to parse

The thrift def

enum Enum {
    A = 1
}

const map<Enum, string> ConstMap = {
  Enum.A: "hurr-durr"
}

fails to parse with

    def cast(self, data):
        dct = {}
        keys = data.keys()

        for key in keys:
>           dct[self[1].cast(key)] = self[2].cast(data[key])
E           AttributeError: 'str' object has no attribute 'cast'

../thriftpy/parser/model.py:141: AttributeError

Server closed unexpected

If client and server have different transport, exp, server side using TFramedTransportFactory, client using TBufferedTransportFactory, if client send buff to server, server will closed unexcepted, with no exception.

In fact, In thriftpy socket.py read() method, it will read 0 bytes and raise exception. But server should not closed.

new cython protocol

needs a new cython protocol with better performance.

this issue is for bountysource test

setup.py must specify exact versions of dependencies

The setup.py file does not specify exact versions ( ==0.0.X and so on) but only specifies minimum versions. This means a new release of one of those dependencies could break an installation of this project (due to breaking changes).

Build fails

thriftpy fails to build if cython isn't available:

$ git clone https://github.com/eleme/thriftpy
Cloning into 'thriftpy'...
remote: Counting objects: 2664, done.
remote: Total 2664 (delta 0), reused 0 (delta 0), pack-reused 2664
Receiving objects: 100% (2664/2664), 525.98 KiB | 913.00 KiB/s, done.
Resolving deltas: 100% (1724/1724), done.
Checking connectivity... done.
$ cd thriftpy/
$ git show
commit 5319ad230d29d2c56456022529239339c0f361eb
Merge: 193c752 3cedc9e
Author: Lx Yu <[email protected]>
Date:   Thu Jun 18 16:11:27 2015 +0800

    Merge pull request #140 from maralla/unix_socket

    fix unix socket in tests
$ python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/server.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/thrift.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/tornado.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/utils.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/rpc.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/hook.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/_compat.py -> build/lib.linux-x86_64-2.7/thriftpy
copying thriftpy/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy
creating build/lib.linux-x86_64-2.7/thriftpy/contrib
copying thriftpy/contrib/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/contrib
creating build/lib.linux-x86_64-2.7/thriftpy/protocol
copying thriftpy/protocol/binary.py -> build/lib.linux-x86_64-2.7/thriftpy/protocol
copying thriftpy/protocol/json.py -> build/lib.linux-x86_64-2.7/thriftpy/protocol
copying thriftpy/protocol/multiplex.py -> build/lib.linux-x86_64-2.7/thriftpy/protocol
copying thriftpy/protocol/exc.py -> build/lib.linux-x86_64-2.7/thriftpy/protocol
copying thriftpy/protocol/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/protocol
creating build/lib.linux-x86_64-2.7/thriftpy/transport
copying thriftpy/transport/socket.py -> build/lib.linux-x86_64-2.7/thriftpy/transport
copying thriftpy/transport/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/transport
creating build/lib.linux-x86_64-2.7/thriftpy/parser
copying thriftpy/parser/lexer.py -> build/lib.linux-x86_64-2.7/thriftpy/parser
copying thriftpy/parser/parser.py -> build/lib.linux-x86_64-2.7/thriftpy/parser
copying thriftpy/parser/exc.py -> build/lib.linux-x86_64-2.7/thriftpy/parser
copying thriftpy/parser/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/parser
creating build/lib.linux-x86_64-2.7/thriftpy/contrib/tracking
copying thriftpy/contrib/tracking/tracker.py -> build/lib.linux-x86_64-2.7/thriftpy/contrib/tracking
copying thriftpy/contrib/tracking/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/contrib/tracking
creating build/lib.linux-x86_64-2.7/thriftpy/transport/memory
copying thriftpy/transport/memory/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/transport/memory
creating build/lib.linux-x86_64-2.7/thriftpy/transport/buffered
copying thriftpy/transport/buffered/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/transport/buffered
creating build/lib.linux-x86_64-2.7/thriftpy/transport/framed
copying thriftpy/transport/framed/__init__.py -> build/lib.linux-x86_64-2.7/thriftpy/transport/framed
copying thriftpy/contrib/tracking/tracking.thrift -> build/lib.linux-x86_64-2.7/thriftpy/contrib/tracking
running build_ext
building 'thriftpy.transport.cybase' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/thriftpy
creating build/temp.linux-x86_64-2.7/thriftpy/transport
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -I/usr/kerberos/include -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c thriftpy/transport/cybase.c -o build/temp.linux-x86_64-2.7/thriftpy/transport/cybase.o
gcc: thriftpy/transport/cybase.c: No such file or directory
gcc: no input files
error: command 'gcc' failed with exit status 1