ethereum/pyrlp

The python RLP serialization library

pyrlp

A package for Recursive Length Prefix encoding and decoding

Read more in the documentation on ReadTheDocs. View the change log.

Quickstart

python -m pip install rlp

Developer Setup

If you would like to hack on pyrlp, please check out the Snake Charmers Tactical Manual for information on how we do:

• Testing
• Pull Requests
• Documentation

We use pre-commit to maintain consistent code style. Once installed, it will run automatically with every commit. You can also run it manually with make lint. If you need to make a commit that skips the pre-commit checks, you can do so with git commit --no-verify.

Development Environment Setup

You can set up your dev environment with:

git clone [email protected]:ethereum/pyrlp.git
cd pyrlp
virtualenv -p python3 venv
. venv/bin/activate
python -m pip install -e ".[dev]"
pre-commit install

Release setup

To release a new version:

make release bump=$$VERSION_PART_TO_BUMP$$

How to bumpversion

The version format for this repo is {major}.{minor}.{patch} for stable, and {major}.{minor}.{patch}-{stage}.{devnum} for unstable (stage can be alpha or beta).

To issue the next version in line, specify which part to bump, like make release bump=minor or make release bump=devnum. This is typically done from the main branch, except when releasing a beta (in which case the beta is released from main, and the previous stable branch is released from said branch).

If you are in a beta version, make release bump=stage will switch to a stable.

To issue an unstable version when the current version is stable, specify the new version explicitly, like make release bump="--new-version 4.0.0-alpha.1 devnum"

pyrlp's Issues

`rlp.decode` does not allow `memoryview` objects

What is wrong

Python docs include memoryview in the list of bytes-like objects:

https://docs.python.org/3/glossary.html#term-bytes-like-object

rlp.decode rejects a memoryview

How can it be fixed.

Include it in the allowed types.

encode differs from ethereumjs/rlp.encode

below is my py code:

info = ['hffpcointomoon', 256, 41, 'fractal.account', 10000000, '0x1', '0xf8649068666670636f696e746f6d6f6f6e31398e68666670636f696e746f6d6f6f6eb841049112dda5f665da48458d3be328feed0ab6a897afb1c2c28ac8fa0b856c4ea867f01883f066ebb2229e6d373419896bf64ca1417fa30d0c5b9c8f6d63c7d9684e80', 1, 0, 0]

info = rlp.encode(info)

for x in info:
    print(x)

i got this:
248 254 142 104 102 102 112 99 111 105 110 116 111 109 111 111 110 130 ....

below is my js code:

const {encode} = require('rlp')

let info = ['hffpcointomoon', 256, 41, 'fractal.account', 10000000, '0x1', '0xf8649068666670636f696e746f6d6f6f6e31398e68666670636f696e746f6d6f6f6eb841049112dda5f665da48458d3be328feed0ab6a897afb1c2c28ac8fa0b856c4ea867f01883f066ebb2229e6d373419896bf64ca1417fa30d0c5b9c8f6d63c7d9684e80', 1, 0, 0]

let enc = encode(info)

for(let v of enc){
     console.log(v)
}

i got this:
248 147 142 104 102 102 112 99 111 105 110 116 111 109 111 111 110 130 1 0 41 143 ...

you can see that the results are different. I hope anyone can give me some help.

below is the code address:

pyrlp
ethereumjs/rlp

Cut minior point release

Need to cut a minor point release with the latest bugfix from #103

Failed while import rlp 1.0.1 in python3.6

  File "utils.py", line 11, in <module>
    import rlp
  File "/home/c/.local/lib/python3.6/site-packages/rlp/__init__.py", line 1, in <module>
    from . import sedes  # noqa: F401
  File "/home/c/.local/lib/python3.6/site-packages/rlp/sedes/__init__.py", line 4, in <module>
    from .big_endian_int import BigEndianInt, big_endian_int  # noqa: F401
  File "/home/c/.local/lib/python3.6/site-packages/rlp/sedes/big_endian_int.py", line 1, in <module>
    from eth_utils import (
  File "/home/c/.local/lib/python3.6/site-packages/eth_utils/__init__.py", line 1, in <module>
    import pkg_resources
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 26, in <module>
    import zipfile
  File "/usr/lib/python3.6/zipfile.py", line 84, in <module>
    sizeEndCentDir = struct.calcsize(structEndArchive)
AttributeError: module 'struct' has no attribute 'calcsize'

Broken coveralls integration

The coveralls integration for this repository looks to be broken.

`make dist` is broken due to deprecated use of `pypandoc`

python setup.py sdist
/usr/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
  warnings.warn(
Traceback (most recent call last):
  File "/home/radek/pyrlp/setup.py", line 44, in <module>
    setup(
  File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 147, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 476, in __init__
    _Distribution.__init__(
  File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 280, in __init__
    self.finalize_options()
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 900, in finalize_options
    ep(self)
  File "/usr/lib/python3.10/site-packages/setuptools/dist.py", line 920, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/home/radek/pyrlp/.eggs/setuptools_markdown-0.4.1-py3.10.egg/setuptools_markdown.py", line 43, in long_description_markdown_filename
    output = pypandoc.convert(markdown_filename, 'rst', format='md')
AttributeError: module 'pypandoc' has no attribute 'convert'
make: *** [Makefile:70: dist] Error 1

I think the error message is pretty self-explanatory. pypandoc no longer exports convert, this issue is quite common among different ETH-related packages...

Migrate to Circle CI

We should probably migrate to Circle CI, as Travis CI is kind of slow and also for the reason that
circle ci is widely used among the other repositories.

Library is not flexible enough to handle Parity's custom compression

Parity compresses and decompresses the rlp objects which it writes into the database. It has a hand-selected list of substitutions which it applies to strings (not lists). Attempting to rlp.decode one of the compressed objects results in rlp.exceptions.DecodingErrors. This is a good thing, because the compressed objects aren't valid rlp!

However, because of the way in which pyrlp is written it is very difficult to use pyrlp for reading these objects. consume_payment tries to recurse into rlp objects and doesn't provide any extension points for swapping out \x81\x02 (an example of an invalid sequence which appears in the compressed object) with the decompressed value. consume_length_prefix isn't usable either because it throws an exception on these objects, in order to read these objects I needed to implement my own.

I think this could be fixed by moving some exceptions around and breaking up a few functions such that users could consume just the logic they needed.

Drop python 2 support.

Time to drop python 2 support (like has been done in the other python ethereum libraries)

rlp decode cant support decode EIP1559 rawtransaction

import rlp
from eth_typing import HexStr
from eth_utils import to_bytes

def hex_to_bytes(data: str) -> bytes:
    return to_bytes(hexstr=HexStr(data))


signed_transaction = '0x02f86b0580847735940085012a05f2008252089498b6ded3b7f2268e2ff997ff9371fdb11945838e7b80c080a0ea398ca69a80cdc2f11106717b1168db06503759c37e2e2fbb525d474fe4750ca00568d8b50e6e162cae79535541c367e05e80122c4f3647dea0aeff19995cd9a0'

tx = rlp.decode(hex_to_bytes(signed_transaction))

Error info

rlp.exceptions.DecodingError: RLP string ends with 109 superfluous bytes

But on https://rawtxdecode.in/

decode result is right

No caching of RLP when decoding a List of Serializable items

When we decode a list of Serializable items, the individual items should get their _cached_rlp set, just like when we decode an individual Serializable

Inconsistent encoding of 0 in int_to_big_endian

Python 3 and python 2.7 implementations encode 0 in diffent ways:

Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import rlp
>>> rlp.utils.int_to_big_endian(0)
b''
>>> 
>python2
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rlp
>>> rlp.utils.int_to_big_endian(0)
'\x00'

I think, the python3 encoding is the right one. 0 should be encoded by a zero length bytestring.

Remove support for non-bytes inputs

Proposal to remove support for non-bytes input values for the RLP API.

Motivation

The RLP API is a low level tool for encoding/decoding. Being a low level tool it seems acceptable for the tool to only support inputs of a specific format.

By supporting both text and bytes inputs the RPL library adds overhead for the checking and conversion step. Python ethereum clients will benefit from the removal of this overhead, especially in situations where lots of reads and writes are happening in the trie database.

Other examples of this

The json library from the stdlib only accepts text (unicode) inputs.

Proposed way to implement this.

1. On all public APIs, rlp.encode, rlp.decode, Serializable.deserialize, rlp.lazy_decode begin issuing DeprecationWarning warnings.
2. After some period of time, replace these deprecation warnings and calls to str_to_bytes with something akin to the following.

if not isinstance(value, bytes):
    raise TypeError("Must be a bytes-type value")   # error message would be more specific for each case.

I suggest giving a 👍 / 👎 to the main issue to gauge support.

Error importing library

Hi i have the following error while importing your library. I use python 2.7

  File "/usr/lib/python2.7/site-packages/idex/client.py", line 10, in <module>
    from ethereum.utils import sha3, ecsign, encode_int32
  File "/usr/lib/python2.7/site-packages/ethereum-2.3.1-py2.7.egg/ethereum/__init__.py", line 11, in <module>
    from . import slogging  # noqa
  File "/usr/lib/python2.7/site-packages/ethereum-2.3.1-py2.7.egg/ethereum/slogging.py", line 6, in <module>
    from ethereum.utils import bcolors, is_numeric
  File "/usr/lib/python2.7/site-packages/ethereum-2.3.1-py2.7.egg/ethereum/utils.py", line 11, in <module>
    import rlp
  File "/usr/lib/python2.7/site-packages/rlp/__init__.py", line 1, in <module>
    from . import sedes  # noqa: F401
  File "/usr/lib/python2.7/site-packages/rlp/sedes/__init__.py", line 1, in <module>
    from . import raw  # noqa: F401
  File "/usr/lib/python2.7/site-packages/rlp/sedes/raw.py", line 9, in <module>
    from rlp.atomic import Atomic
  File "/usr/lib/python2.7/site-packages/rlp/atomic.py", line 4
    class Atomic(metaclass=abc.ABCMeta):
                          ^
SyntaxError: invalid syntax

Version 1.0 breaks functionality in pyethereum

Hello,

The new version breaks an import in pyethereum. Bellow I have attached the traceback.

     from ethereum.utils import check_checksum
   File "/usr/local/lib/python3.6/site-packages/ethereum/__init__.py", line 11, in <module>
     from . import slogging  # noqa
   File "/usr/local/lib/python3.6/site-packages/ethereum/slogging.py", line 6, in <module>
     from ethereum.utils import bcolors, is_numeric
   File "/usr/local/lib/python3.6/site-packages/ethereum/utils.py", line 13, in <module>
     from rlp.utils import decode_hex, encode_hex, ascii_chr, str_to_bytes
 ImportError: cannot import name 'decode_hex'

Live docs link to very old version of docs

Hi,

I found on the readthedocs site:
https://pyrlp.readthedocs.io/en/latest/tutorial.html

>>> from rlp.text import text
>>> encode(u'Ðapp')
'\x85\xc3\x90app'
>>> decode('\x85\xc3\x90app', text)
u'\xd0app'
>>> print decode('\x85\xc3\x90app', text)
Ðapp

Which is wrong by from rlp.text import text,

but instead in the source code:
https://github.com/ethereum/pyrlp/blob/master/docs/tutorial.rst

from rlp.sedes import binary

Maybe the document version is not updated yet with the source code on github?

decode_hex fails for Python3

From utils_py3.py:

def decode_hex(s):
    if isinstance(s, str):
        return bytes.fromhex(s)
    if isinstance(b, (bytes, bytearray)):
        return binascii.unhexlify(s)
    raise TypeError('Value must be an instance of str or bytes')

variable b doesn't seem to be defined which in fact makes that method not work in Python 3.6.1.

Cache rlp in deserialized Serializable

We are heavily using rlp encoding of instances for generating their hashes.

If we received and deserialized them, they can considered to be not mutable, therefore we could cache the originating rlp for fast hash computation.

As a safeguard, deserialized instances could have setter properties for the fields which would raise NotImplementedError.

LazyList.__getitem__ assumes i is a number but it can be a slice

LazyList's getitem implementation assumes i is always a number, but it can be a slice as well

The fix is trivial, but I want to investigate it further because it's not clear to me how that deserialize() call would cause getitem to raise an IndexError, since it checks the list's length before accessing the first element

Here's an example traceback I got from running the hive consensus simulator against pyethapp

TypeErrorTraceback (most recent call last)
<ipython-input-1-59579d93b561> in <module>()
----> 1 from importblock import Importer; Importer(eth).run()

/importblock.py in run(self)
     22             data = open('/blocks/' + block, 'r').read()
     23             block_data = rlp.decode_lazy(data)
---> 24             header = BlockHeader.deserialize(block_data[0])
     25             transactions = rlp.sedes.CountableList(Transaction).deserialize(block_data[1])
     26             uncles = rlp.sedes.CountableList(BlockHeader).deserialize(block_data[2])

/usr/lib/python2.7/site-packages/rlp-0.5.1-py2.7.egg/rlp/sedes/lists.pyc in deserialize(cls, serial, exclude, mutable, **kwargs)
    243     def deserialize(cls, serial, exclude=None, mutable=False, **kwargs):
    244         try:
--> 245             values = cls.get_sedes().deserialize(serial)
    246         except ListDeserializationError as e:
    247             raise ObjectDeserializationError(serial=serial, sedes=cls, list_exception=e)

/usr/lib/python2.7/site-packages/rlp-0.5.1-py2.7.egg/rlp/sedes/lists.pyc in deserialize(self, serial)
     81             if not (sedes_consumed or elements_consumed):
     82                 try:
---> 83                     result.append(sedes.deserialize(element))
     84                 except DeserializationError as e:
     85                     raise ListDeserializationError(serial=serial, element_exception=e, index=index)

/usr/lib/python2.7/site-packages/rlp-0.5.1-py2.7.egg/rlp/sedes/big_endian_int.pyc in deserialize(self, serial)
     36             raise DeserializationError('Invalid serialization (wrong size)',
     37                                        serial)
---> 38         if self.l is None and len(serial) > 0 and serial[0:1] == ascii_chr(0):
     39             raise DeserializationError('Invalid serialization (not minimal '
     40                                        'length)', serial)

/usr/lib/python2.7/site-packages/rlp-0.5.1-py2.7.egg/rlp/lazy.pyc in __getitem__(self, i)
    106         except StopIteration:
    107             assert self.index == self.end
--> 108             raise IndexError('Index %d out of range' % i)
    109         return self._elements[i]
    110 

New PyPI release

PyEthApp's json-rpc tests now depend on the changes in 2ee3201.

We should upload a new release to PyPI.

Track performance benchmarks

Benchmark results:

v1.0.0 release

Block serializations / sec: 1802.48
Block deserializations / sec: 1465.40
TX serializations / sec: 23371.21
TX deserializations / sec: 20408.96

Master @ f30fe1e

Block serializations / sec: 1891.86
Block deserializations / sec: 765.90
TX serializations / sec: 25249.98
TX deserializations / sec: 13829.94

After #57 @ cedba03

Block serializations / sec: 1869.49
Block deserializations / sec: 754.00
TX serializations / sec: 24482.09
TX deserializations / sec: 13933.05

v0.6.0 (691d1e1)

Block serializations / sec: 1584.33
Block deserializations / sec: 638.20
TX serializations / sec: 20081.98
TX deserializations / sec: 11416.72

After removal of python 2 support (d6af5a5)

Block serializations / sec: 1555.26
Block deserializations / sec: 663.43
TX serializations / sec: 21432.37
TX deserializations / sec: 10828.38

After removal of rlp.utils and conversion to use eth-utils (7ee1abf)

Block serializations / sec: 1795.98
Block deserializations / sec: 742.01
TX serializations / sec: 24321.49
TX deserializations / sec: 13368.13

TypeError: Did not find sedes handling type str

Python 3.6.5 on Mac

I'm looking thought the code to see if I can find the problem but Python isn't my strongest language.

I'm going through this tutorial:
https://ocalog.com/post/10/

error occurs on encoding '\x03' (version of message) which are interpreted as a string and not a byte

RLP fields as proper object attributes.

What is the problem

RLP objects are not very good at inheritance.

class MyBaseObject(rlp.Serializable):
    fields = (
        ('field_a', sedes.Binary),
    )

class InheritsFromBase(MyBaseObject):
    fields = (
        ('field_a', sedes.Binary),  # not DRY, repeated from base class
        ('another_field', sedes.Binary),
    )

There is no good pythonic way to extend a base class's fields.

How can we fix it.

Metaclass wizardry!

Much the same way that django models perform magical metaclass voodoo, we can do the same. Implement a metaclass for use that finds all attributes which are valid sedes types and does all the fancy stuff that happens within the __init__ method of Serializable. In general, doing away with Serializable.fields seems like a nice cleanup.

The example above would look something like:

class MyBaseObject(rlp.Serializable):
    field_a = sedes.Binary()

class InheritsFromBase(MyBaseObject):
    field_a = sedes.Binary()  # required to maintain explicit field ordering.
    another_field = sedes.Binary()

Implementation specifics.

Field class

We need a Field class which we can use to define RLP fields. This should be a thin wrapper of sorts around a rlp.sedes object. Ideally, we can have some sort of thin shortcut for converting a sedes object to a Field object using something like a classmethod Field.from_sedes(...)

Metaclass

Unless an alternate solution is presented, the rlp.Serializable class needs to be converted to use a metaclass. This metaclass will need to introspect the current fields as well as all of the fields for the base class to construct the class.

I suggest this part be done in two steps.

• Step 1

The first step would be to have the metaclass construct an rlp.Serializable instance with the proper fields property, and to allow the existing internals of rlp.Serializable to populate the actual attribute setters and getters. Under this approach accessing the field properties from the class would likely result in an AttributeError as they would not be constructed until class instantiation.

• Step 2

The second step is to invert this, and have the metaclass (or the underlying Field class) create these attributes at the class level, likely using descriptors (which the Field class could implement). Then, the fields property on the class would only be present for backwards compatibility.

Field ordering

The field ordering for RLP objects is important. Luckily, this is easy to do using metaclasses.

https://docs.python.org/3/reference/datamodel.html#metaclass-example

However, this gets complicated in cases of inheritance. From the discussion on this issue, we've decided that the best solution to fixing this is to required that either all fields or zero field be overridden when a class is sub classed.

Documentation

The documentation will need to be updated to reflect this new API.

rlp.decode_lazy does not work wit bytearray

ipdb> rlp.decode_lazy(bytearray('\0'))
*** TypeError: ord() expected string of length 1, but int found
ipdb> rlp.decode_lazy('\0')
'\x00'

API refactor for rlp.Serializable

I've some thoughts on how we might change rlp.Serializable.

Current features

• immutable vs. mutable objects (make_mutable, make_immutable).
• attribute access to fields
• instantiation with either *args or **kwargs
• cls.exclude API for creating classes with a subset of fields

Inefficiencies

• mutability checks at runtime
• field assignment done exclusively through __setattr__ (60% faster to use direct attribute assignment)
• obj.fields is non-ideal since it disallows the use of any field named fields

Solution

• use collections.namedtuple for immutable versions of the object. Fields are naturally ordered, and immutability is enforced by default.
• use a dynamically constructed class for mutable versions of the object.

rlp1.0 don't have exclude function.

I'm trying to patch pyethereum to use rlp 1.0, but it doesn't have Serializable.exclude method anymore. It's used like this:

    @property
    def mining_hash(self):
        return utils.sha3(rlp.encode(
            self, BlockHeader.exclude(['mixhash', 'nonce'])))

Instantiating empty Serializable fails

Example:

class Test(rlp.Serializable):
    pass

Test()

Exception:

~/.pyenv/versions/3.6.5/envs/trinity/lib/python3.6/site-packages/rlp/sedes/serializable.py in __init__(self, *args, **kwargs)
    194             field_values = args
    195 
--> 196         if len(field_values) != len(self._meta.field_names):
    197             raise TypeError(
    198                 'Argument count mismatch. expected {0} - got {1} - missing {2}'.format(

AttributeError: 'X' object has no attribute '_meta'

Cannot (de)serialize objects with optional fields

Parity uses a custom BlockDetails object with a very weird last field.

is_finalized is a boolean field with the following representation:

• A value of True is represented as a True boolean field: len(BlockDetails) == 5
• A value of False is represented as nothing: len(BlockDetails) == 4

Presumably they wanted to save a little space in the database by simply not saving this field.

I don't think it's possible to create a subclass of rlp.Serializable which handles this use-case.

Defining a Serializable with an empty field set fails

Example:

class Test(rlp.Serializable):
    fields = ()

Exception:

~/.pyenv/versions/3.6.5/envs/trinity/lib/python3.6/site-packages/rlp/sedes/serializable.py in __new__(cls, name, bases, attrs)
    391 
    392         # split the fields into names and sedes
--> 393         field_names, sedes = zip(*fields)
    394 
    395         # check that field names are unique

ValueError: not enough values to unpack (expected 2, got 0)

pip install --no-binary :all: rlp

What happened?

The command pip install --no-binary :all: rlp fails with error.

Code that produced the error

pip install --no-binary :all: rlp
Collecting rlp
  Using cached rlp-3.0.0.tar.gz (40 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [38 lines of output]
      long_description_markdown_filename: dist = <setuptools.dist.Distribution object at 0x103f75c50>; attr = 'long_description_markdown_filename'; value = 'README.md'
      Traceback (most recent call last):
        File "/Users/vrypan/Downloads/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/vrypan/Downloads/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/vrypan/Downloads/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 396, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 44, in <module>
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 147, in setup
          _setup_distribution = dist = klass(attrs)
                                       ^^^^^^^^^^^^
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 303, in __init__
          _Distribution.__init__(self, dist_attrs)
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
          self.finalize_options()
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 680, in finalize_options
          ep(self)
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 700, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/normal/lib/python3.11/site-packages/setuptools_markdown.py", line 38, in long_description_markdown_filename
          frame = _get_code_object()
                  ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/1x/s5qfp7sn3rj5qmx_nkz1nx9m0000gn/T/pip-build-env-g80p3wct/normal/lib/python3.11/site-packages/setuptools_markdown.py", line 58, in _get_code_object
          code = frame.f_back.f_code
                 ^^^^^^^^^^^^^^^^^^^
      AttributeError: 'NoneType' object has no attribute 'f_code'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Full error output

No response

Fill this section in if you know how this could or should be fixed

I think that the problem is actually in the README.md file:

Between "Snake" and "Charmers" there is a 0x0A (LF) character instead of 0x20 (SPACE).

#146 May fix the problem.

rlp Version

No response

Python Version

3.11.6

Operating System

osx

Output from pip freeze

No response

RLP StreamReader

What feature should we add?

While messages from a stream might normally need delineation, the length prefix of RLP makes delimiters redundant. Also, no delimiter can be valid for RLP. I am requesting a way to read complete rlp-encoded messages from a stream (eg sys.stdin).

Essentially it should work similar to asyncio.StreamReader.readline() but read an entire RLP message rather than a utf-8 line.

Usage

reader = rlp.StreamReader()
pipe_protocol = asyncio.StreamReaderProtocol(reader)
loop = asyncio.get_event_loop()
await loop.connect_read_pipe(
    lambda: pipe_protocol, sys.stdin
)
message = await reader.read()

Add flake8 linting to travis-ci

Lets add flake8 linting to travis-ci

rlp.utils.big_endian_to_int does not work with bytearray

Python 2.7.11 (default, Jun  2 2016, 22:26:50) 
[GCC 6.1.1 20160501] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rlp.utils
>>> rlp.utils.big_endian_to_int(b'\x1b\x99\xf7\xf8\xa4\x1b=\xc4\xaf91P\xa3\x8bD\xe8?\xd4n\xef\x8c\x97\r\xedgQ\xfd\x8e"+\xe5N')
12484485947356138258354564479024948173755300148939136490689561909622144230734L
>>> rlp.utils.big_endian_to_int(bytearray(b'\x1b\x99\xf7\xf8\xa4\x1b=\xc4\xaf91P\xa3\x8bD\xe8?\xd4n\xef\x8c\x97\r\xedgQ\xfd\x8e"+\xe5N'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/extra/work/brainbot/pyrlp/rlp/utils_py2.py", line 48, in big_endian_to_int
    return int(encode_hex(value), 16)
  File "/mnt/extra/work/brainbot/pyrlp/rlp/utils_py2.py", line 63, in encode_hex
    raise TypeError('Value must be an instance of str or unicode')
TypeError: Value must be an instance of str or unicode
>>> 

ImportError: cannot import name str_to_bytes

Hi,

I'm looking to use pyethapp so i've installed pyrlp.

When running pyethapp I face a RLP related error which trace is:
Traceback (most recent call last): File "/usr/local/bin/pyethapp", line 7, in <module> from pyethapp.app import app File "/usr/local/lib/python2.7/dist-packages/pyethapp/app.py", line 11, in <module> import ethereum.slogging as slogging File "/usr/local/lib/python2.7/dist-packages/ethereum-1.2.0-py2.7.egg/ethereum/slogging.py", line 4, in <module> from ethereum.utils import bcolors, isnumeric File "/usr/local/lib/python2.7/dist-packages/ethereum-1.2.0-py2.7.egg/ethereum/utils.py", line 9, in <module> import rlp File "/usr/local/lib/python2.7/dist-packages/rlp/__init__.py", line 1, in <module> from . import sedes File "/usr/local/lib/python2.7/dist-packages/rlp/sedes/__init__.py", line 2, in <module> from .binary import Binary, binary File "/usr/local/lib/python2.7/dist-packages/rlp/sedes/binary.py", line 3, in <module> from ..utils import Atomic, str_to_bytes, bytes_to_str ImportError: cannot import name str_to_bytes

Any ideas?

Serializable.get_sedes() breaks inheritance when child defines different fields

That method caches the sedes in the _sedes class variable, so if you call get_sedes() on a parent class, any childs of that class on which get_sedes() has not been called yet will end up with the parent's sedes no matter what is defined in their fields attribute.

class CustomSerializable(rlp.Serializable):
    pass

class A(rlp.Serializable):
    fields = [('foo', rlp.Serializable)]

class B(A):
    fields = [('foo', CustomSerializable)]

print(A.get_sedes())
print(B.get_sedes())

[<class 'rlp.sedes.lists.Serializable'>]
[<class 'rlp.sedes.lists.Serializable'>]

ImportError: cannot import name 'encode_hex'

from rlp.utils import encode_hex
ImportError: cannot import name 'encode_hex'

why？
is that don't exist？

Error with python2.7

Trying to make a FreeBSD port. Two problems:

• Build error:

byte-compiling /usr/ports/textproc/py-rlp/work/stage/usr/local/lib/python2.7/site-packages/rlp/utils_py3.py to utils_py3.pyc
  File "/usr/local/lib/python2.7/site-packages/rlp/utils_py3.py", line 6
    class Atomic(metaclass = abc.ABCMeta):
                           ^
SyntaxError: invalid syntax

• pydistutils produces the wrong plist:

===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: %%PYTHON_SITELIBDIR%%/rlp/utils_py3.pyc
Error: Missing: %%PYTHON_SITELIBDIR%%/rlp/utils_py3.pyo

Does the package forget to exclude utils_py3.py when built with python2?

Change to use `master` as primary branch.

What is wrong

py-rlp uses develop as it's primary development branch. This is inline with all of the old ethereum python codebases, but incongruent with all of the newer python codebases.

How can it be fixed

I propose we change the primary branch to be master.

Documented feature doesn't exist: rlp.sedes.sedes_list

The documentation goes over the process of inferring which sedes class to use when serializing an object, on this page. It mentions that the function rlp.infer_sedes checks rlp.sedes.sedes_list to find a sedes object which can handle serializing whatever you are trying to encode. But the code for infer_sedes never checks any list like this, and the list rlp.sedes.sedes_list doesn't even exist! Not sure why this discrepancy between the code and the docs exist, thought I'd make an issue to point it out though.

Use cython backend for speedup

1. Review @hyperevo 's prototype cython backend at https://github.com/Helios-Protocol/msgpack-rlp-python and its integration at https://github.com/Helios-Protocol/py-rlp-cython
2. If architecture is acceptable, open a PR to bring it upstream
3. Apply any fixups, refactors, etc
4. Make sure tests cover multiple back-ends
5. Can we do this without breaking compatibility? Maybe just a minor bump, and set the default backend to cython.

New `List` results in `RecursionError: maximum recursion depth exceeded in comparison`

>>> from rlp.sedes import List

>>> attrs = [
...       {
...         "trait_type": "Hat", 
...         "value": "Bayc Flipped Brim"
...       }, 
...       {
...         "trait_type": "Eyes", 
...         "value": "X Eyes"
...       },
...     ]

>>> traits = list() 
>>> for attribute in attrs:
...     properties = [attribute.get("trait_type"), attribute.get("value")]
...     trait = List(properties)
...     traits.append(trait)
... 

Results in:

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/cowboy/repos/loot.st/external-adapter/venv/lib/python3.9/site-packages/rlp/sedes/lists.py", line 57, in __init__
    self.append(List(e))
  File "/Users/cowboy/repos/loot.st/external-adapter/venv/lib/python3.9/site-packages/rlp/sedes/lists.py", line 57, in __init__
    self.append(List(e))
  File "/Users/cowboy/repos/loot.st/external-adapter/venv/lib/python3.9/site-packages/rlp/sedes/lists.py", line 57, in __init__
    self.append(List(e))
  [Previous line repeated 492 more times]
  File "/Users/cowboy/repos/loot.st/external-adapter/venv/lib/python3.9/site-packages/rlp/sedes/lists.py", line 56, in __init__
    elif isinstance(e, Sequence):
  File "/Users/cowboy/.brew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
  File "/Users/cowboy/.brew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
RecursionError: maximum recursion depth exceeded in comparison

Exception description

pyrlp/rlp/sedes/boolean.py

Line 28 in e39474d

'Invalid serialized boolean. Must be either 0x01 or 0x00',

Change Must be either 0x01 or 0x00 for Must be either 0x01 or ''

Add typing

mypy will be added to linting with the next release, but skipping all the important files. Start adding types once that's in.

How to disable the warning "Consider installing rusty-rlp to improve pyrlp"?

Please, tell me how to disable printing "Consider installing rusty-rlp to improve pyrlp" warning.
I was trying logging.getLogger("rlp").setLevel(logging.ERROR), but it doesn't help.