iotic-labs/py-ubjson

Universal Binary JSON draft-12 serializer for Python

py-ubjson's Issues

Please bump the version with 3.5 python support

To be available from pip.

Decoding from file-like object reads too much data (C extension only)

Problem

When decoding from a file-like object (using load() via C extension), up to BUFFER_FP_SIZE bytes are buffered when read()ing from it. This means that data is likely to be consumed past the end of the encoded ubjson block which is subsequently not available to either decode additional ubjson blocks or for other purposes.

Potential solutions

1. seek() back to recover unused read data.
   • File-like objects should not need to support seek()
2. Don't buffer at all and only read as much data as needed (often single bytes).
   • Underlying file-like object might have its own buffering already
   • Pure-Python version already does this

Test case

import ubjson
from io import BytesIO

# Only applies to C extension
assert ubjson.EXTENSION_ENABLED

sample_input = 'something to encode'
output = BytesIO()

# Produce output with multiple serialised ubjson "documents"
for _ in range(10):
    ubjson.dump(sample_input, output)

# Decode all of the documents
output.seek(0)
for i  in range(10):
    print(i)
    assert sample_input == ubjson.load(output)

Expected result

All 10 documents are decoded.

Actual result

0
1
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ubjson.decoder.DecoderException: Insufficient input (Type marker)

sys.stdin.buffer not always available - causes exception

This is triggered from our unit tests on CPython 3.5.1 (with the binary extension):

autobahn/wamp/test/test_serializer.py:34: in <module>
    from autobahn.wamp import serializer
autobahn/wamp/serializer.py:501: in <module>
    import ubjson
../../../cpy351_6/lib/python3.5/site-packages/ubjson/__init__.py:29: in <module>
    from .encoder import dump, dumpb, EncoderException  # noqa
ubjson/encoder.py:23: in init ubjson.encoder (ubjson/encoder.py3.c:6793)
    ???
ubjson/compat.py:64: in init ubjson.compat (ubjson/compat.py3.c:1403)
    ???
E   AttributeError: 'DontReadFromInput' object has no attribute 'buffer'

When I (manually) disable the extension (by commenting ext_modules in setup.py):

autobahn/wamp/serializer.py:501: in <module>
    import ubjson
../../../cpy351_6/lib/python3.5/site-packages/py_ubjson-0.8.3-py3.5.egg/ubjson/__init__.py:29: in <module>
    from .encoder import dump, dumpb, EncoderException  # noqa
../../../cpy351_6/lib/python3.5/site-packages/py_ubjson-0.8.3-py3.5.egg/ubjson/encoder.py:23: in <module>
    from .compat import Mapping, Sequence, INTEGER_TYPES, UNICODE_TYPE, TEXT_TYPES, BYTES_TYPES
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """Python v2.7 (NOT 2.6) compatibility"""

    # pylint: disable=unused-import,invalid-name,redefined-variable-type,wrong-import-position,no-name-in-module
    # pylint: disable=import-error


    from sys import stderr, stdout, stdin, version_info

    PY2 = (version_info[0] == 2)

    if PY2:
        # pylint:disable=undefined-variable
        INTEGER_TYPES = (int, long)  # noqa
        UNICODE_TYPE = unicode  # noqa
        TEXT_TYPES = (str, unicode)  # noqa
        BYTES_TYPES = (str,)

        STDIN_RAW = stdin
        STDOUT_RAW = stdout
        STDERR_RAW = stderr

    else:
        INTEGER_TYPES = (int,)
        UNICODE_TYPE = str
        TEXT_TYPES = (str,)
        BYTES_TYPES = (bytes, bytearray)

>       STDIN_RAW = stdin.buffer  # pylint: disable=no-member
E       AttributeError: 'DontReadFromInput' object has no attribute 'buffer'

../../../cpy351_6/lib/python3.5/site-packages/py_ubjson-0.8.3-py3.5.egg/ubjson/compat.py:64: AttributeError
================================================================= 3 failed, 123 passed, 1 error in 0.71 seconds ==================================================================
make: *** [test_asyncio] Fehler 1

0.11.0 fails floating type test on i686-linux-gnu

0.11 passes it's tests on most architectures, except in i686-linux-gnu:

======================================================================
FAIL: test_float (test.TestEncodeDecodeFp)

Traceback (most recent call last):
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 209, in test_float
no_float32=False)
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 87, in check_enc_dec
self.type_check(encoded[0], expected_type)
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 59, in type_check
self.assertEqual(actual, expected)
AssertionError: 'D' != 'd'

======================================================================
FAIL: test_float (test.TestEncodeDecodePlain)

Traceback (most recent call last):
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 209, in test_float
no_float32=False)
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 87, in check_enc_dec
self.type_check(encoded[0], expected_type)
File "/<>/.pybuild/cpython2_2.7_ubjson/build/test/test.py", line 59, in type_check
self.assertEqual(actual, expected)
AssertionError: 'D' != 'd'

---

Ran 53 tests in 3.112s

FAILED (failures=2)

complete build logs at
https://launchpad.net/ubuntu/+source/py-ubjson/0.11.0-0ubuntu1

src/encoder.c:271: _encode_PyDecimal: Assertion `PyBytes_Check(str)' failed.

The extension crashes while running the tests:

test_array (test.test.TestEncodeDecodeFpExt) ... python3.8: src/encoder.c:271: _encode_PyDecimal: Assertion `PyBytes_Check(str)' failed.
Aborted (core dumped)

To reproduce:

$ python setup.py build build_ext --inplace
$ python setup.py test

Support no-op ("N") in the decoder.

I believe supporting the no-op in the decoder (not the encoder) to be important.

I agree that in encoding/sending it should be in a lower layer. But in the decoder/receiver, the lower layer cannot tell which "N"s are a no-op versus part of a string or binary-encoded value without additional information or fully decoding the stream, which is the decoder's job.

(Also, this would also let me use "N" as a prefix to distinguish JSON and UBJSON-encoded data. :) )

DecoderException does not include offset (extension only)

DecoreException expects an optional parameter fp to be a file-like object with tell() function. In the C extension however the exception is initialised with the integer position in the stream. This means that currently exceptions raised via C extension never show stream offset.

Potential fixes:

1. Allow for DecoreException to accept either an integer or file-like object as argument
2. Update pure Python version to supply integer offset, change DecoderException to expect an integer rather than file-like object.

Issue in installation on window 10 with pip

Error: from ubjson import __version__ as version
ModuleNotFoundError: No module named 'ubjson'

I am installing crossbar its fail due to this error, experimenting whole a day with this minor issue. i just change ubjson to json in setup.py at line 37, then install correctly.
So resolve this issue in global repository that by pip work for rest of the world.

Understanding why my binary extension was not installed for a derived module (bjdata)

this is not a bug report, rather a question.

As I mentioned in a previous PR (#14), I've been working on a binary format derived from UBJSON, called BJData. BJData is designed to be an improved version of UBJSON with several notable changes (summarized here: nlohmann/json#3336 (comment)).

Most of the project files, including setup.py, were derived from your project, and I have been making releases via pypi at https://pypi.org/project/bjdata/.

One thing I noticed recently is that pip install py-ubjson can properly build/install the binary extension on both python3.8 and 3.6, but pip install bjdata only installs the binary extension on python 3.6. I don't know what was missing.

I want to get some suggestions on where to look. My setup.py can be found here. I previously blamed a pyproject.toml file that I added, but removing it does not change anything,

is this related to how the package was uploaded to pypi?

Here is the log for Python 3.8 on Ubuntu 20.04 (where ubjson module has binary but bjdata does not)

fangq@ubuntu20_04:~$ python3 -mpip install --upgrade bjdata
Collecting bjdata
  Using cached bjdata-0.3.3.tar.gz (62 kB)
Requirement already satisfied, skipping upgrade: numpy>=1.8.0 in ./.local/lib/python3.8/site-packages (from bjdata) (1.22.3)
Building wheels for collected packages: bjdata
  Building wheel for bjdata (setup.py) ... done
  Created wheel for bjdata: filename=bjdata-0.3.3-cp38-cp38-linux_x86_64.whl size=21413 sha256=9fd6168445b028c3e206b77556b1ba37696a22de0b8413ae76fe9bc3601c21ad
  Stored in directory: /home/users/fangq/.cache/pip/wheels/3f/08/4c/727407fa742f2c9affb45f384c8301464a003003cdd7709f67
Successfully built bjdata
Installing collected packages: bjdata
  Attempting uninstall: bjdata
    Found existing installation: bjdata 0.3.0
    Uninstalling bjdata-0.3.0:
      Successfully uninstalled bjdata-0.3.0
Successfully installed bjdata-0.3.3
#=======================================================================
fangq@ubuntu20_04:~$ python3 -mpip install --upgrade py-ubjson
Collecting py-ubjson
  Using cached py-ubjson-0.16.1.tar.gz (50 kB)
Building wheels for collected packages: py-ubjson
  Building wheel for py-ubjson (setup.py) ... done
  Created wheel for py-ubjson: filename=py_ubjson-0.16.1-cp38-cp38-linux_x86_64.whl size=152740 sha256=e24e4fff44e47da36179823174115a4861196d15d04cfa8b29b3c655d7989d5c
  Stored in directory: /home/users/fangq/.cache/pip/wheels/33/5a/a5/64272f6d5209c961c880545dc33eb1749dec17214e186e913a
Successfully built py-ubjson
Installing collected packages: py-ubjson
Successfully installed py-ubjson-0.16.1
#=======================================================================
fangq@ubuntu20_04:~/.local/lib/python3.8/site-packages$ lsls
total 484
drwxr-xr-x  2 fangq admin   4096 Apr 27 13:03 py_ubjson-0.16.1.dist-info
drwxr-xr-x  3 fangq admin   4096 Apr 27 13:03 ubjson
-rwxr-xr-x  1 fangq admin 465072 Apr 27 13:03 _ubjson.cpython-38-x86_64-linux-gnu.so  <- _ubjson exists, but not _bjdata
drwxr-xr-x  2 fangq admin   4096 Apr 27 13:02 bjdata-0.3.3.dist-info
drwxr-xr-x  3 fangq admin   4096 Apr 27 13:02 bjdata
...

Here is the log for Python 3.6 on Ubuntu 18.04 (both have binaries)

fangq@ubuntu18_04:~$ python3 -mpip install --upgrade bjdata
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: bjdata in /home/users/fangq/.local/lib/python3.6/site-packages (0.3.0)
Collecting bjdata
  Downloading bjdata-0.3.3.tar.gz (62 kB)
     |████████████████████████████████| 62 kB 952 kB/s             
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.8.0 in /home/users/fangq/.local/lib/python3.6/site-packages (from bjdata) (1.21.5)
Building wheels for collected packages: bjdata
  Building wheel for bjdata (setup.py) ... done
  Created wheel for bjdata: filename=bjdata-0.3.3-cp36-cp36m-linux_x86_64.whl size=132873 sha256=03437cb2568a9c410250e3cb51470be5b60d4dcfb24ea3e2e69806efb6fbf30e
  Stored in directory: /home/users/fangq/.cache/pip/wheels/df/b0/5c/268d37d5090f19678a54481572a83e5b83aa28da1191972d54
Successfully built bjdata
Installing collected packages: bjdata
  Attempting uninstall: bjdata
    Found existing installation: bjdata 0.3.0
    Uninstalling bjdata-0.3.0:
      Successfully uninstalled bjdata-0.3.0
Successfully installed bjdata-0.3.3
#=======================================================================
fangq@ubuntu18_04:~$ python3 -mpip install --upgrade py-ubjson
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: py-ubjson in /usr/lib/python3/dist-packages (0.8.5)
Collecting py-ubjson
  Downloading py-ubjson-0.16.1.tar.gz (50 kB)
     |████████████████████████████████| 50 kB 5.2 MB/s             
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: py-ubjson
  Building wheel for py-ubjson (setup.py) ... done
  Created wheel for py-ubjson: filename=py_ubjson-0.16.1-cp36-cp36m-linux_x86_64.whl size=115755 sha256=3df65b636d74cec86a1c815e13d7ea2ab324c70e7da5ae7b0d9365c56ac09a9c
  Stored in directory: /home/users/fangq/.cache/pip/wheels/45/9e/2b/6284eb7057635660d0f9eb410c25b045b0de0cfc40d5ce13a2
Successfully built py-ubjson
Installing collected packages: py-ubjson
Successfully installed py-ubjson-0.16.1
#=======================================================================
fangq@ubuntu18_04:~/.local/lib/python3.6/site-packages$ ls -lt
total 8232
drwxr-xr-x  2 fangq admin    4096 Apr 27 12:49 py_ubjson-0.16.1.dist-info
drwxr-xr-x  3 fangq admin    4096 Apr 27 12:49 ubjson
-rwxr-xr-x  1 fangq admin  281976 Apr 27 12:49 _ubjson.cpython-36m-x86_64-linux-gnu.so  <- both _ubjson and _bjdata are installed
drwxr-xr-x  2 fangq admin    4096 Apr 27 12:49 bjdata-0.3.3.dist-info
drwxr-xr-x  3 fangq admin    4096 Apr 27 12:49 bjdata
-rwxr-xr-x  1 fangq admin  326864 Apr 27 12:49 _bjdata.cpython-36m-x86_64-linux-gnu.so

Option to disable cython

Could we have an option to disable building of Cython extension modules? Like an env var UBJSON_NO_EXTENSION=1?

Test failures with Python 3.12

When running the test suite with Python 3.12.0, I'm getting two test failures:

======================================================================
FAIL: test_recursion (test.test.TestEncodeDecodeFpExt.test_recursion)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/py-ubjson/test/test.py", line 476, in test_recursion
    with self.assert_raises_regex(RuntimeError, 'recursion'):
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_recursion (test.test.TestEncodeDecodePlainExt.test_recursion)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/py-ubjson/test/test.py", line 476, in test_recursion
    with self.assert_raises_regex(RuntimeError, 'recursion'):
AssertionError: RuntimeError not raised