Decoding from file-like object reads too much data (C extension only)

[iotic-labs]

Decoding from file-like object reads too much data (C extension only)

[vtermanis]

Initial solution - remove buffering completely

Overview

Read the exact requested character count from file-like objects read() method. This results in significantly more calls to said method (as e.g. type markers result in a one byte read) and leads to considerable performance degradation.

Potential alternative

If file-like object is seekable() (e.g. file on local filesystem), enable buffering, otherwise use no buffering (e.g. socket).

General testing

• full suite with added test for issue
• coverage_test.sh with compiled extension
• pympler reference leak test (see bottom of test.py)

Performance

Medium-size varied-content decoding

Steps

python3 -mubjson fromjson test/data/CouchDB4k.compact.json /tmp/CouchDB4k.compact.ubjson
python -mtimeit --n 50000 \
 -s "
from io import BytesIO
from ubjson import load, __version__
print(__version__)
raw = BytesIO()
with open('/tmp/CouchDB4k.compact.ubjson', 'rb') as f:
    raw.write(f.read())
"\
 'raw.seek(0); load(raw)'

Python3 output

Before

0.13.0
0.13.0
0.13.0
50000 loops, best of 3: 23.4 usec per loop

After

0.14.0
0.14.0
0.14.0
50000 loops, best of 3: 73.8 usec per loop

Python2 output

Before

0.13.0
0.13.0
0.13.0
50000 loops, best of 3: 25.5 usec per loop

After

0.14.0
0.14.0
0.14.0
50000 loops, best of 3: 92.5 usec per loop

Large file decoding (62MB with small fields)

python3 -mtimeit -r1 -n1 -s "
from ubjson import load, __version__
print(__version__)
" "
with open('DEFRA.uk_air.ubj', 'rb') as f:
    load(f, intern_object_keys=True)
"

Python3 output

Before

0.13.0
1 loops, best of 1: 2.08 sec per loop

After

0.14.0
1 loops, best of 1: 5.53 sec per loop

[vtermanis]

#11 addresses performance concerns by using three methods for reading input from:

1. Fixed single-dimension byte sequence (as before)
2. Buffered from a seek()-able file-like object (as before)
3. Unbuffered from a file-like object (new)

[vtermanis]

Fixed in 0.14.0.