mre/hyperjson

🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.

A hyper-fast, safe Python module to read and write JSON data. Works as a drop-in replacement for Python's built-in json module. This is alpha software and there will be bugs, so maybe don't deploy to production just yet. 😉

⚠️ NOTE

This project is not actively maintained. orjson is likely the better alternative.

Installation

pip install hyperjson

Usage

hyperjson is meant as a drop-in replacement for Python's json module:

>>> import hyperjson
>>> hyperjson.dumps([{"key": "value"}, 81, True])
'[{"key":"value"},81,true]'
>>> hyperjson.loads("""[{"key": "value"}, 81, true]""")
[{u'key': u'value'}, 81, True]

Motivation

Parsing JSON is a solved problem; so, no need to reinvent the wheel, right?
Well, unless you care about performance and safety.

Turns out, parsing JSON correctly is a hard problem. Thanks to Rust however, we can minimize the risk of running into stack overflows or segmentation faults however.

hyperjson is a thin wrapper around Rust's serde-json and pyo3. It is compatible with Python 3 (and 2 on a best-effort basis).

For a more in-depth discussion, watch the talk about this project recorded at the Rust Cologne Meetup in August 2018.

Goals

• Compatibility: Support the full feature-set of Python's json module.
• Safety: No segfaults, panics, or overflows.
• Performance: Significantly faster than json and as fast as ujson (both written in C).

Non-goals

• Support ujson and simplejson extensions:
  Custom extensions like encode(), __json__(), or toDict() are not supported. The reason is, that they go against PEP8 (e.g. dunder methods are restricted to the standard library, camelCase is not Pythonic) and are not available in Python's json module.
• Whitespace preservation: Whitespace in JSON strings is not preserved. Mainly because JSON is a whitespace-agnostic format and serde-json strips them out by default. In practice this should not be a problem, since your application must not depend on whitespace padding, but it's something to be aware of.

Benchmark

We are not fast yet. That said, we haven't done any big optimizations. In the long-term we might explore features of newer CPUs like multi-core and SIMD. That's one area other (C-based) JSON extensions haven't touched yet, because it might make code harder to debug and prone to race-conditions. In Rust, this is feasible due to crates like faster or rayon.

So there's a chance that the following measurements might improve soon.
If you want to help, check the instructions in the Development Environment section below.

Test machine:
MacBook Pro 15 inch, Mid 2015 (2,2 GHz Intel Core i7, 16 GB RAM) Darwin 17.6.18

Contributions welcome!

If you would like to hack on hyperjson, here's what needs to be done:

• Implement loads()
• Implement load()
• Implement dumps()
• Implement dump()
• Benchmark against json and ujson (see #1)
• Add a CI/CD pipeline for easier testing (see #2)
• Create a proper pip package from it, to make installing easier (see #3).
• Profile and optimize performance (see #16)
• Add remaining keyword-only arguments to methods

Just pick one of the open tickets. We can provide mentorship if you like. 😃

Developer guide

This project uses poetry for managing the development environment. If you don't have it installed, run

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
export PATH="$HOME/.poetry/bin:$PATH"

The project requires the nightly version of Rust.

Install it via rustup:

rustup install nightly

If you have already installed the nightly version, make sure it is up-to-date:

rustup update nightly

After that, you can compile the current version of hyperjson and execute all tests and benchmarks with the following commands:

make install
make test
make bench

🤫 Pssst!... run make help to learn more.

Drawing pretty diagrams

In order to recreate the benchmark histograms, you first need a few additional prerequisites:

• Matplotlib
• Numpy

On macOS, please also add the following to your ~/.matplotlib/matplotlibrc (reference):

backend: TkAgg

After that, run the following:

make plot

License

hyperjson is licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in hyperjson by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

hyperjson's Issues

Add python-rapidjson to benchmarks

python-rapidjson seems to be a serious contender for the fastest JSON Python module around.
The benchmarks look quite impressive. Therefore it might be useful to add it to our benchmarks.

Linking error with make build

make build and by extension cargo build, cargo test and cargo bench fail with the following error:

error: linking with `cc` failed: exit code: 1
..... A lot of output ....
undefined reference to `PyExc_TypeError'
collect2: error: ld returned 1 exit status

Perhaps the build command should be modified to match setuptools-rust one?

E.g.

cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module pyo3/python3 --release -- --crate-type cdylib

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Automate deployment to Pypi

We should automate the deployment process of hyperjson.
Since I had some great experiences with Github actions, I would prefer to write a completely new CI pipeline with it and remove Travis from the project.

The pipeline should...

• run the tests
• publish the Python package for Python 3.5, 3.6, 3.7, and optionally sdist with the help of maturin.
• (optionally) release to crates.io

If someone wants to tackle this, please go ahead. 😊

Speed up boolean encoding/decoding

From our benchmarks we can see that we are consistently slower than everyone else when serializing/deserializing boolean values. We should fix that.

orjson is using an unsafe block to create a reference to a boolean:
https://github.com/ijl/orjson/blob/03d55e99a953ce93cedc05f03e4b63b0bcbbcc7a/src/decode.rs#L81-L96

This avoids additional allocations.
For comparison, this is our code at the moment:

hyperjson/src/lib.rs

Lines 475 to 480 in ded13b4

	fn visit_bool<E>(self, value: bool) -> Result<Self::Value, E>
	where
	E: de::Error,
	{
	Ok(value.to_object(self.py))
	}

I wonder if we could achieve comparable performance without using unsafe.
@konstin, any idea? Maybe there was a recent development in pyo3 that we could leverage here?

[0.2.4] Low-level details in JSONDecodeError message by mistake?

Hi!

I'm happy to find that hyperjson rejects single surrogates as invalid characters but the exception text looks more low-level than expected. Is this a bug? Can the exception message be made more highlevel like "single surrogates not allowed"?

In [8]: hyperjson.__version__
Out[8]: '0.2.4'

In [9]: hyperjson.loads('"\\ud800"')
[..]
JSONDecodeError: Value: PyObject(0x7f481b3a1530), Error: PyErr { type: Py(0x7f48262225c0, PhantomData) }: line 1 column 1 (char 0)

Thanks and best, Sebastian

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv hyperjson-NkoziC6T-py3.8 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

[PackageNotFound]
Package hypothesis (5.24.4) not found.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Use smallvec to avoid heap allocations

We should use smallvec for serializing/deserializing as it has the ability to use the stack for small vector allocations.

Noticed that orjson is doing the same.

Zero-copy string deserialization

Over on Reddit, @mikeyhew mentioned that there might be an option to parse JSON strings without copying:

Just wanted to point out that serde-json isn't zero-copy because it will copy strings to turn escape sequences like "\n" and "" into the character they represent. To parse JSON without copying, you could make a custom string type, JsonStr, which is utf-8 like str but can contain escape sequences.

I forgot about that, but it's actually a great idea!
Here's the upstream discussion on serde-json.
We should give this custom string type some serious consideration, as string allocation takes a big part of the encoding/decoding process at the moment.

If anyone wants to give it a shot, go for it.

Add optional support for simdjson-rs behind a feature flag

simdjson-rs is a Rust port of the extremely fast simdjson JSON parser with serde compatibility. We could make use of that to improve serialization/deserialization performance.
This would have to be guarded behind an optional feature flag because it contains a lot of unsafe code.

Careful here, I don't know how much work this is.

few questions

1. Im working with the rust crowbar maintainers which currently depends on the cpython crate which is currently no longer actively maintained. we're starting to cast our eyes towards PyO3 as a potential alternative. the few things crowbar needs to work is a way to export a python initializer function and a way to bridge type systems python <=> rust. Currently crowbar is using a crate that goes from py objects to json via serde json and back again. When I started poking around at the PyO3 ecosystem you're crate turned up. Do you think hyperjson fills a similar bridge between types?

2. I love the graphs in your readme. How did you make them?

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv hyperjson-evFFnZXK-py3.9 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package orjson (3.4.1) not found.

  at /usr/local/.pyenv/versions/3.9.1/lib/python3.9/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Compare with orjson

orjson is another JSON encoder for Python written in Rust that looks quite performant from their benchmarks. We should add it to our comparisons as well. Probably we should also send them a PR to add hyperjson to their benchmarks.
Support welcome.

Error when installing via pip

Both pipenv install hyperjson and pip install hyperjson produce the following error:

  Could not find a version that satisfies the requirement hyperjson (from versions: )
No matching distribution found for hyperjson

Tried this on Ubuntu 18.10 with python 3.6.7

Use new benchmark framework in `README.md`

With #19 we now have solid benchmark support to build upon (thanks @RSabet!). We should add the new benchmarks to our README.md - preferably as some kind of graph or Markdown table. The easier it is to grasp the current status, the better.

hypers standard script will not install on macosbeta

xcode-select.pdf

In bash, it outputted an error that I did not have command line tools installed. I tried switching path to my developer directory, but it was a nogo. :/

Port Python benchmark to pytest-benchmark

Our current Python benchmarks are based on ujson's script here.
It's a bit clunky to use and not really idiomatic (redundant code, custom implementation for printing the results as a table).
In addition to that, it's hard to make code changes and see the performance impact, because the measurements fluctuate a lot from run to run.

A modern alternative is pytest-benchmark. Since we already use pytest, it's would fit in well with the rest of our testing pipeline.
It provides powerful output methods for different use-cases:

It can also plot histograms to visualize the error bars as a box-plot.

It would be nice to move to pytest-benchmark as we could compare the measurements after each change using Travis CI.

dumps doesn't support indent

As this library should work as "a drop-in replacement for Python's built-in json module" it should support the indent feature for dumps().

import hyperjson

data = {
    'key1': [1,2,3],
    'key2': {
        'a': '1',
        'b': 2
    }
}

text = hyperjson.dumps(data, sort_keys=True, indent=4)
print(text)

Running this code results in the error:

Traceback (most recent call last):
  File "hyperjson_test.py", line 11, in <module>
    text = hyperjson.dumps(data, sort_keys=True, indent=4)
TypeError: dumps() got an unexpected keyword argument: indent

'100 arrays dict' fails

using the benchmarks from rapidjson 100 arrays dict fails with hyperjson.

Find and fix possible performance bottlenecks

Yesterday I did some profiling using the setup described here.
The resulting callgrind file is attached. This can be opened with qcachegrind on Mac or kcachegrind on Linux.

callgrind.out.35583.zip

If you don't have any of those programs handy, I've added a screenshot for the two main bottlenecks that I can see. I'm not an expert, but it looks like we spend a lot of time allocating, converting, and dropping the BTreeMap, which will be converted to a dictionary and returned to Python in the end.

I guess we could save a lot of time by making this part more efficient. E.g. by copying less and instead working on references. Might be mistaken, though. Help and pull requests are very welcome.
😊

Release to PyPI with prebuilt

People should be able to use pip to install the wheel from PyPI.

Fix remaining unit tests

Here is a list of failing tests and their status:

============================ FAILURES =============================
________________ UltraJSONTests.testEncodeSymbols _________________

self = <test_ujson.UltraJSONTests testMethod=testEncodeSymbols>

    def testEncodeSymbols(self):
        s = '\u273f\u2661\u273f'  # ✿♡✿
        encoded = hyperjson.dumps(s)
        encoded_json = hyperjson.dumps(s)
>       self.assertEqual(len(encoded), len(s) * 6 + 2)  # 6 characters + quotes
E       AssertionError: 5 != 20

hyperjson/tests/test_ujson.py:229: AssertionError
_______________ UltraJSONTests.testEncodeUnicodeBMP _______________

self = <test_ujson.UltraJSONTests testMethod=testEncodeUnicodeBMP>

    def testEncodeUnicodeBMP(self):
        s = '\U0001f42e\U0001f42e\U0001F42D\U0001F42D'  # 🐮🐮🐭🐭
        encoded = hyperjson.dumps(s)
        encoded_json = hyperjson.dumps(s)

        if len(s) == 4:
>           self.assertEqual(len(encoded), len(s) * 12 + 2)
E           AssertionError: 6 != 50

hyperjson/tests/test_ujson.py:204: AssertionError
_____________ UltraJSONTests.test_ReadBadObjectSyntax _____________

self = <test_ujson.UltraJSONTests testMethod=test_ReadBadObjectSyntax>

    def test_ReadBadObjectSyntax(self):
        input = '{"age", 44}'
>       self.assertRaises(ValueError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "{\"age\", 44}", Error: expected `:` at line 1 column 7

hyperjson/tests/test_ujson.py:820: JSONDecodeError
_________ UltraJSONTests.test_WriteArrayOfSymbolsFromList _________

self = <test_ujson.UltraJSONTests testMethod=test_WriteArrayOfSymbolsFromList>

    def test_WriteArrayOfSymbolsFromList(self):
        self.assertEqual("[true, false, null]",
>                        hyperjson.dumps([True, False, None]))
E       AssertionError: '[true, false, null]' != '[true,false,null]'
E       - [true, false, null]
E       ?       -      -
E       + [true,false,null]

hyperjson/tests/test_ujson.py:846: AssertionError
________ UltraJSONTests.test_WriteArrayOfSymbolsFromTuple _________

self = <test_ujson.UltraJSONTests testMethod=test_WriteArrayOfSymbolsFromTuple>

    def test_WriteArrayOfSymbolsFromTuple(self):
        self.assertEqual("[true, false, null]",
>                        hyperjson.dumps((True, False, None)))
E       AssertionError: '[true, false, null]' != '[true,false,null]'
E       - [true, false, null]
E       ?       -      -
E       + [true,false,null]

hyperjson/tests/test_ujson.py:850: AssertionError
___________ UltraJSONTests.test_decodeArrayDepthTooBig ____________

self = <test_ujson.UltraJSONTests testMethod=test_decodeArrayDepthTooBig>

    def test_decodeArrayDepthTooBig(self):
        input = '[' * (1024 * 1024)
>       self.assertRaises(RecursionError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "[{{", Error: key must be a string at line 1 column 2

hyperjson/tests/test_ujson.py:397: JSONDecodeError
_______________________________ UltraJSONTests.test_decodeTrueBroken ________________________________

self = <test_ujson.UltraJSONTests testMethod=test_decodeTrueBroken>

    def test_decodeTrueBroken(self):
        input = "tru"
>       self.assertRaises(ValueError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "tru", Error: expected ident at line 1 column 3

hyperjson/tests/test_ujson.py:413: JSONDecodeError
_______________________ UltraJSONTests.test_decodeWithTrailingNonWhitespaces ________________________

self = <test_ujson.UltraJSONTests testMethod=test_decodeWithTrailingNonWhitespaces>

    def test_decodeWithTrailingNonWhitespaces(self):
        input = "{}\n\t a"
>       self.assertRaises(JSONDecodeError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "{}\n\t a", Error: trailing characters at line 2 column 3

hyperjson/tests/test_ujson.py:790: JSONDecodeError
__________________________________ UltraJSONTests.test_dumpToFile ___________________________________

self = <test_ujson.UltraJSONTests testMethod=test_dumpToFile>

    def test_dumpToFile(self):
        f = six.StringIO()
        hyperjson.dump([1, 2, 3], f)
>       self.assertEqual("[1, 2, 3]", f.getvalue())
E       AssertionError: '[1, 2, 3]' != '[1,2,3]'
E       - [1, 2, 3]
E       ?    -  -
E       + [1,2,3]

hyperjson/tests/test_ujson.py:556: AssertionError
_____________________________ UltraJSONTests.test_dumpToFileLikeObject ______________________________

self = <test_ujson.UltraJSONTests testMethod=test_dumpToFileLikeObject>

    def test_dumpToFileLikeObject(self):
        class filelike:
            def __init__(self):
                self.bytes = ''

            def write(self, bytes):
                self.bytes += bytes

        f = filelike()
        hyperjson.dump([1, 2, 3], f)
>       self.assertEqual("[1, 2, 3]", f.bytes)
E       AssertionError: '[1, 2, 3]' != '[1,2,3]'
E       - [1, 2, 3]
E       ?    -  -
E       + [1,2,3]

hyperjson/tests/test_ujson.py:568: AssertionError
_______________________ UltraJSONTests.test_encodeListLongUnsignedConversion ________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeListLongUnsignedConversion>

    def test_encodeListLongUnsignedConversion(self):
        input = [18446744073709551615,
                 18446744073709551615, 18446744073709551615]
        output = hyperjson.dumps(input)

>       self.assertEqual(input, hyperjson.loads(output))
E       AssertionError: Lists differ: [18446744073709551615, 18446744073709551615, 18446744073709551615] != [1.8446744073709552e+19, 1.8446744073709552e+19, 1.8446744073709552e+19]
E
E       First differing element 0:
E       18446744073709551615
E       1.8446744073709552e+19
E
E       - [18446744073709551615, 18446744073709551615, 18446744073709551615]
E       ?                   ^ ^^^^                ^ ^^^^                ^^^
E
E       + [1.8446744073709552e+19, 1.8446744073709552e+19, 1.8446744073709552e+19]
E       ?   +               +++ ^^^ ^               +++ ^^^ ^               +++ ^

hyperjson/tests/test_ujson.py:495: AssertionError
_________________________ UltraJSONTests.test_encodeLongUnsignedConversion __________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeLongUnsignedConversion>

    def test_encodeLongUnsignedConversion(self):
        input = 18446744073709551615
        output = hyperjson.dumps(input)

>       self.assertEqual(input, hyperjson.loads(output))
E       AssertionError: 18446744073709551615 != 1.8446744073709552e+19

hyperjson/tests/test_ujson.py:509: AssertionError
_______________________________ UltraJSONTests.test_encodeOrderedDict _______________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeOrderedDict>

    @unittest.skipIf(sys.version_info < (2, 7), "No Ordered dict in < 2.7")
    def test_encodeOrderedDict(self):
        from collections import OrderedDict
        input = OrderedDict([(1, 1), (0, 0), (8, 8), (2, 2)])
        self.assertEqual('{"1": 1, "0": 0, "8": 8, "2": 2}',
>                        hyperjson.dumps(input))
E       AssertionError: '{"1": 1, "0": 0, "8": 8, "2": 2}' != '{"0":0,"1":1,"2":2,"8":8}'
E       - {"1": 1, "0": 0, "8": 8, "2": 2}
E       + {"0":0,"1":1,"2":2,"8":8}

hyperjson/tests/test_ujson.py:369: AssertionError
___________________________________ UltraJSONTests.test_sortKeys ____________________________________

self = <test_ujson.UltraJSONTests testMethod=test_sortKeys>

    def test_sortKeys(self):
        data = {"a": 1, "c": 1, "b": 1, "e": 1, "f": 1, "d": 1}
        sortedKeys = hyperjson.dumps(data, sort_keys=True)
        self.assertEqual(
>           sortedKeys, '{"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}')
E       AssertionError: '{"a":1,"b":1,"c":1,"d":1,"e":1,"f":1}' != '{"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}'
E       - {"a":1,"b":1,"c":1,"d":1,"e":1,"f":1}
E       + {"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}
E       ?      +  +    +  +    +  +    +  +    +  +    +

hyperjson/tests/test_ujson.py:865: AssertionError
========================= 30 failed, 102 passed, 28 skipped in 9.44 seconds =========================

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv hyperjson-rd9crSLM-py3.9 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package orjson (3.3.1) not found.

  at /usr/local/.pyenv/versions/3.9.1/lib/python3.9/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Run fuzzing tools to discover safety holes

In a discussion on Reddit, user /u/Shnatsel suggested the use of fuzzing to test hyperjson for safety issues and panics. There are a few tools around, which could be helpful:

• cargo-fuzz — currently supports libfuzzer
• hongfuzz-rs — helped discover issues in crates such as hyper and h2
• afl.rs — discovered issues in rustc and serde
• Rust Fuzz Trophy Case — lists issues found in crates and the fuzzers used

(List assembled by @memoryruins taken from here)

If somebody wants to take a stab at it, please comment here with any questions.

Dependabot can't resolve your Python dependency files

Dependabot can't resolve your Python dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Creating virtualenv hyperjson-LZ9WZ9Vb-py3.9 in /home/dependabot/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

  PackageNotFound

  Package pytest (6.2.0) not found.

  at /usr/local/.pyenv/versions/3.9.1/lib/python3.9/site-packages/poetry/repositories/pool.py:144 in package
      140│                     self._packages.append(package)
      141│ 
      142│                     return package
      143│ 
    → 144│         raise PackageNotFound("Package {} ({}) not found.".format(name, version))
      145│ 
      146│     def find_packages(
      147│         self, dependency,
      148│     ):

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Make install doesn't work.

Hi,

I tried installing through sources. Here is the log

----This is DESKITOP----------------------------------
 dinesh@desktop (master *) /home/dinesh/phd/code/rust/hyperjson $  
|  Dell Laptop=> make install
rustup override set nightly
info: using existing install for 'nightly-x86_64-unknown-linux-gnu'
info: override toolchain for '/home/dinesh/phd/code/rust/hyperjson' set to 'nightly-x86_64-unknown-linux-gnu'

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.37.0-nightly (929b48ec9 2019-06-21)

pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /home/dinesh/phd/code/rust/hyperjson/Pipfile
Using /usr/bin/python3 (3.6.5) to create virtualenv...
⠧ Creating virtual environment...Using base prefix '/usr'
New python executable in /home/dinesh/.virtualenvs/hyperjson-XqQ5-qXf/bin/python3
Not overwriting existing python script /home/dinesh/.virtualenvs/hyperjson-XqQ5-qXf/bin/python (you must use /home/dinesh/.virtualenvs/hyperjson-XqQ5-qXf/bin/python3)
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/bin/python3

OK Successfully created virtual environment! 
Virtualenv location: /home/dinesh/.virtualenvs/hyperjson-XqQ5-qXf
Installing dependencies from Pipfile.lock (73f958)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 32/32 — 00:00:08
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
pipenv run pyo3-pack develop
Found pyo3 bindings
Found Python 3.6m at python
[==========================================================> ]  61/63  pyo3
error[E0432]: unresolved import `pyo3::types::exceptions`
  --> src/lib.rs:17:18
   |
17 | use pyo3::types::exceptions::TypeError as PyTypeError;
   |                  ^^^^^^^^^^ could not find `exceptions` in `types`


error[E0432]: unresolved import `pyo3::types::exceptions`
  --> src/lib.rs:18:18
   |
18 | use pyo3::types::exceptions::ValueError as PyValueError;
   |                  ^^^^^^^^^^ could not find `exceptions` in `types`


error[E0432]: unresolved import `pyo3::types::PyObjectRef`
  --> src/lib.rs:19:44
   |
19 | use pyo3::types::{PyDict, PyFloat, PyList, PyObjectRef, PyTuple};
   |                                            ^^^^^^^^^^^ no `PyObjectRef` in `types`


error[E0277]: can't compare `&pyo3::types::PyAny` with `pyo3::PyRef<'_, pyo3::types::PyAny>`
   --> src/lib.rs:340:28
    |
340 |                     if key == self.py.None().as_ref(self.py) {
    |                            ^^ no implementation for `&pyo3::types::PyAny == pyo3::PyRef<'_, pyo3::types::PyAny>`
    |
    = help: the trait `std::cmp::PartialEq<pyo3::PyRef<'_, pyo3::types::PyAny>>` is not implemented for `&pyo3::types::PyAny`


error: aborting due to 4 previous errors


Some errors have detailed explanations: E0277, E0432.

For more information about an error, try `rustc --explain E0277`.


Cargo build finished with "exit code: 101": `cargo rustc --message-format json --quiet --manifest-path Cargo.toml --features pyo3/python3 --lib --`
Failed to build a native library through cargo
Failed to build a native library through cargo
Makefile:24: recipe for target 'install' failed
make: *** [install] Error 1
 

I am running it on ubuntu 18.

Benchmark

We should test hyperjson with the benchmark data provided in gojay.
Thanks to @arnecls for the link.

Profiling support

Although we have a somewhat reasonable benchmark suite, we still need to take care of proper profiling support in order to find performance bottlenecks without guesswork (thanks to @dtolnay for the input).

As a first step, it would be great to have a few guidelines in the docs on how to do a proper profiling run from a local machine (Linux and hopefully Mac).
Next, we could also think about web-services for profiling Rust code. This way, we could integrate it into our CI and get profiling results on every pull request.

This issue is very much open for discussion and all suggestions are appreciated.

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

CI for build and test

hyperjson should have CI too, I know the most of the deails are in pyo3 or serde-json, but we still need to make sure the wrapper around them are passing the test cases.

And we can prebuilt the wheel by CI, support multiple platforms.