kashifrazzaqui/json-streamer

A fast streaming JSON parser for Python that generates SAX-like events using yajl

json-streamer's Issues

Partially parsing large items under dictionary key

I have a large JSON that has a lot of items under certain dictionary key.
E.g.

{
"images": ["image1.jpg", "image2.jpg"],
"annotations: ["dog", "cat"]
}

The number of images and annotations values won't fit in memory. I'd like to iterate over them without loading all at once.
But it seems this library only loads a single key? E.g. the pair event is triggered once all "images" list is consumed.
Is it possible to use this library for a case like this?

Want to split a 22M JSON file into smaller files to track a problem

I have a large JSON file that has an error somewhere. I want to split the up the JSON file into smaller files that are also JSON so that I can find out where the error is. Possible with your package ?

ModuleNotFoundError: No module named 'jsonstreamer.yajl'

Can somebody help me?

pypi version ahead of master branch

Please update the PyPI entry of json-streamer
https://pypi.python.org/pypi/jsonstreamer/1.3.6
and consider linking there from the short text description here.

Trouble using 'jsonstreamer` with 'yajl' on Windows 10

Hey @kashifrazzaqui

I have been trying to use your library json-streamer for implementing a Streaming API..

As directed, I have installed yajl on my Windows 10 system and installed it as below:

C:\Users\mianand\Downloads\lloyd-yajl-2.1.0-0-ga0ecdde\lloyd-yajl-66cb08c\build>nmake install

Microsoft (R) Program Maintenance Utility Version 14.00.24210.0
Copyright (C) Microsoft Corporation. All rights reserved.

[ 30%] Built target yajl_s
[ 60%] Built target yajl
[ 66%] Built target yajl_test
[ 72%] Built target gen-extra-close
[ 78%] Built target json_reformat
[ 84%] Built target json_verify
[ 90%] Built target parse_config
[100%] Built target perftest
Install the project...
-- Install configuration: "Release"
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/lib/yajl.lib
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/lib/yajl.dll
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/lib/yajl_s.lib
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/include/yajl/yajl_parse.h
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/include/yajl/yajl_gen.h
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/include/yajl/yajl_common.h
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/include/yajl/yajl_tree.h
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/include/yajl/yajl_version.h
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/share/pkgconfig/yajl.pc
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/bin/json_reformat.exe
-- Up-to-date: C:/Program Files (x86)/YetAnotherJSONParser/bin/json_verify.exe

Still, on running the conda with python 3.6 gives me the following :

from jsonstreamer import JSONStreamer
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\mianand\AppData\Local\Continuum\anaconda3\envs\pycharm_venv\lib\site-packages\jsonstreamer_init_.py", line 9, in
from jsonstreamer.jsonstreamer import JSONStreamer, ObjectStreamer
File "C:\Users\mianand\AppData\Local\Continuum\anaconda3\envs\pycharm_venv\lib\site-packages\jsonstreamer\jsonstreamer.py", line 14, in
from .yajl.parse import YajlParser, YajlListener, YajlError
File "C:\Users\mianand\AppData\Local\Continuum\anaconda3\envs\pycharm_venv\lib\site-packages\jsonstreamer\yajl\parse.py", line 32, in
yajl = load_lib()
File "C:\Users\mianand\AppData\Local\Continuum\anaconda3\envs\pycharm_venv\lib\site-packages\jsonstreamer\yajl\parse.py", line 29, in load_lib
raise OSError('Yajl cannot be found.')
OSError: Yajl cannot be found.

Any pointers on this one ?
Help appreciated.

Not looking for yajl.dll when loading Yajl

In the method load_lib(), there is never an attempt to load Yajl from yajl.dll, which is the name of Yajl on windows. I think it would be rather easy to add this, and make this package useful on Windows as well.

Missing tests & tags

PyPI has 1.3.6 , and no tests.

GitHub only has a tag for v1.0.0 , so I cant use that.

Could you tag v1.3.6 in GitHub, so I can use it to get tests, and finish https://build.opensuse.org/package/show/home:jayvdb:py-new/python-jsonstreamer after kashifrazzaqui/again#8 is also fixed.

SyntaxError: invalid syntax

Traceback (most recent call last):
File "test_jsonstreamer.py", line 3, in
from jsonstreamer import JSONStreamer
File "/usr/local/lib/python2.7/dist-packages/jsonstreamer/init.py", line 9, in
from jsonstreamer.jsonstreamer import JSONStreamer, ObjectStreamer
File "/usr/local/lib/python2.7/dist-packages/jsonstreamer/jsonstreamer.py", line 12, in
from again import events
File "/usr/local/lib/python2.7/dist-packages/again/init.py", line 4, in
from .events import EventSource, AsyncEventSource
File "/usr/local/lib/python2.7/dist-packages/again/events.py", line 49
yield from each(*args, **kwargs)
^
SyntaxError: invalid syntax
python --version
Python 2.7.3

Ensure exception __str__ methods return strings

Hi there,

Issues that throw JSONStreamerException classes are difficult to debug because there is no expectation that a str will be returned. This makes debugging a PITA.

awesome_module.py", line 51, in map_step
    url + '\n' + str(e))
TypeError: __str__ returned non-string (type bytes)

outdated pypi package

Hi,

Could you update the pypi package?
As far as I see, there were some commits since the last pypi upload.
Also, I think it is a bit confusing that there is one tagged release, which is 1.0,
while pypi package has 1.3.6 version number, but both of them almost a year
older than some important fixes, e.g. the exponential floats.
(I can install the file on my own, but I think it would be nice to update the releases.)

Trouble using 'jsonstreamer` with 'yajl-2' on Ubuntu 14.04

Hey @kashifrazzaqui

I have been trying to use your library json-streamer for implementing a Streaming API..

As directed, I have installed yajl on my Ubuntu 14.04 system and also verified it's presence and correct installation (refer: [1] & [2])

Still, on running the command python3 -m jsonstreamer.jsonstreamer < test.json i.e. using it with jsonstreamer gives me the following :

  File "/usr/local/lib/python3.4/dist-packages/jsonstreamer/yajl/parse.py", line 29, in load_lib
    raise OSError('Yajl cannot be found.')
OSError: Yajl cannot be found.

Following up in lloyd/yajl#190 it seems that there might be an issue in the parse.py file itself ? Maybe it's looking for yajl1 and not yajl2.

Any pointers on this one ?
Help appreciated.

---

[1] Running gcc -lyajl yields:

jigyasa@spin:~$ gcc -lyajl
....
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status

[2] And sudo ldconfig -p | grep yajl results in:

jigyasa@spin:~$ sudo ldconfig -p | grep yajl
    libyajl.so.2 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libyajl.so.2

consider adding support for context managers