Giter Site home page Giter Site logo

lua-simdjson's People

Contributors

fouriertransformer avatar striezel avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

jdonaldson striezel-stash picodata meowboy326

lua-simdjson's Issues

Version 1.0

Version 1.0

https://ift.tt/3ncDaYq

Release 1.0.0 of the simdjson library builds on earlier pre-1.0 release that made the On Demand frontend our default. The On Demand front-end is a new way to build parser. With On Demand, if you open a file containing 1000 numbers and you need just one of these numbers, only one number is parsed. If you need to put the numbers into your own data structure, they are materialized there directly, without being first written to a temporary tree. Thus we expect that the simdjson On Demand might often provide superior performance, when you do not need to intermediate materialized view of a DOM tree. The On Demand front-end was primarily developed by @jkeiser.

If you adopted simdjson from an earlier version and relied on the DOM approach, it remains as always. Though On Demand is our new default, we remain committed to supporting the conventional DOM approach in the future, as there are instances where it is more appropriate.

Release 1.0.0 adds several key features:

  • In big data analytics, it is common to serialize large sets of records as multiple JSON documents separated by while spaces. You can now get the benefits of On Demand while parsing almost infinitely long streams of JSON records (see iterate_many). At each step, you have access to the current document, but a secondary thread indexes the following block. You can thus access enormous files while using a small amount of memory and achieve record-breaking speeds. Credit: @NicolasJiaxin.
  • In some cases, JSON documents contain numbers embedded within strings (e.g., "3.1416"). You can access these numbers directly using methods such as get_double_in_string(). Credit: @NicolasJiaxin
  • Given an On Demand instance (value, array, object, etc.), you can now convert it to a JSON string using the to_json_string method which returns a string view in the original document for unbeatable speeds. Credit: @NicolasJiaxin
  • The On Demand front-end now supports the JSON Pointer specification. You can request a specific value using a JSON Pointer within a large document. Credit: @NicolasJiaxin
  • Arrays in On Demand now have a count_elements() method. Objects have a count_fields() method. Arrays and objects have a reset method for when you need to iterate through them more than once. Document instances now have a rewind method in case you need to process the same document multiple times.

Other improvements include:

  • We have extended and improved our documentation and we have added much testing.
  • We have accelerated the JSON minification function (simdjson::minify) under ARM processors (credit @dougallj)

We encourage users of previous versions of the simdjson library to update. We encourage users to deploy it for production uses.



via Release notes from simdjson https://ift.tt/3eoyyad
September 07, 2021 at 04:55PM

Version 0.4.2

Version 0.4.2

https://ift.tt/2NEtDq2

Fix an installation bug whereas the header may not get installed when only the library was built (issue #1000)



via Release notes from simdjson https://ift.tt/3eoyyad
June 28, 2020 at 12:14PM

Version 2.0.2

Version 2.0.2

https://ift.tt/A9OKX6s

This is a patch release which fixes compilation errors for 2.0.1 under GCC 8 specifically. The path contains two "workarounds" specific to GCC 8.



via Release notes from simdjson https://ift.tt/pTRjaPH
June 02, 2022 at 08:21AM

Version 0.9.2

Version 0.9.2

https://ift.tt/3rHhkL5

This is a patch release which fixes a bug for users of the On Demand front-end. In some instances, when trying to access keys that are missing, the parser will fail with a generic error (TAPE_ERROR) under versions 0.9.0 and 0.9.1. Thanks to @jpalus for reporting the issue (#1521) and to @jkeiser for reviewing the patch.



via Release notes from simdjson https://ift.tt/3eoyyad
April 01, 2021 at 10:13AM

Version 0.9.6

Version 0.9.6

https://ift.tt/34SvdwT

This is a patch release fixing issue #1601 That is, users reusing the same document instance may get spurious errors in following parsing attempts because the "error" flag in the document is 'sticky'. This patch fixes the issue by modifying two lines of code.



via Release notes from simdjson https://ift.tt/3eoyyad
June 06, 2021 at 04:55PM

v2.2.3

v2.2.3

https://ift.tt/sz8ScTL

What's Changed

New Contributors

Full Changelog: v2.2.2...v2.2.3



via Release notes from simdjson https://ift.tt/6oOpng1
October 02, 2022 at 03:32PM

Version 2.0.0

Version 2.0.0

https://ift.tt/PLtrxom

Adding new icelake kernel with AVX-512 support. When the compiler and the processor has the adequate AVX-512 support, you might see a performance boost of between 25% to 40% on several tasks compared to the best previous kernel (haswell).

We rely on the fact that AVX-512 instructions no longer produce systematic frequency throttling on recent Intel processors (Ice Lake), https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html

Credit: Fangzheng Zhang and Weiqiang Wan (both from Intel) with indirect contributions by Kim Walisch and Jatin Bhateja.



via Release notes from simdjson https://ift.tt/pMH5nOa
May 25, 2022 at 10:49AM

Version 0.5.0

Version 0.5.0

https://ift.tt/3hjQkgs

Performance

  • Faster and simpler UTF-8 validation with the lookup4 algorithm #993
  • We improved the performance of simdjson under Visual Studio by about 25%. Users will still get better performance with clang-cl (+30%) but the gap has been reduced. #1031

Code usability

  • In parse_many, when parsing streams of JSON documetns, we give to the users runtime control as to whether threads are used (via the parser.threaded attribute). #925
  • Prefixed public macros to avoid name clashes with other libraries. #1035
  • Better documentation regarding package managers (brew, MSYS2, conan, apt, vcpkg, FreeBSD package manager, etc.).
  • Better documentation regarding CMake usage.

Standards

  • We improved standard compliance with respect to both the JSON RFC 8259 and JSON Pointer RFC 6901. We added the at_pointer method to nodes for standard-compliant JSON Pointer queries. The legacy at(std::string_view) method remains but is deprecated since it is not standard-compliant as per RFC 6901.
  • We removed computed GOTOs without sacrificing performance thus improving the C++ standard compliance (since computed GOTOs are compiler-specific extensions).
  • Better support for C++20 #1050


via Release notes from simdjson https://ift.tt/3eoyyad
August 19, 2020 at 05:13PM

Version 0.4.5

Version 0.4.5

https://ift.tt/3dQUcmi

Fixed an issue whereas the installed library would not get versioned properly on some systems.



via Release notes from simdjson https://ift.tt/3eoyyad
June 30, 2020 at 06:37PM

Version 1.0.1

Version 1.0.1

https://ift.tt/3pmHNzR

It is a patch release on version 1.0.1.

It fixes issue #1736 (number parse bug in get_uint64_in_string())

Credit for the bug report to @madhur4127



via Release notes from simdjson https://ift.tt/3eoyyad
October 20, 2021 at 11:20AM

Version 0.9.5

Version 0.9.5

https://ift.tt/34ogsld

Patch release fixing issue 1588: Unexpected behavior while traversing a json array containing json objects containing a subset of known keys. This patch adds a single line. Users relying on the On Demand front-end should update.



via Release notes from simdjson https://ift.tt/3eoyyad
May 27, 2021 at 07:38PM

Version 0.8.0

Version 0.8.0

https://ift.tt/366X6Cm

The high-performance On Demand front-end introduced in version 0.7 has received major quality-of-life and performance improvements (credit @jkeiser).

  • Runtime dispatching is now supported, achieving high performance without compiling for a specific CPU.
  • Object field lookup is now order-insensitive: double x = object["x"]; double y = object["y"]; will work no matter which order the fields appear in the object. Reading fields in order still gives maximum performance.
  • Object lookup and array iteration can now be used against untyped values, enabling things like chained lookup (object["a"]["b"])
  • Numbers, strings and boolean values can be saved and parsed later by storing the ondemand::value, allowing more efficient filtering and bulk parsing, as well as fixing smaller quality-of-life issues.

We have improved our CMake build with respect to installed artefacts so that
CMake dependencies automatically handle thread dependencies.

We have greatly improved our benchmarks with a set of realistic tasks on
realistic datasets, using Google Benchmark as a framework.

skylake-clang10



via Release notes from simdjson https://ift.tt/3eoyyad
January 22, 2021 at 11:09AM

Version 0.6.0

Version 0.6.0

https://ift.tt/35qnZjI

Release 0.6 features faster serialization, updated number parsing and a new parsing API (On Demand).

Novel parsing approach: Prior to release 0.6, the simdjson only supported the DOM model. In the DOM model, you parse the entire document eagerly and then query the resulting parsed document. With release 0.6, we have introduced an innovative new query model called "On Demand" (credit: @jkeiser). As the name suggests, the programmer can now access only the components of the JSON documents that are needed. It is useful when the dialect of the JSON document is known at compile-time. It can multiply the performance: on a benchmark where we retrieve unique identifiers from a file, we go from 2.3 GB/s to 4.6 GB/s. Compared to the generic DOM approach, the On Demand strategy may improve parsing speed and reduce memory usage. The On Demand API is both easy to use and highly efficient. We encourage our users to try out the new On Demand API and to provide constructive feedback. Though it is well tested, we consider On Demand to be a novel and experimental feature that will receive further updates in the upcoming releases including runtime dispatching. Users should be mindful that the On Demand API is subject to change in future releases, we encourage testing more than deployment. Please check our documentation:
https://github.com/simdjson/simdjson/blob/master/doc/ondemand.md

Performance:

  • Serializing a parsed JSON element is about 10x faster compared to previous releases.

New features:

  • We make it easier to check whether a given implementation/kernel is available at runtime with a new function my_implementation->supported_by_runtime_system().

Correctness:

  • Serialization is now guaranteed to be locale-independent.
  • We have redone the number parsing routines and extended our testing. We no longer fall back on the system's C library thus avoiding potential sources of unexact parsing.
  • Fuzz testing has been extended. E.g., our standalone UTF-8 validation is now fuzz tested. (Credit @pauldreik)


via Release notes from simdjson https://ift.tt/3eoyyad
October 23, 2020 at 08:35AM

Version 2.2.1

Version 2.2.1

https://ift.tt/fKGnE37

What's Changed

  • update JsonCpp to version 1.9.5 by @striezel in #1863
  • update dependency nlohmann/json for benchmarks to current version 3.10.5 by @striezel in #1862
  • Documenting a specific use case where you need a value if and only if it is another key is not present by @lemire in #1865
  • Adding DOM benchmark to msgpack by @lemire in #1866
  • Improve build times for debug builds by @strager in #1859
  • Fixing issue 1870 by @lemire in #1871
  • We change slightly on development checks are enabled. by @lemire in #1869

Full Changelog: v2.2.0...v2.2.1



via Release notes from simdjson https://ift.tt/7XSAGCW
July 19, 2022 at 03:48PM

Version 2.2.0

Version 2.2.0

https://ift.tt/dnrXCOi

This release is a bug-fix release and is recommended for users relying on the icelake kernel.

What's Changed

  • Increased SIMDJSON_PADDING to 64 bytes (for AVX-512)
  • Improved C++11 compatibility by replacing 0b1111... with 0xfff...


via Release notes from simdjson https://ift.tt/8zvhAge
July 04, 2022 at 08:57PM

Version 0.7.0

Version 0.7.0

https://ift.tt/2L9x221

This version improves our support for streams of documents (ndjson). We have improved the documentation, especially with how one might deal with truncated inputs and track the location of the current JSON. We have added fuzz testing and other tests. We added some minor fixes.

Performance:

  • SIMD accelerated (AltiVec) kernel for POWER processors.

Platform support:

  • We can now disable exceptions when compiling with MSVC


via Release notes from simdjson https://ift.tt/3eoyyad
December 04, 2020 at 06:06PM

Version 1.1.0

Version 1.1.0

https://ift.tt/f83TaVD

This is a maintenance release. Except for the new 'current_depth()' method, no new API component was added.

  • Adds 'current_depth()'.
  • Fixing a minor bug in value::get_number_type()
  • fixing compiler warnings
  • various documentation fixes
  • prefixing some internal macros
  • operator<< uses ADL


via Release notes from simdjson https://ift.tt/M8LsruW
May 17, 2022 at 05:14PM

Version 0.4.4

Version 0.4.4

https://ift.tt/2ZomcZl

CMake fix for when one builds using Visual Studio 2015 via Ninja.



via Release notes from simdjson https://ift.tt/3eoyyad
June 30, 2020 at 08:42AM

Heads up on new fork repo : https://github.com/jdonaldson/hx-lua-simdjson

I just wanted to pass along the fork I made : https://github.com/jdonaldson/hx-lua-simdjson

I stripped it down from your original repo here. I'm not going to maintain benchmarks, specs, etc. (I'll do that over in haxe-land).

I retained the license, and added you to the acknowledgements along with the simdjson authors.

Anything else I'm missing there?

FWIW, I think a fork wound up being the right call here for a lot of reasons. I had to tweak things a lot to get them working properly for my Haxe target. It'll be very difficult to unify our efforts here, unfortunately. :(

As a side note, I did resolve one json spec issue by updating the simdjson single header cpp/h files. I'd do that at your earliest convenience to ensure you've got up to date json format compliance.

Cheers!

(close this whenever you're happy with it)

Version 0.6.1

Version 0.6.1

https://ift.tt/3oS9I8h

This is a minor patch release for version 0.6.0 to support legacy libc++ builds.

#1285

#1286



via Release notes from simdjson https://ift.tt/3eoyyad
November 03, 2020 at 09:20PM

Version 0.5.0

Version 0.5.0

https://ift.tt/3hjQkgs

Performance

  • Faster and simpler UTF-8 validation with the lookup4 algorithm #993
  • We improved the performance of simdjson under Visual Studio by about 25%. Users will still get better performance with clang-cl (+30%) but the gap has been reduced. #1031

Code usability

  • In parse_many, when parsing streams of JSON documetns, we give to the users runtime control as to whether threads are used (via the parser.threaded attribute). #925
  • Prefixed public macros to avoid name clashes with other libraries. #1035
  • Better documentation regarding package managers (brew, MSYS2, conan, apt, vcpkg, FreeBSD package manager, etc.).
  • Better documentation regarding CMake usage.

Standards

  • We improved standard compliance with respect to both the JSON RFC 8259 and JSON Pointer RFC 6901. We added the at_pointer method to nodes for standard-compliant JSON Pointer queries. The legacy at(std::string_view) method remains but is deprecated since it is not standard-compliant as per RFC 6901.
  • We removed computed GOTOs without sacrificing performance thus improving the C++ standard compliance (since computed GOTOs are compiler-specific extensions).
  • Better support for C++20 #1050


via Release notes from simdjson https://ift.tt/3eoyyad
August 19, 2020 at 05:13PM

Version 0.4.6

Version 0.4.6

https://ift.tt/2ZpHdTD

  • Fixing issue 1013 (removing minor extra work when SIMDJSON_JUST_LIBRARY is set.
  • Changing soname.


via Release notes from simdjson https://ift.tt/3eoyyad
July 01, 2020 at 01:16PM

Version 0.7.1

Version 0.7.1

https://ift.tt/2Ko12qq

It is a patch release for version 0.7.0 which mistakenly disabled, by default, optimized ARM NEON and POWER kernels. The result was a substantial lost of performance, by default, on these platforms. Users could still work around the limitation by passing macro values to the compiler.



via Release notes from simdjson https://ift.tt/3eoyyad
December 14, 2020 at 09:46AM

Version 3.1.0

Version 3.1.0

https://ift.tt/5Z3WivQ

What's Changed

  • Disabling fallback kernel on systems where it is not needed by @lemire in #1930
  • Testing and fixing log issues. by @lemire in #1939
  • complete number_in_string for documents, stringify the version macro and improving trailing content errors by @lemire in #1934

Full Changelog: v3.0.1...3.1.0



via Release notes from simdjson https://ift.tt/eJqdOlZ
January 21, 2023 at 05:08PM

Version 0.9.0

Version 0.9.0

https://ift.tt/30SIq72

The high-performance On Demand front-end introduced in version 0.7 and optimized in version 0.8 becomes the primary simdjson front-end in version 0.9 (credit @jkeiser). The On Demand front-end benefits from new features:

  • The On Demand elements have received a type() method so branching on the schema becomes possible.
  • We make it safer to recover one JSON element as multiple types #1414
  • We added safety checks for out-of-order iterations #1416

Other changes :

  • You can now parse a DOM document as a separate entity from the parser #1430
  • Improve compatibility with the Qt library.
  • We encourage users to replace templated get<T>() methods with specific methods, using get_object() instead of get<simdjson::dom::object>(), and get_uint64() instead of get<uint64_t>().
  • dom::document_stream::iterator has received a default constructor and is copyable


via Release notes from simdjson https://ift.tt/3eoyyad
March 17, 2021 at 10:09AM

Version 0.4.1

Version 0.4.1

https://ift.tt/3eBc99S

This is a patch release to fix minor issues with release 0.4.0.

  • We removed a spurious 'optional' header (which was unused).
  • We document std::string_view aliasing for pre-C++17 compilers.
  • We made the cmake build more robust (missing git, ninja).
  • When used inside another cmake project, we just build the library by default (no test or benchmark).
  • We improved compatibility with 32-bit systems.


via Release notes from simdjson https://ift.tt/3eoyyad
June 26, 2020 at 08:08PM

Version 0.4.0

Version 0.4.0

https://ift.tt/3hV9mKG

Highlights

  • Test coverage has been greatly improved and we have resolved many static-analysis warnings on different systems.

New features:

  • We added a fast (8GB/s) minifier that works directly on JSON strings.
  • We added fast (10GB/s) UTF-8 validator that works directly on strings (any strings, including non-JSON).
  • The array and object elements have a constant-time size() method.

Performance:

  • Performance improvements to the API (type(), get<>()).
  • The parse_many function (ndjson) has been entirely reworked. It now uses a single secondary thread instead of several new threads.
  • We have introduced a faster UTF-8 validation algorithm (lookup3) for all kernels (ARM, x64 SSE, x64 AVX).

System support:

  • C++11 support for older compilers and systems.
  • FreeBSD support (and tests).
  • We support the clang front-end compiler (clangcl) under Visual Studio.
  • It is now possible to target ARM platforms under Visual Studio.
  • The simdjson library will never abort or print to standard output/error.


via Release notes from simdjson https://ift.tt/3eoyyad
June 23, 2020 at 07:57PM

Version 0.4.7

Version 0.4.7

https://ift.tt/2Bd0gsc

This is a patch release to remove an overhead of 100 ns per query. It will improve performance on small files. The fix involves the removal of a single line of code. Credit to @vitlibar from Yandex.



via Release notes from simdjson https://ift.tt/3eoyyad
July 17, 2020 at 02:19PM

OSX performance

OSX, latest Luajit 2.1.

Totals:
{
cjson = 0,
dkjson = 0,
rapidjson = 0.07904502,
simdjson = 0.57141524
}

Version 2.0.3

Version 2.0.3

https://ift.tt/9TBb1pP

This is a third patch release which allows the AVX-512 kernel under Visual Studio 2019 and up.



via Release notes from simdjson https://ift.tt/pTRjaPH
June 02, 2022 at 12:59PM

Version 2.0.1

Version 2.0.1

https://ift.tt/T8pYgIU

It is a patch release which makes sure that AVX-512 is always enabled by default, even when using single-header files. It was the intended behaviour.



via Release notes from simdjson https://ift.tt/CO5KVi6
May 26, 2022 at 03:27PM

Version 3.1.0

Version 3.1.0

https://ift.tt/vUoA9JP

What's Changed

  • Disabling fallback kernel on systems where it is not needed by @lemire in #1930
  • Testing and fixing log issues. by @lemire in #1939
  • complete number_in_string for documents, stringify the version macro and improving trailing content errors by @lemire in #1934

Full Changelog: v3.0.1...3.1.0



via Release notes from simdjson https://ift.tt/LQTW9li
January 23, 2023 at 06:23PM

Version 0.4.3

Version 0.4.3

https://ift.tt/2VwAc2a

  • MinGW tests (32-bit and 64-bit)
  • Visual Studio 2015 patch


via Release notes from simdjson https://ift.tt/3eoyyad
June 29, 2020 at 08:14PM

Version 0.8.1

Version 0.8.1

https://ift.tt/3ptLelA

This is a patch release fixing the find_package(simdjson) functionality for CMake for users that updated to simdjson 0.8.0.



via Release notes from simdjson https://ift.tt/3eoyyad
January 28, 2021 at 08:50AM

Version 0.8.2

Version 0.8.2

https://ift.tt/3qbDzsr

This patch adds an explicit include for string_view which is needed by Visual Studio 2017. It also includes other minor Visual Studio 2017 fixes. Visual Studio 2017 are reminded that they should target specifically x64 builds.

credit to @NancyLi1013 for reporting the issue



via Release notes from simdjson https://ift.tt/3eoyyad
February 10, 2021 at 06:03PM

Version 3.0.0

Version 3.0.0

https://ift.tt/bPNHtS7

What's Changed (gist)

The main change in version 3.0.0 is that the is_null() methods may return an error. Previously, they would return merely true or false.

What's Changed (details)

  • [skip ci] Add an .editorconfig for .cpp/.h/.md for whitespace settings by @TysonAndre in #1901
  • Minor fix (documentation and safety) regarding max. depth in ondemand. by @lemire in #1906
  • Documenting how one can check for the end of the document. by @lemire in #1907
  • Check for trailing tokens in json2msgpack ondemand benchmark by @TysonAndre in #1908
  • Documents better the type method and makes is_null return an error condition in some instances by @lemire in #1909

Full Changelog: v2.2.3...v3.0.0



via Release notes from simdjson https://ift.tt/8EDM9I1
October 06, 2022 at 10:52AM

Version 0.9.1

Version 0.9.1

https://ift.tt/3cK3ORn

This is a patch release removing dead code (get_root_value). #1504



via Release notes from simdjson https://ift.tt/3eoyyad
March 18, 2021 at 10:32AM

Version 0.9.4

Version 0.9.4

https://ift.tt/3ypCPod

This fourth patch release is the second and final fix on an issue with the ondemand front-end where when we search for a key and it is not found, we can end up in a poor state from which follow-up queries will lead to spurious errors even with a valid JSON.



via Release notes from simdjson https://ift.tt/3eoyyad
May 20, 2021 at 03:26PM

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.