Giter Site home page Giter Site logo

yamafaktory / jql Goto Github PK

View Code? Open in Web Editor NEW
1.4K 19.0 30.0 4.06 MB

A JSON Query Language CLI tool

Home Page: https://crates.io/crates/jql

License: Apache License 2.0

Rust 97.84% Shell 1.81% Just 0.35%
json rust rustlang cli tool utility cargo tools devops-tools

jql's Introduction

jql


GitHub Workflow Status Crates.io Docs.rs Docs.rs

jql is a JSON Query Language tool built with Rust ๐Ÿฆ€.

Pronounce it as jackal ๐Ÿบ.

๐Ÿ“œ Philosophy

  • โšกBe fast
  • ๐Ÿชถ Stay lightweight
  • ๐ŸŽฎ Keep its features as simple as possible
  • ๐Ÿง  Avoid redundancy
  • ๐Ÿ’ก Provide meaningful error messages
  • ๐Ÿฐ Eat JSON as input, process, output JSON back

๐Ÿš€ Installation

Alpine Linux

The package is maintained by @jirutka.

apk add jql

Archlinux

The AUR package is maintained by @barklan.

yay -S jql

Cargo

cargo install jql

Fedora

dnf install jql

FreeBSD

pkg install jql

Homebrew

brew install jql

Nix

nix-env -i jql

openSUSE

zypper install jql

Manual installation from GitHub

Compiled binary versions are automatically uploaded to GitHub when a new release is made. You can install jql manually by downloading a release.

๐Ÿ› ๏ธ Usage

To make a selection from a JSON input, jql expects a query as a sequence of tokens.

To be fully compliant with the JSON format, jql always expect key selectors to be double-quoted, see The JavaScript Object Notation (JSON) Data Interchange Format.

{
  ".valid": 1337,
  "": "yeah!",
  "\"": "yup, valid too!"
}

Consequently, to be shell compliant, a query must be either enclosed by single quotation marks or every inner double quotation mark must be escaped.

Separators

Group separator

Group separators build up an array from sub-queries.

JSON input

{ "a": 1, "b": 2, "c": 3 }

Query

'"a","b","c"'

JSON output

[1, 2, 3]

Selectors

Arrays

Array index selector

Indexes can be used in arbitrary order.

JSON input

[1, 2, 3]

Query

'[2,1]'

JSON output

[3, 2]
Array range selector

Range can be in natural order [0:2], reversed [2:0], without lower [:2] or upper bound [0:].

JSON input

[1, 2, 3]

Query

'[2:1]'

JSON output

[3, 2]
Lens selector

Lens can be a combination of one or more selectors with or an optional value, a value being any of boolean | null | number | string.

JSON input

[
  { "a": 1, "b": { "d": 2 } },
  { "a": 2, "b": "some" },
  { "a": 2, "b": { "d": null } },
  { "a": 2, "b": true },
  { "c": 3, "b": 4 }
]

Query

'|={"b""d"=2, "c"}'

JSON output

[
  { "a": 1, "b": { "d": 2 } },
  { "c": 3, "b": 4 }
]

Objects

Key selector

Any valid JSON key can be used.

JSON input

{ "a": 1, "b": 2, "c": 3 }

Query

'"c"'

JSON output

3
Multi key selector

Keys can be used in arbitrary order.

JSON input

{ "a": 1, "b": 2, "c": 3 }

Query

'{"c","a"}'

JSON output

{ "c": 3, "a": 1 }
Object index selector

Indexes can be used in arbitrary order.

JSON input

{ "a": 1, "b": 2, "c": 3 }

Query

'{2,0}'

JSON output

{ "c": 3, "a": 1 }
Object range selector

Range can be in natural order {0:2}, reversed {2:0}, without lower {:2} or upper bound {0:}.

JSON input

{ "a": 1, "b": 2, "c": 3 }

Query

'{2:1}'

JSON output

{ "c": 3, "b": 2 }

Operators

Flatten operator

Flattens arrays and objects.

JSON input

[[[[[[[[[[[[[[{ "a": 1 }]]]]]]]]]]]]], [[[[[{ "b": 2 }]]]], { "c": 3 }], null]

Query

'..'

JSON output

[{ "a": 1 }, { "b": 2 }, { "c": 3 }, null]

JSON input

{ "a": { "c": false }, "b": { "d": { "e": { "f": 1, "g": { "h": 2 } } } } }

Query

'..'

JSON output

{
  "a.c": false,
  "b.d.e.f": 1,
  "b.d.e.g.h": 2
}
Pipe in operator

Applies the next tokens in parallel on each element of an array.

JSON input

{ "a": [{ "b": { "c": 1 } }, { "b": { "c": 2 } }] }

Query

'"a"|>"b""c"'

JSON output

[1, 2]
Pipe out operator

Stops the parallelization initiated by the pipe in operator.

JSON input

{ "a": [{ "b": { "c": 1 } }, { "b": { "c": 2 } }] }

Query

'"a"|>"b""c"<|[1]'

JSON output

2
Truncate operator

Maps the output into simple JSON primitives boolean | null | number | string | [] | {}.

JSON input

{ "a": [1, 2, 3] }

Query

'"a"!'

JSON output

[]

๐Ÿ’ป Shell integration

How to save the output

jql '"a"' input.json > output.json

How to read from stdin

cat test.json | jql '"a"'

Available flags

Inline the JSON output

By default, the output is pretty printed in a more human-readable way, this can be disabled.

-i, --inline

Read the query from file

The command will read the provided query from a file instead of the stdin.

-q, --query <FILE>

Write to stdout without JSON double-quotes

This can be useful to drop the double-quotes surrounding a string primitive.

-r, --raw-string

Read a stream of JSON data line by line

This flag is only about reading processing any JSON output streamed line by line (e.g. Docker logs with the --follow flag). This is not an option to read an incomplete streamed content (e.g. a very large input).

-s, --stream

Validate the JSON data

The command will return a matching exit code based on the validity of the JSON content or file provided.

-v, --validate

Print help

-h, --help

Print version

-V, --version

Help

jql -h
jql --help

๐Ÿฆ€ Workspace

This project is composed of following crates:

Development

Some commands are available as a justfile at the root of the workspace (testing / fuzzing).

Prerequisites

Commands

just --list

โš ๏ธ Non-goal

There's no plan to align jql with jq or any other similar tool.

โšก Performance

Some benchmarks comparing a set of similar functionalities provided by this tool and jq are available here.

๐Ÿ“” Licenses

jql's People

Contributors

aliesbelik avatar barklan avatar blinxen avatar cailloumajor avatar fivenp avatar github-actions[bot] avatar gsquire avatar haraldh avatar ignatenkobrain avatar jameschenjav avatar jqnatividad avatar kmaasrud avatar patrickelectric avatar sdroege avatar tianlangstudio avatar yamafaktory avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jql's Issues

jq incompatibility or bug?

Hi,

thanks for the tool!

Seems there is a bug or the tool is just not compatible with jq, can you please clarify this.

For example, let's take this Bitwarden export (sensitive info stripped off for obvious reason)

{
  "encrypted": false,
  "folders": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "name": "A***"
    }
  ],
  "items": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "organizationId": "",
      "folderId": "",
      "type": 1,
      "reprompt": 0,
      "name": "4pda.ru",
      "notes": "",
      "favorite": false,
      "login": {
        "uris": [
          {
            "match": "",
            "uri": "http://4pda.ru"
          }
        ],
        "username": "***",
        "password": "***",
        "totp": ""
      },
      "collectionIds": ""
    },
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "organizationId": "",
      "folderId": "",
      "type": 1,
      "reprompt": 0,
      "name": "192.168.178.1",
      "notes": "",
      "favorite": false,
      "login": {
        "uris": [
          {
            "match": "",
            "uri": "http://192.168.178.1"
          }
        ],
        "username": "",
        "password": "***",
        "totp": ""
      },
      "collectionIds": ""
    }
  ]
}
๏ฃฟ workstation ~/Downloads 
โœ–1 โฏ /bin/cat bitwarden.json | jq ".items[].id"

gives the following output (which I assume is right)

"00000000-0000-0000-0000-000000000000"
"00000000-0000-0000-0000-000000000000"

on the other hand, jql spits out the whole json, regardless the selector:

๏ฃฟ workstation ~/Downloads 
โฏ /bin/cat bitwarden.json | jql ".items[0]"
{
  "encrypted": false,
  "folders": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "name": "A***"
    }
  ],
  "items": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "organizationId": "",
      "folderId": "",
      "type": 1,
      "reprompt": 0,
      "name": "4pda.ru",
      "notes": "",
      "favorite": false,
      "login": {
        "uris": [
          {
            "match": "",
            "uri": "http://4pda.ru"
          }
        ],
        "username": "***",
        "password": "***",
        "totp": ""
      },
      "collectionIds": ""
    },
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "organizationId": "",
      "folderId": "",
      "type": 1,
      "reprompt": 0,
      "name": "192.168.178.1",
      "notes": "",
      "favorite": false,
      "login": {
        "uris": [
          {
            "match": "",
            "uri": "http://192.168.178.1"
          }
        ],
        "username": "",
        "password": "***",
        "totp": ""
      },
      "collectionIds": ""
    }
  ]
}

tried using jql

Hello,

I used brew install and have a sample.json file inside my folder directory. I was trying the commands as shown on the README, but could not get any data returned or get "unable to parse input".

my JSON

{
  "employee": {
    "name":"employee01",
    "age": "30",
    "city":"New York"
  },
  "boss" : {
    "name": "boss person",
    "role": "regional supervisor"
  },
  "stuff" : {
    "a": 1,
    "b": 2,
    "c": 3
  }

}

the commands i tried:

  • jql "boss" sample.json among variations
  • jql "stuff{'a'}" sample.json
  • jql employee -s -r sample.json
  • jsql "employee['city']" -s -r sample.json
    all of these return "unable to parse"

what am i doing wrong here?

Conditional filters

A feature to conditionally filter items of an array would be very useful! This would be akin to jq's select function.

My feeling is that the jql syntax is stronger than jq in it's simplicity, and thus should not adopt the "function" type signature (select(...)). I propose the following:

Given the below JSON

[
    { "key": "give-me-this", "another-key": 42 },
    { "key": "dont-give-me-this", "another-key": 13 },
    { "key": "give-me-this", "another-key": 69 }
]

one could run this query:

$ jql '.|{"key" == "give-me-this"}' file.json
[
    { "key": "give-me-this", "another-key": 42 },
    { "key": "give-me-this", "another-key": 69 },
]

Inequalities would obviously be specified with !=. Other conditions could be supported as well, such as a way of checking if an array contains a specific element.

What do you think about this proposal? Also, thank you very much for making this amazing crate, it has been indispensable as a library in making mdzk happen!

Read json from stdin.

jql may support read json data from stdin, then we can use it in a pipeline.

It's really useful.

cat example.json | jql .

Cannot select or filter on lens-produced arrays

When filtering with regular filters, I can run another filter or do property selections on the resulting array in order to get child elements. This does not seem to work with the new lens feature.

Example

Take the following example JSON file:

{
  "menu": {
    "id": "file",
    "value": "File",
    "popup": {
      "menuitem": [
        {"value": "New", "onclick": "CreateNewDoc()"},
        {"value": "Open", "onclick": "OpenDoc()"},
        {"value": "Close", "onclick": "CloseDoc()"}
      ]
    }
  }
}

Filters work as expected:

$ jql '"menu"."popup"."menuitem"|{"value","onclick"}' test.json
[
  {
    "value": "New",
    "onclick": "CreateNewDoc()"
  },
  {
    "value": "Open",
    "onclick": "OpenDoc()"
  },
  {
    "value": "Close",
    "onclick": "CloseDoc()"
  }
]
$ jql '"menu"."popup"."menuitem"|{"value","onclick"}|"onclick"' test.json
[
  "CreateNewDoc()",
  "OpenDoc()",
  "CloseDoc()"
]
$ jql '"menu"."popup"."menuitem"|{"value","onclick"}."value"' test.json
[
  "New",
  "Open",
  "Close"
]

But I'm unable to do the same with lenses:

$ jql '"menu"."popup"."menuitem"|={"value":"New"}' test.json
[
  {
    "value": "New",
    "onclick": "CreateNewDoc()"
  }
]
$ jql '"menu"."popup"."menuitem"|={"value":"New"}|"onclick"' test.json
[
  {
    "value": "New",
    "onclick": "CreateNewDoc()"
  }
]
$ jql '"menu"."popup"."menuitem"|={"value":"New"}."onclick"' test.json
[
  {
    "value": "New",
    "onclick": "CreateNewDoc()"
  }
]
$ jql '"menu"."popup"."menuitem"|={"value":"New"}.[0]."onclick"' test.json
[
  {
    "value": "New",
    "onclick": "CreateNewDoc()"
  }
]

This does not seem like the correct behavior. Am I doing something wrong, or is this simply a bug?

csv export

hey,

Thanks for what seems to be one excellent tool.

What i'm looking for is a tool which can to this and also export to CSV (or SQL)

This because the .json i'm working on is so very json it's very hard to convert but your page actually documents the case using objects and arrays with cross references etc.

Flattened/Minified json file cannot be parsed

I have a very big json file that was flattened / minified to a single line, and I would like to explore this document, however jq don't work and neither does jql.

$ jql --raw-output '.' aggregate-raw.json
Invalid JSON file or content

That would be nice if JQL could process this document.

Pipe JSON

Hey, I'm wondering if piping JSON is a feature you're considering adding.

Filtering doesn't work when key is assigned to a variable name.

Can't get filtering work when key is assigned to a variable?

$ FOO=\'\"foo\"\' && echo "{\"foo\":\"bar\"}" | jql -r "$FOO"
Empty group
$ echo "{\"foo\":\"bar\"}" | jql -r '"foo"'
bar

jq seems to be handling this very well.

$ FOO=".foo" && echo "{\"foo\":\"bar\"}" | jq -r "$FOO"
bar
$ echo "{\"foo\":\"bar\"}" | jq -r ".foo"
bar

I think, the input to the jql can be improved in similar way @yamafaktory.

jql always exit with return code 0

jql always seems to be exit with error code 0 so there is no good way to catch the errors in scripts. Any suggestions?

$ jql -r '"foo"' config.json 
bar
$ echo $?
0
$ jql -r '"foo"' config1.json 
File "config1.json" not found
$ echo $?
0
$ jq -r ".foo" config.json 
bar
$ echo $?
0
jq -r ".foo" config1.json 
jq: error: Could not open file config1.json: No such file or directory
$ echo $?
2
$ jq -er ".foo" config1.json 
jq: error: Could not open file config1.json: No such file or directory
$ echo $?
2

Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects - the results are available here. So that's why I think it's worth trying to apply PGO to jql. I already performed some benchmarks and want to share my results here.

Test environment

  • Fedora 38
  • Linux kernel 6.5.5
  • AMD Ryzen 9 5900x
  • 48 Gib RAM
  • SSD Samsung 980 Pro 2 Tib
  • Compiler - Rustc 1.73
  • jql version: the latest for now from the main branch on commit 7729cbafaea0367c2f86227234fb3f8e9a8fd905
  • Disabled Turbo boost

Benchmark setup

For benchmarking purposes, I use the scenario from https://github.com/yamafaktory/jql/blob/main/performance.sh , just a bit tweaked (removed vanilla jq invocations, add multiple jql versions to the script) - edited version is available here. Release build is done with cargo build --release, PGO optimized build is done with cargo-pgo.

I tested 3 configurations:

  • Default Release, binary called jql_z
  • Tweked Release (use opt-level = 3), binary called jql_opt_3
  • PGO-optimized, binary called jql_optimized

PGO profiles were collected from the same workload in performance.sh and merged via llvm-profdata during the PGO optimization phase.

All benchmarks are done multiple times, on the same hardware/software setup, with the same background "noise" (as much I can guarantee ofc).

Results

I got the following results from running performance.sh:

./performance.sh
Benchmark 1: echo '{ "foo": "bar" }' | ./jql_z '"foo"'
  Time (mean ยฑ ฯƒ):       4.5 ms ยฑ   0.2 ms    [User: 1.5 ms, System: 7.6 ms]
  Range (min โ€ฆ max):     4.0 ms โ€ฆ   6.1 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: The first benchmarking run for this command was significantly slower than the rest (6.1 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark 2: echo '{ "foo": "bar" }' | ./jql_opt_3 '"foo"'
  Time (mean ยฑ ฯƒ):       4.5 ms ยฑ   0.4 ms    [User: 1.4 ms, System: 7.8 ms]
  Range (min โ€ฆ max):     4.1 ms โ€ฆ  14.4 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: The first benchmarking run for this command was significantly slower than the rest (14.4 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark 3: echo '{ "foo": "bar" }' | ./jql_optimized '"foo"'
  Time (mean ยฑ ฯƒ):       4.4 ms ยฑ   0.2 ms    [User: 1.2 ms, System: 7.7 ms]
  Range (min โ€ฆ max):     4.1 ms โ€ฆ   6.4 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: The first benchmarking run for this command was significantly slower than the rest (6.4 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Summary
  echo '{ "foo": "bar" }' | ./jql_optimized '"foo"' ran
    1.00 ยฑ 0.05 times faster than echo '{ "foo": "bar" }' | ./jql_z '"foo"'
    1.01 ยฑ 0.09 times faster than echo '{ "foo": "bar" }' | ./jql_opt_3 '"foo"'
Benchmark 1: echo '[1, 2, 3]' | ./jql_z '[0]'
  Time (mean ยฑ ฯƒ):       4.6 ms ยฑ   0.2 ms    [User: 1.4 ms, System: 7.9 ms]
  Range (min โ€ฆ max):     4.2 ms โ€ฆ   5.9 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: echo '[1, 2, 3]' | ./jql_opt_3 '[0]'
  Time (mean ยฑ ฯƒ):       4.6 ms ยฑ   0.1 ms    [User: 1.3 ms, System: 8.1 ms]
  Range (min โ€ฆ max):     4.2 ms โ€ฆ   5.2 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.

Benchmark 3: echo '[1, 2, 3]' | ./jql_optimized '[0]'
  Time (mean ยฑ ฯƒ):       4.6 ms ยฑ   0.1 ms    [User: 1.2 ms, System: 7.9 ms]
  Range (min โ€ฆ max):     4.2 ms โ€ฆ   5.6 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.

Summary
  echo '[1, 2, 3]' | ./jql_z '[0]' ran
    1.00 ยฑ 0.05 times faster than echo '[1, 2, 3]' | ./jql_optimized '[0]'
    1.01 ยฑ 0.05 times faster than echo '[1, 2, 3]' | ./jql_opt_3 '[0]'
Benchmark 1: echo '[1, [2], [[3]]]' | ./jql_z '..'
  Time (mean ยฑ ฯƒ):       4.7 ms ยฑ   0.2 ms    [User: 1.8 ms, System: 8.0 ms]
  Range (min โ€ฆ max):     4.2 ms โ€ฆ   6.2 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: echo '[1, [2], [[3]]]' | ./jql_opt_3 '..'
  Time (mean ยฑ ฯƒ):       4.7 ms ยฑ   0.2 ms    [User: 1.6 ms, System: 8.1 ms]
  Range (min โ€ฆ max):     4.3 ms โ€ฆ   5.7 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.

Benchmark 3: echo '[1, [2], [[3]]]' | ./jql_optimized '..'
  Time (mean ยฑ ฯƒ):       4.7 ms ยฑ   0.1 ms    [User: 1.5 ms, System: 8.1 ms]
  Range (min โ€ฆ max):     4.3 ms โ€ฆ   5.4 ms    1000 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.

Summary
  echo '[1, [2], [[3]]]' | ./jql_optimized '..' ran
    1.00 ยฑ 0.04 times faster than echo '[1, [2], [[3]]]' | ./jql_opt_3 '..'
    1.00 ยฑ 0.04 times faster than echo '[1, [2], [[3]]]' | ./jql_z '..'
Benchmark 1: cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_z '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null
  Time (mean ยฑ ฯƒ):      20.6 ms ยฑ   1.3 ms    [User: 14.7 ms, System: 26.0 ms]
  Range (min โ€ฆ max):    18.6 ms โ€ฆ  31.5 ms    1000 runs

Benchmark 2: cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_opt_3 '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null
  Time (mean ยฑ ฯƒ):      19.9 ms ยฑ   1.2 ms    [User: 11.8 ms, System: 26.0 ms]
  Range (min โ€ฆ max):    17.8 ms โ€ฆ  26.3 ms    1000 runs

Benchmark 3: cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_optimized '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null
  Time (mean ยฑ ฯƒ):      19.3 ms ยฑ   1.3 ms    [User: 10.6 ms, System: 26.6 ms]
  Range (min โ€ฆ max):    17.1 ms โ€ฆ  26.2 ms    1000 runs

Summary
  cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_optimized '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null ran
    1.04 ยฑ 0.09 times faster than cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_opt_3 '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null
    1.07 ยฑ 0.10 times faster than cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json | ./jql_z '|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null

The same results in performance.md format:

โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
       โ”‚ File: PERFORMANCE.md
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
   1   โ”‚ | Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
   2   โ”‚ |:---|---:|---:|---:|---:|
   3   โ”‚ | `echo '[1, [2], [[3]]]' \| ./jql_z '..'` | 4.7 ยฑ 0.2 | 4.2 | 6.2 | 1.00 ยฑ 0.04 |
   4   โ”‚ | `echo '[1, [2], [[3]]]' \| ./jql_opt_3 '..'` | 4.7 ยฑ 0.2 | 4.3 | 5.7 | 1.00 ยฑ 0.04 |
   5   โ”‚ | `echo '[1, [2], [[3]]]' \| ./jql_optimized '..'` | 4.7 ยฑ 0.1 | 4.3 | 5.4 | 1.00 |
   6   โ”‚
   7   โ”‚ | Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
   8   โ”‚ |:---|---:|---:|---:|---:|
   9   โ”‚ | `echo '[1, 2, 3]' \| ./jql_z '[0]'` | 4.6 ยฑ 0.2 | 4.2 | 5.9 | 1.00 |
  10   โ”‚ | `echo '[1, 2, 3]' \| ./jql_opt_3 '[0]'` | 4.6 ยฑ 0.1 | 4.2 | 5.2 | 1.01 ยฑ 0.05 |
  11   โ”‚ | `echo '[1, 2, 3]' \| ./jql_optimized '[0]'` | 4.6 ยฑ 0.1 | 4.2 | 5.6 | 1.00 ยฑ 0.05 |
  12   โ”‚
  13   โ”‚ | Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
  14   โ”‚ |:---|---:|---:|---:|---:|
  15   โ”‚ | `echo '{ "foo": "bar" }' \| ./jql_z '"foo"'` | 4.5 ยฑ 0.2 | 4.0 | 6.1 | 1.00 ยฑ 0.05 |
  16   โ”‚ | `echo '{ "foo": "bar" }' \| ./jql_opt_3 '"foo"'` | 4.5 ยฑ 0.4 | 4.1 | 14.4 | 1.01 ยฑ 0.09 |
  17   โ”‚ | `echo '{ "foo": "bar" }' \| ./jql_optimized '"foo"'` | 4.4 ยฑ 0.2 | 4.1 | 6.4 | 1.00 |
  18   โ”‚
  19   โ”‚ | Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
  20   โ”‚ |:---|---:|---:|---:|---:|
  21   โ”‚ | `cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json \| ./jql_z '\|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null` | 20.6 ยฑ
       โ”‚  1.3 | 18.6 | 31.5 | 1.07 ยฑ 0.10 |
  22   โ”‚ | `cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json \| ./jql_opt_3 '\|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null` | 19
       โ”‚ .9 ยฑ 1.2 | 17.8 | 26.3 | 1.04 ยฑ 0.09 |
  23   โ”‚ | `cat /home/zamazan4ik/open_source/bench_jql/github-repositories.json \| ./jql_optimized '\|>{"name", "url", "language", "stargazers_count", "watchers_count"}' > /dev/null`
       โ”‚ | 19.3 ยฑ 1.3 | 17.1 | 26.2 | 1.00 |
  24   โ”‚
โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€

According to the tests, sometimes user time is significantly decreased with the PGO optimization.

Further steps

I can suggest the following things to do:

  • Evaluate PGO's applicability to jql in more scenarios.
  • If PGO helps to achieve better performance - add a note to jql's documentation about that (probably somewhere in the README file). In this case, users and maintainers will be aware of another optimization opportunity for jql.
  • Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.
  • Optimize prebuilt binaries with PGO.

Here are some examples of how PGO is already integrated into other projects' build scripts:

After PGO, I can suggest evaluating LLVM BOLT as an additional optimization step after PGO.

Option for validation

First, thanks for the tool, great to see a binary option for working with json that does not require a VM or anything alike!

I noticed there is no option to validate a JSON file without providing a selector. I often have the case that I have to work with files from somewhere else and often they are not even valid JSON so I either manually or in pipelines validate the file first.

Was there a discussion about a feature like this as well? Couldn't find one in the closed issues so far.

State the performance

I suggest you to state whether jql is fast or not, and if the throughput is one of the goals of the project. As simple as it sounds, it may drive adoption.

Publishing benchmarks result may also help.

Good luck!

Crash when piping to a pager like less, bat

jql crashes for me when piping the output to a pager like less or bat and the output is not fully read (because it's longer than the viewport). I'm running version 2.9.3. installed via brew on macos (latest version).

This works as it only provides very few lines of output for my example file:

RUST_BACKTRACE=1 jql '.!'  tmp.json | less

However, when outputting a bit more, it crashes when I quit the pager without scrolling to the end:

RUST_BACKTRACE=1 jql '.'  tmp.json | less
thread 'main' panicked at 'failed printing to stdout: Broken pipe (os error 32)', library/std/src/io/stdio.rs:1021:9
stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: jql::render_output
   3: <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll
   4: jql::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[1]    60860 abort      RUST_BACKTRACE=1 jql '.' tmp.json |
       60861 done       less

When I scroll the less to the end (e.g. by pressing G key) or when I pipe to cat which also reads everything, there's no issue.

Please let me know when you need more information.

Windows release artifact always fails to upload

the Windows release artifact of github action always fails to upload.

release workflow log

 Run softprops/action-gh-release@v1
  with:
    files: /d/a/jql/jql/jql-v5.1.1-x86_64-pc-windows-msvc.zip
  /d/a/jql/jql/jql-v5.1.1-x86_64-pc-windows-msvc.zip.sha[2](https://github.com/yamafaktory/jql/actions/runs/3255090557/jobs/5344040326#step:6:2)56
  
    token: ***
  env:
    ASSET_PATH: /d/a/jql/jql/jql-v5.1.1-x86_6[4](https://github.com/yamafaktory/jql/actions/runs/3255090557/jobs/5344040326#step:6:4)-pc-windows-msvc.zip
    CHECKSUM_PATH: /d/a/jql/jql/jql-v[5](https://github.com/yamafaktory/jql/actions/runs/3255090557/jobs/5344040326#step:6:5).1.1-x8[6](https://github.com/yamafaktory/jql/actions/runs/3255090557/jobs/5344040326#step:6:6)_64-pc-windows-msvc.zip.sha256
    GITHUB_TOKEN: ***
๐Ÿค” Pattern '/d/a/jql/jql/jql-v5.1.1-x[8](https://github.com/yamafaktory/jql/actions/runs/3255090557/jobs/5344040326#step:6:8)6_64-pc-windows-msvc.zip' does not match any files.
๐Ÿค” Pattern '/d/a/jql/jql/jql-v5.1.1-x86_64-pc-windows-msvc.zip.sha256' does not match any files.
๐Ÿค” /d/a/jql/jql/jql-v5.1.1-x86_64-pc-windows-msvc.zip,/d/a/jql/jql/jql-v5.1.1-x86_64-pc-windows-msvc.zip.sha256 not include valid file.
๐ŸŽ‰ Release ready at https://github.com/yamafaktory/jql/releases/tag/v5.1.1

Windows release artifacts fail to upload since v5.1.3

The release builds are no longer including the pre-built binaries for Windows (since v5.1.3) due to a breaking change in the softprops/action-gh-release github action used.

Release

  Run softprops/action-gh-release@v1
    with:
      files: D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip
    D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip.sha256
    
      token: ***
    env:
      ASSET_PATH: D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip
      CHECKSUM_PATH: D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip.sha256
      GITHUB_TOKEN: ***
  ๐Ÿค” Pattern 'D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip' does not match any files.
  ๐Ÿค” Pattern 'D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip.sha256' does not match any files.
  ๐Ÿค” D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip,D:\a\jql\jql\jql-v5.1.3-x86_64-pc-windows-msvc.zip.sha256 not include valid file.
  ๐ŸŽ‰ Release ready at https://github.com/yamafaktory/jql/releases/tag/v5.1.3

(full release workflow log)

Sometime mid November, softprops/action-gh-release@v1 updated the node-glob dependency to 8.0, which changed the behavior of \ to only be an escape character rather than a path separator.

More info here:
softprops/action-gh-release#280

I've tried 2 different workarounds to fix Windows binaries:

  1. pin version of softprops/action-gh-release to v0.1.13 instead of v1:
  uses: softprops/[email protected]

until original issue will be fixed;
(tested here)

  1. switch to forward slashes in PATH-variables in Windows case:
Run softprops/action-gh-release@v1
  with:
    files: D:/a/jql/jql/jql-v6.0.7-x86_64-pc-windows-msvc.zip
  D:/a/jql/jql/jql-v6.0.7-x86_64-pc-windows-msvc.zip.sha256
  
    token: ***
  env:
    ASSET_PATH: D:/a/jql/jql/jql-v6.0.7-x86_64-pc-windows-msvc.zip
    CHECKSUM_PATH: D:/a/jql/jql/jql-v6.0.7-x86_64-pc-windows-msvc.zip.sha256
    GITHUB_TOKEN: ***
โฌ†๏ธ Uploading jql-v6.0.7-x86_64-pc-windows-msvc.zip...
โฌ†๏ธ Uploading jql-v6.0.7-x86_64-pc-windows-msvc.zip.sha256...
๐ŸŽ‰ Release ready at https://github.com/aliesbelik/jql/releases/tag/jql-v6.0.7

(tested here)

and both completed successfully.

Just in case you are not planning to discount Windows binary option. ๐Ÿ˜จ

Support for filtering children

I might just not know how to do this, but I'm interested in being able to query for top level properties without seeing their children, is this possible?

I'm attempting to query a large json file that has an unknown structure to me.

Proposal: pre-compile selectors

I'm looking for jq like solution to be used as a library in the hot inner loop to crunch massive amount of data.
It would be nice to expose additional functions to pre-compile selectors and re-use them to get parsing out of the inner loop.

Add LICENSE files

It would be very useful since that there are tens of different variations of MIT licenses.

I would appreciate if you could publish new version after that.

README example doesn't return the same output

jql '"laptops".[1:0]|"laptop"|"brand","laptops"|"laptop"|"brand"' example.json

returns

[
  [
    "Asus",
    "Apple"
  ],
  [
    "Apple",
    "Asus"
  ]
]

while README says that it should return

[
  "Apple",
  "Asus"
]

lens conjunction doesn't work

I would expect the following to output only the second element:

printf '[{"cond1": "false", "cond2": "true"}, {"cond1": "true", "cond2": "true"}]' | jql '.[]|={"cond1":"true"}|={"cond2":"true"}'

but it outputs

[
  {
    "cond1": "false",
    "cond2": "true"
  },
  {
    "cond1": "true",
    "cond2": "true"
  }
]

How do you do a conjunction on the lens conditions?

Streaming by line inconsistencies

Hey @yamafaktory, first of all, thank you! JQL is fantastic

However, I've noticed issues when streaming logs into JQL. I've looked at #26 and tried every option I can think of but I'm getting zero output.

This works as expected:

docker logs container_name 2>&1 | jql '.!'

Fails when trying to tail via docker logs -f

docker logs -f container_name 2>&1 | jql '.!'

Right now I'm using JQ to process them, (docker logs -f container_name 2>&1 | jq '.') but I honestly love the '.!' option.

Any ideas on how to make this work? Wondering if it's similar to many JQ comments suggesting you need --unbuffered --stream.

Cannot process some json as expected

I can make it work with jq, but not with jql ๐Ÿ˜ญ, do you thinks it's possible or jql miss some feature that could make this possible?

Working jq command

jq -r '[ .[][][] ]'

Input

[
  {
    "key1": [
      {
        "subkey1": "value"
      }
    ]
  },
  {
    "key2": [
      {
        "subkey2": "value"
      }
    ]
  }
]

Expected output

[
  {
    "subkey1": "value"
  },
  {
    "subkey2": "value"
  }
]

ranges on top-level arrays are not handeled properly

data-input

(shortened from https://api.github.com/orgs/tauri-apps/repos?per_page=100&type=sources):

[
  {
    "id": 196701619,
    "node_id": "MDEwOlJlcG9zaXRvcnkxOTY3MDE2MTk=",
    "name": "tauri",
    "full_name": "tauri-apps/tauri",
    "html_url": "https://github.com/tauri-apps/tauri",
    "open_issues_count": 55,
    "forks": 51,
    "open_issues": 55,
    "watchers": 1629,
    "default_branch": "dev"
  },
  {
    "id": 206315614,
    "node_id": "MDEwOlJlcG9zaXRvcnkyMDYzMTU2MTQ=",
    "name": "examples",
    "full_name": "tauri-apps/examples",
    "html_url": "https://github.com/tauri-apps/examples",
    "open_issues_count": 7,
    "forks": 4,
    "open_issues": 7,
    "watchers": 6,
    "default_branch": "master"
  }
]

example of the bug

jql '[0]."name"' org.json
"tauri"

jql '[0:]."name"' org.json
Node "name" not found on parent range [0:]

Syntax: Is it necessary to have quotes and dots as delimiters?

Looking at one of the first examples:
jql '"primes".[0]' example.json

My first thought was... could it have just been:
jql 'primes[0]' example.json

(no quotes and no period separator)

I can understand the need for the quotes around a key that contains a space or special character, but in most cases names will not have such things.

I figure you have some reasoning behind this, so I just wanted to ask as I didn't see any other issues asking this.

Sort keys of objects

The sorting flag from jq is very helpful when dealing with JSON APIs.

  -S               sort keys of objects on output;

There is an example:

โฏ echo '{ "foo": 1, "bar": 2, "nested": { "will": 1, "sort": 2, "as": 3, "well": 4 } }' | jq -S .
{
  "bar": 2,
  "foo": 1,
  "nested": {
    "as": 3,
    "sort": 2,
    "well": 4,
    "will": 1
  }
}

Can we have this feature as well? Also, as a new rustacean I really appreciate your work. I had some issue to link lib-jq on my Mac, so this crate will save me a lot of time.

Format the json output

I have a return value like

[
  ".editorconfig",
  ".gitattributes",
  ".gitignore",
  ".revision-hash",
  ".travis.yml",
  ".version",
  "COPYING.md",
  "HACKING.md",
  "INSTALL.md",
  "Makefile",
  "README.md",
  "changelog.md",
  "docs",
  "docs/highlighters.md",
  "docs/highlighters",
  "docs/highlighters/brackets.md",
  "docs/highlighters/cursor.md",
  "docs/highlighters/line.md"
]

I want the return value to be like

.editorconfig
.gitattributes
.gitignore
.revision-hash
.travis.yml
.version
COPYING.md
HACKING.md
INSTALL.md
Makefile
README.md
changelog.md
docs
docs/highlighters.md
docs/highlighters
docs/highlighters/brackets.md
docs/highlighters/cursor.md
docs/highlighters/line.md

The current command I am using is:

curl -H "Accept: application/vnd.github.full+json" -H "Authorization: 7c258a9869f33c1e1e1f74fbb32f07c86cb5a75b" "https://api.github.com/repos/zsh-users/zsh-syntax-highlighting/git/trees/master?recursive=1" | jql '"tree"|"path"'

Benchmark and query question

I edited the performance.sh with a benchmark to test jql an jq with a large, approximately 5.9 MB, json file (First 1000 Microsoft repos on github).

#!/usr/bin/env bash

REPORT=$(pwd)/PERFORMANCE.md
PERFORMANCE_TMP_DIR=$(pwd)/performance_tmp
MD_FILES="$PERFORMANCE_TMP_DIR/"*".md"
MIN_RUNS=1000
JSON=$(pwd)/ms.json
JSON_TMP=$(pwd)/tmp.json

# Remove export file if present.
rm -f $REPORT

# Create the directory.
mkdir $PERFORMANCE_TMP_DIR

# Get a large json file
if [ ! -f $JSON ]; then
    for PG in {1..10}; do
        echo -ne "\rGettings microsoft github repos ... ${PG}0% "
        curl -S -s -H "Accept: application/vnd.github.v3+json" \
        https://api.github.com/orgs/microsoft/repos?per_page=100\&page=$PG \
        -o $JSON_TMP
        cat $JSON_TMP >> $JSON 
        rm $JSON_TMP
    done
    echo -e "\nms.json is $(du -h ${JSON} | | awk '{print $1}')"
fi

Bench1="'{\"foo\": \"bar\"}'"
Bench2="'[1, 2, 3]'"
Bench3="'[1, [2], [[3]]]'"

# Run the benchmarks.
hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/OBJECT.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench1 | jq '.foo'" \
    "echo $Bench1 | jql '.\"foo\"'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/ARRAY_INDEX.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench2 | jq '.[0]'" \
    "echo $Bench2 | jql '.[0]'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/ARRAY_FLATTEN.md" \
    --min-runs $MIN_RUNS \
    "echo $Bench3 | jq 'flatten'" \
    "echo $Bench3 | jql '...'"

hyperfine \
    --export-markdown "$PERFORMANCE_TMP_DIR/LARGE_JSON.md" \
    --min-runs $MIN_RUNS \
    "cat $JSON | jq -r '.[] | .name, .url, .language, .stargazers_count, .watchers_count'" \
    "cat $JSON | jql '.|{\"name\",\"url\", \"language\", \"stargazers_count\", \"watchers_count\"}'"

# Merge all the markdown files into the performance one.
for md_file in $MD_FILES; do (cat "${md_file}"; echo) >> $REPORT; done

# Remove the directory.
rm -R -f $PERFORMANCE_TMP_DIR
Command Mean [ms] Min [ms] Max [ms] Relative
cat /home/doc/Code/Bench/ms.json | jq -r '.[] | .name, .url, .language, .stargazers_count, .watchers_count' 104.0 ยฑ 9.8 89.2 121.3 1.00
cat /home/doc/Code/Bench/ms.json | jql '.|{"name","url", "language", "stargazers_count", "watchers_count"}' 2317.6 ยฑ 178.6 1801.3 2653.9 22.29 ยฑ 2.71

jql ended up being much slower so I figure I probably did something wrong. How can I make this a fairer comparison?

index out of bounds panic in array_walker

Given this json response from http://api.worldbank.org/v2/country/br?format=json

[
    {
      "page": 1,
      "pages": 1,
      "per_page": "50",
      "total": 1
      },
      [
        {
          "id": "BRA",
          "iso2Code": "BR",
          "name": "Brazil",
          "region": {
            "id": "LCN",
            "iso2code": "ZJ",
            "value": "Latin America & Caribbean (all income levels)"
          },
          "adminregion": {
            "id": "LAC",
            "iso2code": "XJ",
            "value": "Latin America & Caribbean (developing only)"
          },
          "incomeLevel": {
            "id": "UMC",
            "iso2code": "XT",
            "value": "Upper middle income"
          },
          "lendingType": {
            "id": "IBD",
            "iso2code": "XF",
            "value": "IBRD"
          },
          "capitalCity": "Brasilia",
          "longitude": "-47.9292",
          "latitude": "-15.7801"
        }
      ]
    ]

I can get incomeLevel by:

$ jql '[1].[0]."""incomeLevel"""."""value"""' br.json

However, when I try to fetch an out-of-bounds value, instead of failing gracefully, jql panics:

$ jql '[2].[0]."""incomeLevel"""."""value"""' br.json

thread 'main' panicked at 'index out of bounds: the len is 4 but the index is 18446744073709551615', C:\Users\jsmith\.cargo\registry\src\github.com-1ecc6299db9ec823\jql-3.1.1\src\array_walker.rs:37:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Panic when slicing empty array

When attempting to slice an empty array, jql panics.

Given the input {"data":[]} and the query "data".[], jql panics.
The desired output would be []

Example:

$ echo '{"data":[]}' | RUST_BACKTRACE=full jql '"data".[]' 
thread 'main' panicked at 'attempted to index slice up to maximum usize', src\libcore\slice\mod.rs:2597:5           
stack backtrace:                                                                                                      
0:     0x7ff722cde087 -                                                                                    
1:     0x7ff722cbd5ab -                                                                                   
2:     0x7ff722cdd127 -                                                                                    
3:     0x7ff722cdcd2c -                                                                                   
4:     0x7ff722cdc47d -                                                                                    
5:     0x7ff722ce2461 -                                                                                    
6:     0x7ff722ce7d79 -                                                                                   
7:     0x7ff722cbb709 -                                                                                    
8:     0x7ff722cbbb8b -                                                                                   
9:     0x7ff722cbdf90 -                                                                                   
10:     0x7ff722cc9e71 -                                                                                   
11:     0x7ff722cc8ba4 -                                                                                   
12:     0x7ff722cc86e2 -                                                                                   
13:     0x7ff722cc7e61 -                                                                                  
14:     0x7ff722cc48ca -                                                                                   
15:     0x7ff722cc38e0 -                                                                                   
16:     0x7ff722cc3306 -                                                                                   
17:     0x7ff722c85983 -                                                                                   
18:     0x7ff722c87ca4 -                                                                                   
19:     0x7ff722c8251f -                                                                                  
20:     0x7ff722c88f6d -                                                                                   
21:     0x7ff722ce8e34 -                                                                                   
22:     0x7ffd334a37e4 - BaseThreadInitThunk                                                                        
23:     0x7ffd35f5cb81 - RtlUserThreadStart                                                                      
Illegal instruction

Version:

$ jql --version                                                                                                     
jql 2.4.6

$rustc --version                                                                                                   
rustc 1.39.0 (4560ea788 2019-11-04)

Platform:
stable-x86_64-pc-windows-msvc

Access children of an array without using an array index

Hi I am trying to use jql to parse some elasticsearch query outputs but it seems its unable to fetch children of an array without an index.

Example:

curl localhost:9200/index/_search | jql '"hits"."hits".[]."_source"'

The output from the curl looks like this:

{
  "hits": {
    "hits": [
      {
        "_id": "b541465c-fd6a-11e8-becd-8c859022e616",
        "_index": "index",
        "_score": 1.0,
        "_source": {
            "key": "val"
        },
        "_type": "index_type"
      }
   ]
}

jql will return the entire object in the array instead of just the _source field. Though when I use an index it seems to work:

curl localhost:9200/index/_search | jql '"hits"."hits".[0]."_source"'

Thanks, awesome library, hopefully I might have sometime to fix this.

How does it compare to `jq`?

Is it expected to be a drop-in replacement? Or just inspired by it? Any difference besides being implemented in Rust?

jql 6.0.3 test failure

๐Ÿ‘‹ trying to upgrade jql to the latest release, 6.0.3, but failed with the following failure

jql --raw-string --inline  '"cats".[2:1].[0]."third"' /private/tmp/jql-test-20230422-25743-2ghswh/example.json
Unable to parse input .[2:1].[0]."third" after Key Selector "cats"

While just "cats" as QUERY string, it works out fine

jql --raw-string --inline  '"cats"' /private/tmp/jql-test-20230422-25743-2ghswh/example.json
[{"first":"Pixie"},{"second":"Kitkat"},{"third":"Misty"}]

relates to Homebrew/homebrew-core#129018

lens conjunction doesn't work

Discussed in #171

Originally posted by LLFourn August 4, 2022
I would expect the following to output only the second element:

printf '[{"cond1": "false", "cond2": "true"}, {"cond1": "true", "cond2": "true"}]' | jql '.[]|={"cond1":"true"}|={"cond2":"true"}'

but it outputs

[
  {
    "cond1": "false",
    "cond2": "true"
  },
  {
    "cond1": "true",
    "cond2": "true"
  }
]

How do you do a conjunction on the lens conditions?

Add support for raw-output.

Is there any plan to add support for raw input and raw-output? That will be really helpful to use this for CLI usage.

Example for how jq provides this:

$ echo '{"foo": "bar"}' | jq '.foo'
"bar"

$ echo '{"foo": "bar"}' | jq -r '.foo'
bar

An example of working with arrays in the docs?

The readme for this project is quite good and I've been been very productive with it so far.

However, I'm having problems getting jql to print only the values. The closest I can get is screened below.

My question is, how does one start by selecting all the objects in an array then filtering out by keys, then displaying the children?

I haven't been able to figure it out from looking at the docs.
[
{
"txid": "15a10e97de1dca95dfeb3d8d06dac87dbb494eab8acc00abca0f96c140b97a73",
"vout": 0,
"address": "tb1qd2cvqk30f8kw3hj53wlj053dd9amnhhefg02gx",
"label": "",
"scriptPubKey": "00146ab0c05a2f49ece8de548bbf27d22d697bb9def9",
"amount": 0.001,
"confirmations": 61,
"spendable": true,
"solvable": true,
"desc": "wpkh([b9a5ac96/0'/0'/2']03d94b78001ec7bdfffe41cca947e77219342272bb62946af779311358f835e57f)#3e78xplt",
"safe": true
},
{
"txid": "393e1b08b7d2afd21f41f23025f5d79a06a7d12bc95a598de8b3c2f0340293a0",
"vout": 1,
"address": "tb1qd2cvqk30f8kw3hj53wlj053dd9amnhhefg02gx",
"label": "",
"scriptPubKey": "00146ab0c05a2f49ece8de548bbf27d22d697bb9def9",
"amount": 0.001,
"confirmations": 61,
"spendable": true,
"solvable": true,
"desc": "wpkh([b9a5ac96/0'/0'/2']03d94b78001ec7bdfffe41cca947e77219342272bb62946af779311358f835e57f)#3e78xplt",
"safe": true
},
{
"txid": "faa64c516c90302c624c7f7ffc26366376ac12813b1cc6e4eb8a192e3e0f08e4",
"vout": 0,
"address": "tb1qtlefx6f36vyx4l4g2wyvas4ghyuynn46xptyya",
"label": "",
"scriptPubKey": "00145ff2936931d3086afea85388cec2a8b93849ceba",
"amount": 0.0022,
"confirmations": 61,
"spendable": true,
"solvable": true,
"desc": "wpkh([b9a5ac96/0'/0'/1']03a430c19febf6d59786fe600ff185e2e8c4efa81212ead8f5a741a9427090e766)#9lt0k576",
"safe": true
}
]

Final Goal;
$ bt listunspent | jql -r 'something something something'
15a10e97de1dca95dfeb3d8d06dac87dbb494eab8acc00abca0f96c140b97a73
0
0.001
393e1b08b7d2afd21f41f23025f5d79a06a7d12bc95a598de8b3c2f0340293a0
1
0.001
faa64c516c90302c624c7f7ffc26366376ac12813b1cc6e4eb8a192e3e0f08e4
0
0.0022

selecting arrays

Support TOML

Json is nice, but what about other formats? For example, TOML.

Allow streaming by line

File a:

{"a":1, "b":2}
{"a":3, "b":4}
{"a":5, "b":6}

< a jql '.'
Currently says Invalid JSON file or content

It should take every line as a record and run the query (as with jq).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.