Giter Site home page Giter Site logo

trentm / json Goto Github PK

View Code? Open in Web Editor NEW
1.6K 29.0 121.0 1.04 MB

A "json" command for massaging JSON on your Unix command line.

Home Page: https://trentm.com/json

License: Other

Makefile 3.26% Python 3.02% JavaScript 68.55% Shell 1.02% Perl 24.15%

json's Introduction

json is a fast CLI tool for working with JSON. It is a single-file node.js script with no external deps (other than node.js itself). A quick taste:

$ echo '{"foo":"bar"}' | json
{
  "foo": "bar"
}

$ echo '{"foo":"bar"}' | json foo
bar

$ echo '{"fred":{"age":42}}' | json fred.age    # '.' for property access
42

$ echo '{"age":10}' | json -e 'this.age++'
{
  "age": 11
}

# `json -ga` (g == group, a == array) for streaming mode
$ echo '{"latency":32,"req":"POST /widgets"}
{"latency":10,"req":"GET /ping"}
' | json -gac 'this.latency > 10' req
POST /widgets

Features:

  • pretty-printing JSON
  • natural syntax (like JS code) for extracting particular values
  • get details on JSON syntax errors (handy for config files)
  • filter input JSON (see -e and -c options)
  • fast stream processing (see -ga)
  • JSON validation
  • in-place file editing

See https://trentm.com/json for full docs and examples as a man page.

Follow @trentmick for updates to json.

Installation

  1. Get node.

  2. npm install -g json

    Note: This used to be called 'jsontool' in the npm registry, but as of version 8.0.0 it has taken over the 'json' name. See npm Package Name below.

OR manually:

  1. Get the 'json' script and put it on your PATH somewhere (it is a single file with no external dependencies). For example:

     cd ~/bin
     curl -L https://github.com/trentm/json/raw/master/lib/json.js > json
     chmod 755 json
    

You should now have "json" on your PATH:

$ json --version
json 9.0.0

WARNING for Ubuntu/Debian users: There is a current bug in Debian stable such that "apt-get install nodejs" installed a nodejs binary instead of a node binary. You'll either need to create a symlink for node, change the json command's shebang line to "#!/usr/bin/env nodejs" or use chrislea's PPA as discussed on issue #56. You can also do "apt-get install nodejs-legacy" to install symlink for node with apt.

Test suite

npm test   # or 'make test'

This is using node-tap, so you can use all its options, for example filtering which tests to run:

npm test -- -g stream

License

MIT (see the fine LICENSE.txt file).

Module Usage

Since v1.3.1 you can use "json" as a node.js module:

var json = require('json');

However, so far the module API isn't that useful and the CLI is the primary focus.

npm Package Name

Once upon a time, json was a different thing (see zpoley's json-command here), and this module was called jsontool in npm. As of version 8.0.0 of this module, npm install json means this tool.

If you see documentation referring to jsontool, it is most likely referring to this module.

Alternatives you might prefer

json's People

Contributors

afanasy avatar ainthek avatar antonmedv avatar felixrabe avatar fkuo avatar inator avatar isaacs avatar pijewski avatar quentinvaut avatar trentm avatar twhiteman avatar vi avatar yaniv-aknin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

json's Issues

json is broken on 0.5 builds of node

$ json:223
data = Script.runInNewContext("(" + JSON.stringify(data) + ")" + lookup)
^
TypeError: Cannot call method 'runInNewContext' of undefined
at Socket. (/Users/mark/work/foo/json:223:21)
at Socket.emit (events.js:61:17)
at Socket._onReadable (net.js:657:51)
at IOWatcher.onReadable as callback

json is broken on 0.5 builds of node

$ json:223
data = Script.runInNewContext("(" + JSON.stringify(data) + ")" + lookup)
^
TypeError: Cannot call method 'runInNewContext' of undefined
at Socket. (/Users/mark/work/foo/json:223:21)
at Socket.emit (events.js:61:17)
at Socket._onReadable (net.js:657:51)
at IOWatcher.onReadable as callback

json that is a hash with a key like 'platform-release' breaks

My json:

{
"platform-release": "develop"
}

My command:

cat file.json | ./bin/json 'platform-release'

Expected result:

develop

Actual result:

/path/to/json:175
data = Script.runInNewContext("(" + JSON.stringify(data) + ")" + lookup)
^
ReferenceError: release is not defined
at evalmachine.:1:33
at Stream. (/path/to/json:175:21)
at Stream.emit (events:27:15)
at IOWatcher.callback (net:471:53)
at node.js:773:9

json 1.1.3 broke using bracket notation

with json 1.1.3:

$ echo '{"foo-bar": "baz"}' | json '["foo-bar"]'

/.../json:221
      data = Script.runInNewContext("(" + JSON.stringify(data) + ")" + lookup)
                    ^
TypeError: Cannot read property 'foo-bar' of undefined
    at evalmachine.<anonymous>:1:24
    at Stream.<anonymous> (json:221:21)
    at Stream.emit (events:28:17)
    at IOWatcher.callback (net:471:53)
    at node.js:773:9

`echo '{"foo": "bar"}' | ./json nosuchkey` goes boom

$ echo '{"foo": "bar"}' | ./json nosuchkey

/Users/trentm/tm/json/json:354
    if (output.length) {
              ^
TypeError: Cannot read property 'length' of null
    at Socket.<anonymous> (/Users/trentm/tm/json/json:354:15)
    at Socket.emit (events.js:61:17)
    at Socket._onReadable (net.js:652:51)
    at IOWatcher.onReadable [as callback] (net.js:177:10)

Would like ability to use -i with color disabled

I ran into a problem where I was using json in a shell script and I wanted the -i option to distinguish between a '[]' and [], but when using the -i I am also forced to get color output which causes problems in my script. This is forced as true is hardcoded in the util.inspect call for the color option.

I'd like the ability to set an environment variable at least which will disable color on the output. It could also be that json could be a bit smarter about when to use color (checking the terminal is a tty, etc.) but for now an env var would be plenty.

In this case what I had to do was use json to parse the field I want then remove all the color formatting in my bash script.

doesn't always flush all output

json doesn't always flush all output to stdout. For example:

$ cat package.json 
{
  "name": "xxxx-xxxxx",
  "version": "6.1.0",
  "private": true,
  "dependencies": {
    "dirsum": "0.1.1",
    "httpu": "0.0.1",
    "nopt": "1.0.6",
    "node-uuid": "1.2.0",
    "restify": "0.3.15",
  },
  "devDependencies": {
    "jshint": "0.2.3"
  },
  "bin": {
    "amon-agent": "./bin/amon-agent"
  },
  "scripts": {
    "postactivate": "smf_scripts/postinstall.sh",
    "predeactivate": "smf_scripts/preuninstall.sh"
  }
}
$ cat package.json  | json
json: error: doesn't look like JSON: SyntaxError: Unexpected token } (buffer="{\n  \"name\": \"xxxx-xxxxx\",\n  \"version\": \"6.1.0\",\n  \"private\": true,\n  \"dependencies\": {\n    \"dirsum\": \"0.1.1\",\n    \"httpu\": \"0.0.1\",\n    \"nopt\": \"1.0.6\",\n    \"node-uuid\": \"1.2.0\",\n    \"restify\": \"0.3.15\",\n  },\n  \"devDependencies\": {\n    \"jshint\": \"0.2.3\"\n  },\n  \"bin\": {\n    \"amon-agent\": \"./bin/amon-agent\"\n  },\n  \"scripts\": {\n    \"postactivate\": \"smf_scripts/postinstall.sh\",\n    \"predeactivate\": \"smf_scripts/preuninstall.sh\"\n  }\n}\n")
{
  "name": "xxxx-xxxxx",
  "version": "6.1.0",
  "private": true,
  "dependencies": {
    "dirsum": "0.1.1",
    "httpu": "0.0.1",
    "nopt": "1.0.6",
    "node-uuid": "1.2.0",
    "restify": "0.3.15",
  },
  "devDependencies": {
    "jshint": "0.2.3"
  },
  "bin": {
    "amon-agent": "./bin/amon-agent"
  },
  "scripts": {
    "postactivate": "smf_scripts/postinstall.sh",
    "predeactivate": "smf_scripts/pr$ 

It seems to depend on the input content. In any case, we should flush stdtou on exit.

input is not JSON

It would be great if the tool supported input that was javascript object and not JSON, converting it to JSON

{
  userId: "foobar"
}

note, without quotes.

grouping of arrays (-g) fails on empty arrays: `[]`

$ cat arrays.chunks
[
    {"one": 1}
]
[
    {"two": 2}
]
[]
[]

$ json -f arrays.chunks -g
json: error: input is not JSON: Syntax error at line 4, column 1:
        [
        ^
[
    {"one": 1}
]
[
    {"two": 2}
]
[]
[]

would be nice to support this.

json 2 "-a" output is wrong if requested key doesn't exist in one of the items

Get this:

$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -a a
1
2
$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -a a b
1 2 3
$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -a a b -d,
1,2,3

Want this:

$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -a a
1
2
$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -a a b
1 
2 3
$ echo '[{"a":1}, {"a":2, "b":3}]' | bin/json -d, -a a b
1,
2,3

can't get a key with a period ('.')

$ echo '{"foo.bar": 42}' | json '["foo.bar"]'

/Users/trentm/tm/json/json:223
      data = Script.runInNewContext("(" + JSON.stringify(data) + ")" + lookup)
                    ^
SyntaxError: Unexpected token ILLEGAL
    at Socket.<anonymous> (/Users/trentm/tm/json/json:223:21)
    at Socket.emit (events.js:39:17)
    at Socket._onReadable (net.js:629:51)
    at IOWatcher.onReadable [as callback] (net.js:156:10)

Would like 'this' set for conditional processing on arrays

I would like to be able to act on each member of an array, in turn, as part of conditional/exec processing. It would be neat if each member of an array could be broken out, inserted into the context as this, and the resulting filtered/modified array reconstituted.

For example, with a dataset like this:

[
  "some text",
  "some more text",
  "a third string"
]

It would be cool to be able to say:

json -f input_array.json -ga -c "this.match(/some/)"

And receive the output:

[
  "some text",
  "some more text"
]

Merge doesn't work if first { is followed by newline

Using version 1b0951f for merging, if the first { is followed by a newline (\n), json dies with syntax error. Unfortunately, it's a common case with indented JSON input files. Here's an example:

$ echo -ne '{"key": {"foo": "bar"}}{\n"baz": "qux"}' | json --deep-merge
json: error: input is not JSON: Syntax error at line 1, column 24:
        {"key": {"foo": "bar"}}{
        .......................^
{"key": {"foo": "bar"}}{
"baz": "qux"}

Removing the newline leads to normal behavior:

$ echo -ne '{"key": {"foo": "bar"}}{"baz": "qux"}' | json --deep-merge
{
  "key": {
    "foo": "bar"
  },
  "baz": "qux"
}

`-g|--group` option is unsafe for `][` in a string

$ cat foo.json
["foo"]
["bar"]

$ json -f foo.json -g
[
  "foo",
  "bar"
]

$ cat boom.json
["foo"]
["boom]["]
["bar"]

$ json -f boom.json -g
json: error: input is not JSON: Syntax error at line 2, column 1:
        ["boom]["]
        ^
["foo"]
["boom]["]
["bar"]

"OM_JSON" style output is just broken with a *single* lookup

[root@trent0b:~/joy/mountain-gorilla]$ echo '{"foo": {"bar": "car"}}' | json foo
{
  "bar": "car"
}
[root@trent0b:~/joy/mountain-gorilla]$ echo '{"foo": {"bar": "car"}}' | json foo -j
{
  "foo": {
    "bar": "car"
  }
}

That is just broken. The second time should have the same output. Need to revisit that 3.0 change.

json 3 bug handling lookup with a dot

$ echo '[
  {
    "name": "ba87e5e1-9c86-4c00-903c-050ee11e21f7",
    "alias": "amon0",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "ram": 256,
    "disk": 10240,
    "swap": 512,
    "cpu_shares": 256,
    "ips": [
      {
        "updated_at": "2012-03-29T20:30:37+00:00",
        "address": "10.2.207.13",
        "primary": true,
        "addressable_uuid": "ba87e5e1-9c86-4c00-903c-050ee11e21f7"
      }
    ],
    "hostnames": [],
    "credentials": [],
    "server": {
      "hostname": "headnode",
      "server_role_id": 1,
      "setup_at": "2012-03-29T20:18:40+00:00",
      "updated_at": "2012-04-09T22:54:45+00:00",
      "current_status": "running",
      "uuid": "44454c4c-3200-1042-804d-c2c04f575231",
      "vm_capable": true,
      "ip_address": "10.2.207.2",
      "ips": [
        "10.2.127.2",
        "10.2.207.2"
      ]
    },
    "primary_network": "/networks/1",
    "dataset_name": "smartos-1.3.18",
    "dataset_uuid": "47e6af92-daf0-11e0-ac11-473ca1173ab0",
    "dataset_urn": "sdc:sdc:smartos:1.3.18",
    "type": "zone",
    "tags": {
      "smartdc_role": "amon"
    },
    "server_id": 1,
    "running_status": "running",
    "zfs_io_priority": 10,
    "created": "2012-03-29T20:31:03+00:00",
    "updated": "2012-04-09T22:49:44+00:00",
    "internal_metadata": {
      "package_name": "sdc_256",
      "package_version": "1.0.0",
      "sdc:instance_image": "sdc:sdc:smartos:1.3.18 47e6af92-daf0-11e0-ac11-473ca1173ab0",
      "sdc:instance_uuid": "ba87e5e1-9c86-4c00-903c-050ee11e21f7"
    }
  }
]' | jsondev -a server.uuid

Please add the ability to spit multiple items out of an object to the argument pipeline

I want this to work:

bluesnoop:ufds mark$ ./tools/ufdssearch -u ldap://localhost:1389 -w secret --base "login=user_99, ou=customers, o=smartdc" -s base 
[
 {
   "dn": "login=user_99, ou=customers, o=smartdc",
   "address": [
     "Joyent, Inc.",
     "345 California Street, Suite 2000"
   ],
   "city": "San Francisco",
   "cn": "Test",
   "company": "markc",
   "country": "USA",
   "email": "[email protected]",
   "login": "user_99",
   "objectclass": "sdcPerson",
   "phone": "+1 415 400 0600",
   "postalcode": "94104",
   "sn": "User",
   "state": "CA",
   "uuid": "a249d2f2-5dd6-402f-8418-6a3209fa8568"
 }
]
bluesnoop:ufds mark$ ./tools/ufdssearch -u ldap://localhost:1389 -w secret --base "login=user_99, ou=customers, o=smartdc" -s base | json 0.dn 0.email | xargs ...

perhaps an option to accept "unstrict" JSON, as from sys.inspect

for example:

$ npm info irc 
{ name: 'irc',
  description: 'An IRC client library for node',
  'dist-tags': { latest: '0.2.0' },
  versions: [ '0.1.2', '0.2.0' ],
  maintainers: 'martyn <[email protected]>',
  author: 'Martyn Smith <[email protected]>',
  repository: 
   { type: 'git',
     url: 'git://github.com/martynsmith/node-irc.git' },
  time: 
   { '0.1.2': '2011-04-29T10:58:41.621Z',
     '0.2.0': '2011-04-29T10:58:41.621Z' },
  version: '0.2.0',
  contributors: [],
  bugs: 
   { email: '[email protected]',
     url: 'http://github.com/martynsmith/node-irc/issues' },
  main: 'lib/irc',
  engines: { node: '>=0.1.91' },
  licenses: 
   { type: 'GPL',
     url: 'http://github.com/martynsmith/node-irc/raw/master/COPYING' },
  dependencies: {},
  devDependencies: {},
  dist: 
   { shasum: '6b59b7ef30a16467247c7abb1d776687aaa936d3',
     tarball: 'http://registry.npmjs.org/irc/-/irc-0.2.0.tgz' },
  scripts: {},
  directories: {} }

Would be nice to be able to pipe that into json. Granted that perhaps this is featuritis and I should just browbeat Isaac into outputing actual JSON instead of sys.inspect with npm.

json -e cannot handle spaces in keys

daniel@ICN:~$ json --version
json 5.1.1
daniel@ICN:~$ echo '{"foo bar": "1"}' | json
{
  "foo bar": "1"
}
daniel@ICN:~$ echo '{"foo bar": "1"}' | json -e 'foo bar="2"'

vm.js:26
  return new exports.Script(code, ctx, name);
         ^
SyntaxError: Unexpected identifier
    at Object.createScript (vm.js:26:10)
    at parseChunk (/home/daniel/bin/json:1133:26)
    at /home/daniel/bin/json:1066:7
    at Socket.<anonymous> (/home/daniel/bin/json:618:7)
    at Socket.emit (events.js:64:17)
    at Pipe.onread (net.js:388:51)

piping json output through utilities that don't read stdin throws a stack trace

sdc-mapi /servers | json -H 0.uri | basename

.

net.js:391
throw new Error('Socket is not writable');

Error: Socket is not writable
at Socket._writeOut (net.js:391:11)
at Socket.write (net.js:377:17)
at Socket. (/usr/bin/json:336:20)
at Socket.emit (events.js:61:17)
at Socket._onReadable (net.js:650:51)
at IOWatcher.onReadable [as callback] net.js:177:10

'json -g -a' should stream

Description

In version 4.0.0 and previous, json attempts to build its entire result set in memory before sending output. This prevents its use in filtering and processing large Bunyan-style log files.

Steps to reproduce:

  • Feed a long newline-delimited JSON stream into json. If you're mean, make it infinitely long:

    yes '{"a":1}' | json -g -a -o inspect
    

Expected result:

  • json should start output more or less immediately:

    { a: 1 }
    { a: 1 }
    { a: 1 }
    
  • json should not consume all memory while trying to build result

Actual Result

  • json does not print any output

  • json attempts to build a result in memory, growing in VIRT and RES until it crashes:

    FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
    

regexps aren't callable (any more, as of node 0.5)

This fixes it:

diff --git a/bin/json b/bin/json
index 14c51c1..2a37446 100755
--- a/bin/json
+++ b/bin/json
@@ -268,7 +268,7 @@ function processDatum(datum, args) {
       var bit = bits[i];
       if (bit[0] === '[') {
         lookup += bit;
-      } else if (! isJSIdentifier(bits[i])) {
+      } else if (! isJSIdentifier.exec(bits[i])) {
         // Allow a non-JS-indentifier token, e.g. `json foo-bar`.
         lookup += '["' + bits[i].replace('"', '\\"') + '"]';
       } else {
@@ -330,7 +330,7 @@ function processDatumExperimental(datum, args) {
         data = newdata;
       } else if (bit[0] === '[') {
         lookup += bit;
-      } else if (! isJSIdentifier(bits[i])) {
+      } else if (! isJSIdentifier.exec(bits[i])) {
         // Allow a non-JS-indentifier token, e.g. `json foo-bar`.
         lookup += '["' + bits[i].replace('"', '\\"') + '"]';
       } else {

Inconsistent behavior for -a option

Why don't these 2 commands produce identical output?

$ echo '[1,2,3,4]' | json -a
1
2
3
4
$ echo '{"a": [1,2,3,4]}' | json -a a
[
  1,
  2,
  3,
  4
]

Undefined properties

I'm writing a script that sets up some shell variables

IMGDIR=`cat $PROJECTDIR/package.json | json paths.images`

except if paths.images doesn't exist I get the following error message

/usr/local/lib/node_modules/jsontool/lib/jsontool.js:496
  return vm.runInNewContext("(" + JSON.stringify(datum) + ")" + lookupCode);
            ^
TypeError: Cannot read property 'images' of undefined
    at evalmachine.<anonymous>:1:307
    at lookupDatum (/usr/local/lib/node_modules/jsontool/lib/jsontool.js:496:13)
    at Socket.<anonymous> (/usr/local/lib/node_modules/jsontool/lib/jsontool.js:804:23)
    at Socket.emit (events.js:64:17)
    at Pipe.onread (net.js:388:51)

Obviously it's a valid error, but it doesn't help in the context of this script. I think if there was something like a silent flag that didn't log errors? I'm not sure that's the best solution... but something like that

No SIGPIPE or EPIPE

yes '{"a":"b"}' | ./json -ga a | head -n 3
b
b
b
...

This command never end. It prints the first three lines and then it blocks because json refuses to die. I checked with strace and each write generates a SIGPIPE and a EPIPE error but json keeps trying to write in stdout. I tried to see if I could catch SIGPIPE with process.on, without success. The EPIPE error never seem to show up.

`json '"foo"'` is broken

I.e. lookups with multiple double-quotes to be escaped are broken.

$ echo '{"\"a\"": "first letter"}' | json '"a"'

.../bin/json:496
  return vm.runInNewContext("(" + JSON.stringify(datum) + ")" + lookupCode);
            ^
SyntaxError: Unexpected token ILLEGAL
    at lookupDatum (/Users/trentm/tm/dotfiles/home/bin/json:496:13)
    at Socket.<anonymous> (/Users/trentm/tm/dotfiles/home/bin/json:804:23)
    at Socket.emit (events.js:64:17)
    at Pipe.onread (net.js:388:51)

json should return non-zero on invalid input

It would be nice to be able to use "json" to validate that something is valid JSON. In that case, you'd use

... | json > /dev/null

So "json" would need to return non-zero in this case. Relatedly, scripts that are trying to extract data from a JSON payload should fail if "json" couldn't parse the input.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.