nytlabs/gojee

JSON expression evaluator

jee

jee (json expression evaluator) transforms JSON through logical and mathematical expressions. jee can be used from the command line or as a Go package. It is inspired by the fantastic (and much more fully featured) jq.

jee was created out of the need for a simple JSON query language in streamtools. jee is designed for stream processing and provides a reusable token tree.

####get the library

go get github.com/nytlabs/gojee

####make and install the binary

cd $gopath/src/github.com/nytlabs/gojee/jee
go install

usage (binary)

querying JSON

get the entire input object:

> echo '{"a": 3}' | jee '.'
{
    "a": 3
}

get a value for a specific key:

> echo '{"a": 3, "b": 4}' | jee '.a'
3

get a value from an array:

> echo '{"a": [4,5,6]}' | jee '.a[0]'
4

get all values from an array:

> echo '{"a": [4,5,6]}' | jee '.a[]'
[
    4,
    5,
    6
]

query all objects inside array 'a' for key 'id':

> echo '{"a": [{"id":"foo"},{"id":"bar"},{"id":"baz"}]}' | jee '.a[].id'
[
    "foo",
    "bar",
    "baz"
]

arithmetic

+ - * /

> echo '{"a": 10}' | jee '(.a * 100)/-10 * 5'
-500

comparison

> >= < <= !=

> echo '{"a": 10}' | jee '(.a * 100)/-10 * 5 == -500'
true
> echo '{"a": 10}' | jee '(.a * 100)/-10 * 5 > 0'
false

logical

|| &&

> echo '{"a": false}' | jee '!(.a && true) || false  == true'
true

functions

types

$num(x {bool, float64, string, nil})
Converts x to a float64. If x is a bool, 1 is returned for true and 0 for false. If x is nil, 0 is returned.

$str(x {bool, float64, string, nil, object, []*))
Converts x to a string. If x is a bool, "true" is returned for true and "false" for false. "null" is returned for nil. If x is an object or an array, it is marshaled into a JSON string.

$bool(x {bool, string})
Converts x to a bool. See strconv.ParseBool

$~bool(x {bool, float64, string, nil, object, []*})
Truthy conversion of x to a bool. Falsey values:null,NaN,0,false, and arrays with a length of 0.

math

$sqrt(x float64)
Returns square root of x.

$pow(x float64, y float64)
Returns x^y.

$floor(x float64)
Returns nearest downward integer for x.

$abs(x float64)
Returns absolute value of x.

arrays

$len(a []interface{})
Returns the length of array a.

$has( a {[]bool, []float64, []string, []nil}, val {bool, float64, string, nil} )
Checks to see if array a contains val. Returns bool. val cannot be an object.

$sum(a []float64)
Returns the sum of array a.

$min(a []float64)
Returns the minumum of array a.

$max(a []float64)
Returns the maximum of array a.

objects

$keys(o object)
Returns an array of keys in object o.

$exists(o object, key string)
Checks to see if key exists in map o. Returns bool. $exists() does a map lookup and is faster than $has($keys(o), "foo")

date and time

$now()
Returns current system time in float64 (epoch milliseconds).

$parseTime(layout string, t string)
Accepts a time layout in golang time format. t is parsed and returned as epoch milliseconds in float64.

$fmtTime(layout string, t float64)
Accepts a time layout in golang time format. t is expected in epoch milliseconds. Returns a formatted string.

strings

$contains(s string, substr string)
see strings.Contains

$regex(pattern string, s string)
see regexp.MatchString. Much slower than $contains()

see jee_test.go for examples.

package usage

#####Lexer(string) []*Token, error converts a jee query string into a slice of tokens

#####Parser([]*Tokens) *TokenTree, error builds a parse tree out token slice from Lexer()

#####Eval(*TokenTree, {}interface) {}interface, error evaluates a variable of type interface{} with a *TokenTree generated from Parser(). Only types given by json.Unmarshal are supported.

quirks

• Types are strictly enforced. false || "foo" will produce a type error.
• null and 0 are not falsey
• Using a JSON key as an array index or an escaped key in bracket notation will not currently be evaluated. ie: .a[.b]
• All numbers in a jee query must start with a digit. numbers <1 should start with a 0. use 0.1 instead of .1
• Bracket notation is available for keys that need escaping .["foo"]["bar"]]
• Queries for JSON keys or indices that do not exist return null (to test if a key exists, use $exists)
• jee does not support variables, conditional expressions, or assignment
• jee may be very quirky in general.

changes

• .0.1.1 addition of $bool, $~bool, $num, $str, $now, $fmtTime, $parseTime. Fix for non-alphanumeric characters in JSON keys.
• .0.1.0 initial release

gojee's Issues

ternary operation

$~bool(.a) == true ? "yes" : "no"

join()

join array of strings. useful for get URL arrays

type conversion functions

ability to $str( float64 )

return nil on eval case key

if we are in eval and

• the case is Key
• the input interface is nil

immediately return nil without calling getkeyvalues because we wont return anything else.

use go-style lexing/parsing

rewrite lexer so that it works like this: http://golang.org/src/pkg/text/template/parse/

• more robust error checking
• easier to read code
• easier to modify/change syntax
• it's the right thing to do

inline objects, arrays

gojee should accept script such as {"a": .A} or [.a,.b,.c]as part of its native syntax

Add functions to stringify and parse for json

Maybe something like:

$stringify({x: 5, y: 6})
$parse( '{"x":5,"y":6}')

string formatting and escaping

similar to jq's
@uri, @json, @html, etc

$max pukes on integer arrays

on recieving ints:

panic: interface conversion: interface is int, not float64

goroutine 1811 [running]:
runtime.panic(0x3cfca0, 0xc2102689c0)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/nytlabs/gojee.func·030(0x316260, 0xc2102cd580, 0xc2102c3590, 0x4, 0xc21005d4e8, ...)
    /Users/mikedewar/go/src/github.com/nytlabs/gojee/jee.go:622 +0xb9
github.com/nytlabs/gojee.Eval(0xc2102e7000, 0x327ee0, 0xc21009aea0, 0x21609, 0xc2102c36b0, ...)
    /Users/mikedewar/go/src/github.com/nytlabs/gojee/jee.go:1089 +0x2661
github.com/nytlabs/gojee.Eval(0xc2102b9180, 0x327ee0, 0xc21009aea0, 0xc2102b9180, 0xc210239901, ...)
    /Users/mikedewar/go/src/github.com/nytlabs/gojee/jee.go:1112 +0x2d24
github.com/nytlabs/streamtools/st/library.evalMap(0xc2101f9c90, 0xc21009aea0, 0xc2101f9c90, 0xc2101f9c90, 0x329020)
    /Users/mikedewar/go/src/github.com/nytlabs/streamtools/st/library/map.go:62 +0x2bd
github.com/nytlabs/streamtools/st/library.(*Map).Run(0xc21014cbb0)
    /Users/mikedewar/go/src/github.com/nytlabs/streamtools/st/library/map.go:154 +0x57c
created by github.com/nytlabs/streamtools/st/blocks.BlockRoutine
    /Users/mikedewar/go/src/github.com/nytlabs/streamtools/st/blocks/blocks.go:202 +0xef

To recreate in streamtools:

{
  "Connections": [
    {
      "ToRoute": "poll",
      "ToId": "2",
      "FromId": "5",
      "Id": "6"
    },
    {
      "ToRoute": "in",
      "ToId": "3",
      "FromId": "2",
      "Id": "7"
    },
    {
      "ToRoute": "in",
      "ToId": "4",
      "FromId": "3",
      "Id": "8"
    },
    {
      "ToRoute": "poll",
      "ToId": "4",
      "FromId": "9",
      "Id": "10"
    }
  ],
  "Blocks": [
    {
      "Position": {
        "Y": 646,
        "X": 370
      },
      "Rule": {
        "Map": {
          "foo": "$max(.Histogram[].Count)"
        },
        "Additive": true
      },
      "Type": "map",
      "Id": "1"
    },
    {
      "Position": {
        "Y": 247,
        "X": 355
      },
      "Rule": {
        "Rate": 1
      },
      "Type": "poisson",
      "Id": "2"
    },
    {
      "Position": {
        "Y": 378,
        "X": 411
      },
      "Rule": {
        "Map": {
          "sample": "$str(.sample)"
        },
        "Additive": true
      },
      "Type": "map",
      "Id": "3"
    },
    {
      "Position": {
        "Y": 511,
        "X": 436
      },
      "Rule": {
        "Window": "10s",
        "Path": ".sample"
      },
      "Type": "histogram",
      "Id": "4"
    },
    {
      "Position": {
        "Y": 153,
        "X": 309
      },
      "Rule": {
        "Interval": "1s"
      },
      "Type": "ticker",
      "Id": "5"
    },
    {
      "Position": {
        "Y": 404,
        "X": 575
      },
      "Rule": {
        "Interval": "1s"
      },
      "Type": "ticker",
      "Id": "9"
    }
  ]
}

$now

time w/ format

keys with underscores need escaping

Must use

.['random_float']

instead of

.random_float

which seems like a bummer.

crash using $exists

panic: interface conversion: interface is nil, not map[string]interface {}

goroutine 255055 [running]:
runtime.panic(0x94f020, 0xc210a1e7c0)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/nytlabs/gojee.func·043(0x0, 0x0, 0x865de0, 0xc210c48e50, 0xc210052248, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:788 +0xb2
github.com/nytlabs/gojee.Eval(0xc2109b6900, 0x863380, 0xc210438900, 0x864700, 0x0, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:1107 +0x2ad2
github.com/nytlabs/gojee.Eval(0xc2109b6880, 0x863380, 0xc210438900, 0x864700, 0x1, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:983 +0x83f
github.com/nytlabs/gojee.Eval(0xc2109b6600, 0x863380, 0xc210438900, 0xc2103cd701, 0x1, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:1112 +0x2d24
github.com/nytlabs/streamtools/st/library.(*Filter).Run(0xc2105f25b0)
/home/ubuntu/go/src/github.com/nytlabs/streamtools/st/library/filter.go:46 +0x151
created by github.com/nytlabs/streamtools/st/blocks.BlockRoutine
/home/ubuntu/go/src/github.com/nytlabs/streamtools/st/blocks/blocks.go:244 +0xfb

getkeyvalues error

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x40708e]

goroutine 3186217 [running]:
[fp=0x7f2d386dc380] runtime.throw(0x941e97)
/usr/lib/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0x7f2d386dc398] runtime.sigpanic()
/usr/lib/go/src/pkg/runtime/os_linux.c:239 +0xe7
[fp=0x7f2d386dc400] runtime.mapaccess1_faststr()
/usr/lib/go/src/pkg/runtime/hashmap.c:458 +0x2de
[fp=0x7f2d386dc6b8] github.com/nytlabs/gojee.getKeyValues(0xc201214fc0, 0x655840, 0xc2008fac00, 0xc2002a58d1, 0x7, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:758 +0x1a4
[fp=0x7f2d386dcf80] github.com/nytlabs/gojee.Eval(0xc201214fc0, 0x655840, 0xc2008fac00, 0xc2002574b8, 0x2, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:964 +0x322a
----- stack segment boundary -----
[fp=0x7f2d39f55f80] github.com/nytlabs/gojee.Eval(0xc200e32040, 0x655840, 0xc2008fac00, 0x65e300, 0x7f2d3865d600, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:873 +0x758
----- stack segment boundary -----
[fp=0x7f2d386f2de0] github.com/nytlabs/gojee.Eval(0xc201214f80, 0x655840, 0xc2008fac00, 0xc2008fac00, 0xc2007f1578, ...)
/home/ubuntu/go/src/github.com/nytlabs/gojee/jee.go:1000 +0x2f1a
[fp=0x7f2d386f2fb0] github.com/nytlabs/streamtools/blocks.Filter(0xc2003e1a80)
/home/ubuntu/go/src/github.com/nytlabs/streamtools/blocks/filter.go:24 +0xdc
[fp=0x7f2d386f2fb8] runtime.goexit()
/usr/lib/go/src/pkg/runtime/proc.c:1223
created by github.com/nytlabs/streamtools/daemon.(*Daemon).CreateBlock
/home/ubuntu/go/src/github.com/nytlabs/streamtools/daemon/daemon.go:438 +0x310