json-e/json-e

A data-structure parameterization system written for embedding context in JSON objects

JSON-e

JSON-e is a data-structure parameterization system for embedding context in JSON objects.

The central idea is to treat a data structure as a "template" and transform it, using another data structure as context, to produce an output data structure.

There are countless libraries to do this with strings, such as mustache. What makes JSON-e unique is that it operates on data structures, not on their textual representation. This allows input to be written in a number of formats (JSON, YAML, etc.) or even generated dynamically. It also means that the output cannot be "invalid", even when including large chunks of contextual data.

JSON-e is also designed to be safe for use on untrusted data. It never uses eval or any other function that might result in arbitrary code execution. It also disallows unbounded iteration, so any JSON-e rendering operation will finish in finite time.

Language Definition and Implementations

This repository both defines the JSON-e language and contains several implementations of that language.

The language definition is embodied in ./specification.yml and the documentation for the language constructs.

The language and implementations are all versioned together, following semantic versioning. Breaking changes to the language specification are not common, and typically result from bugfixes rather than new features. Generally, you may consider the language stable.

Any discrepancy in behavior between the implementations that is not documented as undefined behavior is considered a bug. If fixing that bug can cause backward incompatibility in an implementation, then the change may be considered breaking and be released with a new major version.

Other Implementations

• json-everything provides a implementation in .NET.

Full Documentation

See json-e.js.org.

Changes

See CHANGELOG.rst for the changes in each version of this library.

In Action

Where is JSON-e used?

• AlterSchema specifies upgrades as a set of declarative transformation rules written in JSON Schema itself and JSON-e.
• Taskcluster uses JSON-e extensively to generate task definitions from various JSON-shaped inputs such as GitHub events.

(feel free to submit a PR to add your project here)

json-e's Issues

unary +/- should not work on null or booleans

-null is currently not an error -- it should be. Similarly -true and +true

fix, document truthiness, falsiness

❌ indicates not currently working, and marked as a todo in the specification. I've chosen to follow python semantics here since that's what our expression language is mimicking.

• empty string -> false
• nonempty string -> true
• empty array -> false ❌
• nonempty array -> true
• empty object -> false ❌
• nonempty object -> true
• null -> false

Include a rationale for this language over others in the README

Automatically ensure bundle is updated

We don't want to do automated check-ins of an updated bundle.js, but we can add an automated test to ensure that re-generating bundle.js doesn't generate any diffs. Running that check on PRs will catch any changes that forget to update it.

Use YAML in the demo site

Per #43 - let's use YAML inputs in the demo site instead of eval'ing strings.

Test for one non-array thing being flattened

template: {$flatten: [[1, 2], [3], 4]}

I think the use of values.some in the JS implementation will fail to generate an appropriate error for this.

Parameterize .taskcluster.yml in tc-github

https://bugzilla.mozilla.org/show_bug.cgi?id=1335915

support injecting arbitrary webhook payload

https://bugzilla.mozilla.org/show_bug.cgi?id=1324807

Allow linking to the demo site with specific template, context

It should be possible to create a link to a demo page so that we can demonstrate a particular concept (or bug). Something like https://json-e.github.io/try-it.html?context={..}&template={..}

Compare things deeply

That JS does weird undefined things when you use == on objects or arrays is a JS bug, and we shouldn't propagate that into JSON-e. Instead, we should do deep object comparisons.

Change repository settings for website

Repository settings should be changed to display the website in the docs/ directory in the master branch. Consequently, the gh-pages branch should also be deleted.

Slicing should work for both arrays and strings

([1,2,3])[1:-1] is cool, why can we do it for strings too: "hello world"[5:-1] seems we'll want to have that right?

Add examples of expression language

As described in https://github.com/taskcluster/json-e/pull/105#pullrequestreview-45264466

It would be good to include at least a few examples of the expression language. Maybe that could be left to a later PR though.

Making json-e asynchronous

@jonasfj
Inspired by the fact we may define some functions in context which may do some async work like file I/O, DB access etc.

Demo website for json-e

We should make a github pages site that looks this:

With a live "try it" editor... that way we can easily test bugs etc...

Use codemirror to edit examples in demo site

Per #43. Codemirror is the editor we use in https://tools.taskcluster.net/task-creator/ for example

Only support unicode strings

We should reject bytestrings outright and always expect unicode input.

That's a little tricky with py2/py3 compatibility, but it's not so hard.

question: rename $dumps -> $json

I just realized that dumps might only be obvious to python users. And even then they only know it as json.dumps.

Should we consider renaming the construct {$json: ...} or {$stringify: ...}, any other ideas?

Allow support for stuff like ${myfunction({'key': 23})}

We probably should support ${myfunction({'key': 23})}, but this won't because the regex will match: ${myfunction({'key': 23}

Fix whatever is going wrong with demo playground links

As can be seen in https://github.com/taskcluster/json-e/issues/65#issuecomment-309842448

this should render

template: foo: {$eval: 'bar'}
context: {bar: null}
result: {"foo": null}

and this should render

template: 'tc_${name}': '${value}'
context: {name: foo, value: null}
result: {"tc_foo": "null"}

use JSON-e to parameterize action tasks

https://bugzilla.mozilla.org/show_bug.cgi?id=1335920

Define behavior for accessing undefined keys in an object

Example: {$eval: '({"test": 4}).foo'}

Questions:

• Does this eval to null
• Does it throw an error
• How do you get an object with default like x.foo || 'default-value' in js

As @djmitche said in comment:

So I can see a few options:

• x['foo'] and x.foo are null - but this will not "go away" when users want it to
• x['foo'] and x.foo are a delete marker - this will require handling delete markers within the expression language, where currently they're only handled in array/object construction
• x['foo'] and x.foo are errors, but get(x, 'foo', 123) is 123, and the get built-in always requires that third argument.
  Preferences?

I don't know how feasible it is... But I'm tempted to suggest an or operator... as in x.foo or 'default-value'. But having such an operator only allowed with a property access on the right hand side seems a bit crazy (and very context sensitive)...

I suggest that we tweak it such that x.foo == null and x.foo || 'default-value' both works. This is supposed to be a simple expression language, being super explicit as is preferred in python might not be the best choice for an embedded expression language.

Operator for exponentiation

I suggest that we use ** as exponentiation operator @jonasfj .

move mozilla-taskcluster info in-tree

https://bugzilla.mozilla.org/show_bug.cgi?id=1328727

mozilla-taskcluster use json-e for .taskcluster.yml

https://bugzilla.mozilla.org/show_bug.cgi?id=1335955

Add $merge

This is part of https://github.com/taskcluster/json-e/issues/10.

I agree with @jonasfj 's statement:

I'm thinking {$merge: [{a: 1, b: 1}, {b: 2}]} = {a: 1, b: 2}, ie. right side overwrites.
$mergeDeep is the same except it does it recursively to all objects.

Escape sequences for strings in expressions

Let's say I have an expression: {$eval: " 'ab' "} with more spacing.

What kind of escape sequences should we support.

Given that json-e is usually written as JSON or YAML.
We might not need a lot of escape sequences, example: {"$eval": " 'a\nb' "}, should already work, because when you parse it as JSON/YAML \n gets interpreted...

So in order to make a string that contains ", I would have to write (as text):
{"$eval": " \"a\\\"b\" "}

Which also currently works, because \" is intepreted at json-e level... But all other escape sequences don't work at json-e level... Should we add those, or not?

Perhaps we don't want more escape sequences because you don't need them... when they are handled at JSON/YAML level... Perhaps we don't want any escape sequences, even for \\\" since it's kind of easy to get that wrong.

Thought everybody?

Play with Pratt Parsers

I was looking for how to easily implement the same parser in multiple languages... Peg.js isn't a good fit for portability. Nor is it super simple.

For expressions we should probably look at:
http://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-expression-parsing-made-easy/

I couldn't stop myself from playing with a lever today... This might be the way to write a simple parser and ensure consistency across multiple implementations....

Simplify input parameters for decision task

https://bugzilla.mozilla.org/show_bug.cgi?id=1324767

document that numbers, booleans, and nulls can be interpolated

These values interpolate as their JSON literal values (numbers are numbers, true and false in lowercase, and null).

Note that null does not currently interpolate, so that will need to be fixed too.

Fail on undefined {$operator: ..}

We should reserve all object keys named with a single $ prefix, forcing users to always write $$. Then we can introduce new operators without concern for backward compatibility.

Make `fromNow` a built-in function, instead of an operator

Replace {$fromNow: '1 second'} with {$eval: 'fromNow("1 second")'}.

Split demo site into three pages

As described in https://github.com/taskcluster/json-e/pull/105#pullrequestreview-45264466

Maybe split this into three pages (entrypoints): home, docs, and playground. The home page would have the header that's baked into app.js at the moment; the docs would be a faithful rendition of the full README (without special-casing certain headings), and the playground would be a cleaner view. It's a bit confusing right now that the first of 100 template/context/result rows is a playground, while the rest start with interesting content.

Add `$let`

It would be really convenient to write

{$let: {
    ts: "fromNow('100 days')",
    foo: "bar + 10",
  },
 in: [{$eval: "parseDate(ts).year"}, {$eval: "parseDate(ts).month"}, {$eval: "parseDate(ts).day"}]}

This doesn't introduce any potential for looping or recursion, but it does make the context handling a little more complicated since within the in there are additional "variables" available that are not in the global context.

Note that the let variables are not ordered, so the cannot depend on one another (that is, foo: "ts + 10" would not work).

truthyness of functions

functions are first class, we need to define truthyness, and equality... add tests for this and provide that it works...

Stop checking in the generated site

Now that we're using webpack, each generation of the site will be slightly different (I think webpack uses random IDs).

https://gist.github.com/domenic/ec8b0fc8ab45f39403dd might help to get Travis to automatically build and export the result to the gh-pages branch.

in-operator should support deep-equality on array

{a: 45} in ["string", 46, {a: 45}] should be legal, and true..

Note: "a" in {"a": 45} is also true, I'm not sure we should allow: 5 in {"5": 45}... I suspect we don't want implicit type conversion... so I suggest we keep those semantics as is...

Allow escaping operators

How do I write a literal {$eval: '2+2'}?

Handle top-level deletion markers as an error

Currently {$if: 0, then: 1} will render to undefined, which isn't JSON-able. It should be an error.

Implement the in-operator from python

We should implement the in operator from python... Like 3 in [1,3,4,6].
It's a very common thing to test for, and it's one less function we need in our standard library.

Handling non-existing properties

Based on https://github.com/taskcluster/json-e/pull/44#discussion_r117988198

We need to agree on how to handle properties that don't exist and a possible keyword that silences errors.

syntax for array access and slicing

@jonasfj -> I am thinking of this kind of syntax

var context = {key: [1,2,3,4,5]};
var template = {
a: '${ $key(0) }',
b: '${ $key(1) }',
c: {$eval: '$key(0,2)'}
};

result = {a: '1' b: '2',c: [1, 2]};

Sort numbers lexically

Numbers should sort in numeric order -- they are currently sorted lexically.

Strings should sort in lexical order -- this works now.

Sorting between the two, without a by function, can probably remain undefined (and we don't need to check for that case).

Need `exists` syntax in expression language

We may want to conditionalize something on the presence of an object property, or on the presence of a property in the context.

Depending on how we handle missing object properties, the latter might be easy. But for context properties, we will need a function. Perhaps exists(abc) and exists(abc.def), where both would fail if there is no abc in the context, and the latter would fail if abc is not an object or does not have a def property.

TODO: raise error when performing 'number + string'

we should handle cases and raise error where user might try to something like "hello " + 5

Need object literal support in expression language

So we can do something like {$eval: "{a: x+10, b: x+20}"}.

@jonasfj I'm not totally convinced we need this. The above example could be written {a: {$eval: "x+10"}, b: {$eval: "x+20"}}. Is this never necessary?

Support escaping from string interpolation

What if you wanted a literal "${" in a string?

try-this: "a literal ${ in a string"

Define standard library (a base context with functions)

@hammad13060,

I'm thinking that json-e should have some standard library... Well, a set of functions that a built-in.
What do you think?

My thinking is things like:

• sqrt(x) for square root
• min/max(a,b)
• str(x) for converting to string
• uppercase(x)
• lowercase(x)
• abs(c) absolute value
• hex(x) to hexa decimal encoding
• floor/ceil(x)
• len(x) length of x
• sort(x)
• pow(a, b)
• parse(json)
• stringify(json)
• round(x)
• sum(x) compute sum of values in x
• codepoint(x) get integer value of character x, where x is a string of length 1

I'm not sure if we system put them under a namespace like builtin.pow or math.pow or if we should name the otherwise, or which functions we might be missing...

But we probably need things like getting the length of a list, converting integer to string. Those we can't live without. I suspect a few others like uppercase/lowercase are useful too.

TODO: propose syntax for $merge, $mergeDeep, $flatten, $flattenDeep, $concat/$join

I'm thinking {$merge: [{a: 1, b: 1}, {b: 2}]} = {a: 1, b: 2}, ie. right side overwrites.
$mergeDeep is the same except it does it recursively to all objects.

{$flatten: [[1,2], [3,4], [5]]} = [1,2,3,4,5]

Maybe we don't need $concat/$join once we have $flatten, or the other way around... Something to think about.

Get a `json-e.js.org` CNAME

Then our docs URLs will not include "github" or "taskcluster".