schibsted/jslt

JSON query and transformation language

JSLT

JSLT is a complete query and transformation language for JSON. The language design is inspired by jq, XPath, and XQuery.

JSLT can be used as:

• a query language to extract values from JSON (.foo.bar[0]),
• a filter/check language to test JSON objects (starts-with(.foo.bar[0], "http://")) ,
• a transformation language to convert between JSON formats.

An example transform:

{
    "time": round(parse-time(.published, "yyyy-MM-dd'T'HH:mm:ssX") * 1000),
    "device_manufacturer": .device.manufacturer,
    "device_model": .device.model,
    "language": .device.acceptLanguage,
    "os_name": .device.osType,
    "os_version": .device.osVersion,
    "platform": .device.platformType,
    "user_properties": {
        "is_logged_in" : boolean(.actor."spt:userId")
    }
}

Demo playground.

Language tutorial.

Function documentation.

More examples.

Javadoc.

API introduction.

Quick reference

Operation	Explanation
.	The context node
.<name>	Get value of key "<name>" inside an object
.[<index>]	Get value <index> inside an array
.[<from> : <to>]	Array slicing
if (<expr>) <expr> else <expr>	If test to decide which value to return
let <name> = <expr>	Define a variable
$<name>	Refer to a variable
[for (<expr>) <expr>]	Transform an array
{for (<expr>) <expr> : <expr>}	Transform an object
def <name>(<name>, <name>...) <expr>	Declare a function
// <anything up to end of line>	Comment
{ <key> : <expr> }	Object constructor
{ <key> : <expr>, * : . }	Specify one key, copy rest of input
5 * 7 + 23.2	Arithmetic operations
7 < 5	Comparators
7 < 5 and .foo == "yes"	Boolean operators

Using the library

To include JSLT in your project, depend on:

<dependency>
  <groupId>com.schibsted.spt.data</groupId>
  <artifactId>jslt</artifactId>
  <version>0.1.14</version>
</dependency>

At runtime JSLT depends on Jackson, and nothing else.

To transform one JsonNode into another, do:

import com.schibsted.spt.data.jslt.Parser;
import com.schibsted.spt.data.jslt.Expression;

JsonNode input = ...;
Expression jslt = Parser.compileString(transform);
JsonNode output = jslt.apply(input);

For more alternatives, see the javadoc.

Command-line

To run transforms on the command-line, first build with ./gradlew clean shadowJar. Then you can run with:

java -cp build/libs/*.jar com.schibsted.spt.data.jslt.cli.JSLT transform.jslt input.json

The result is written to standard out.

Extension functions

You can implement your own functions and add them to the language. See the extension function tutorial.

Feedback

If you have questions about how to use JSLT, please ask the question on StackOverflow, with the tag jslt.

If you have problems, feature requests, or think you found a bug, please open an issue.

Status

The language design is not finished, so features may be added. The language as it stands is not likely to change.

The entire language is implemented, and all of the function library. Functions may be added.

The language has been used in production at Schibsted since January 2018, performing about 9 billion transforms per day, and many times more queries.

Building JSLT

To build JSLT as a jar file, run ./gradlew jar.

To build a fat jar with all dependencies included, run ./gradlew shadowJar.

To run the tests: ./gradlew check.

There is a pom.xml file, but Maven is not used for building JSLT, and the file is not intended to work. It's only there to make Github dependency tracking work.

More information

Developing a language for JSON processing: video of talk, slides only.

Running the playground yourself

Anthony Sparks is working on a VM-based implementation in Java

A paper describing (among other things) some of the ways Schibsted uses JSLT.

Visual Studio syntax highlighter for JSLT.

IntelliJ JSLT plugin.

Apache Camel JSLT component.

JSLT is also integrated in Apache NiFi as a processor.

How Willhaben.at uses JSLT with Kafka Connect

IBM Cloud Pak for Business Automation.

Pincette event sourcing framework uses JSLT.

LICENSE

Copyright (c) 2018 Schibsted Marketplaces Products & Technology AS

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

What is missing

Things to be done:

• Move the tests out into YAML files.
• Write a proper spec with EBNF and everything.
• Fix the syntax ambiguity problem with let and def.
• Implement toString() throughout the object tree, so that it's possible to turn expressions back to strings.
• Optimizer:
  • Optimize ... and boolean( ... ) by removing boolean().
  • Implement parse tree traversal API.
  • Make sure entire tree is traversed (inside function decls and variables, for example).
  • Complete constant folding. Particularly constant folding for variables would be valuable.
  • Inlining of functions.
  • Eliminate unused variables.
• Use property-based testing and fuzz testing to harden the parser.

See also the list of ideas.

jslt's Issues

Can't parse perfectly valid float number.

Found another float which JSLT can't parse: 1000.

Use JMH to run benchmarks

This is useful to help us determine whether or not optimizations have the desired effect.

Is it possible to invoke a specific function with a variable as name.

I have an input is:

{
  "A" : 60,
  "B" : 2
}

and this Jslt currently doesnt work 😢

def AFunc(value)
   if ($value >=60 )
    1
   else if($value >= 40)
    2
   else if($value >=20)
    3
   else if($value >=0)
    4
   else
    0

def BFunc(value)
   if ($value >=34 )
    3
   else if($value >=0)
    4
   else
    0

{for(.) .key: .key+"Func"(.value)}

and expected an output :

{
  "A" : 1,
  "B" : 4
}

Is there a possible way I can get the expected result without jslt like the following one, because I could have 500 key value pair in the input json.

def AFunc(value)
   if ($value >=60 )
    1
   else if($value >= 40)
    2
   else if($value >=20)
    3
   else if($value >=0)
    4
   else
    0

def BFunc(value)
   if ($value >=34 )
    3
   else if($value >=0)
    4
   else
    0

{
 "A":AFunc(.A),
 "B":AFunc(.B)
}

Allow trailing commas `,` before the closing bracket `}`

JSON does not allow trailing commas like in

{ 
  "@id": 1,
}

The comma after 1 will make the parser believe a new key is expected. This is usually an annoyance and JSLT could support trailing commas to make it easier to write JSLT templates, without worrying about the commas

Simplify package namespace to com.schibsted.jslt

The added levels don't help.

It should be relatively straight forward to make this backwards compatible?

Add option to preserve objects order

Even if the JSON standard defines list of objects as unordered (http://www.rfc-editor.org/rfc/rfc7159.txt) add option to preserve objects order to maintain output JSON readability

Gradle version is out of date

The gradle version in github currently is 4.0.1, which doesn't run with more recent versions of Java. It would be better to package gradle 4.7 at least.

I'm happy to submit a pull request for this if that would be helpful.

Apply two rules together

I have input json like this one :

{
  "A":12,
  "B":1,
  "C":111
}

Expected output:

{
  "A" : {
    "value" : 12, "min":0
  },
  "B" : {
    "value" : 1
  },
  "C" : {
    "value" : 111
  }
}

I two JSLT .

one is :

{for (.) .key:{"value":.value}} 

the other is:

let a = {"A":{"value":.A,"min":0}} 
$a

The two JSLT works seperately.
when I do "{for (.) .key:{"value":.value}} +$a"
I couldn't get the expected result.

Help Needed in Transformation.

I am trying to write a adapter between two json formats. The incoming json is of the type

{
  "data" : {
    "payload" : [
      {"x.com.da.power" : "off"},
      {"x.com.da.state" : "stop"},
      {"x.com.da.progess":  "Prewash"}
    ]
  }
}

The output of the transform should look like

{
  "events" : [
    {
      "name" : "switch",
      "value" : "off",
      "description" : "Your device was turned off"
    },
    {
      "name" : "state",
      "value" : "stop",
      "description" : "your device state was changed to stop"
    },
    {
      "name" : "progress",
      "value" : "prewash",
      "description" : "Your device progress state is prewash"
    }
  ]
}

The correlation between input and output is, All the keys inside the root payload maps to one or more objects in the output. i.e. "x.com.da.power" key maps to the object with name attribute "switch". In this case it is one-to-one, but in other cases it might be one-to-many too.

My logic in JSLT is to loop over the objects in the payload object and perform an if-else-if logic and create the resulting json output. Please find my JSLT code below.

def parse(node)
  if($node.data != null and $node.data.payload != null and size($node.data.payload) > 0)
    {"events" : [for($node.data.payload)

        let k = .key
        let v = .value

        if(test($k, "x.com.da.power"))
          {
            "name" : "switch",
            "value" : $v,
            "descriptionTest": "your device is turned on"
          }


        else if(test($k, "x.com.da.state"))
          {
            "name" : "machineState",
            "value" : lowercase($v),
            "descriptionTest": "Machine State changed to"
          }


        else if(test($k, "x.com.da.progress"))
          {
            "name" : "washerJobState",
            "value" : lowercase($v),
            "descriptionTest": "Washer Job State changed to"
          }

     ]}

parse(.)

Please help me with this.

Add strict validation mode to throw exception when dot expression resolves to null

Currently JSLT is quite forgiving where if the evaluation of a dot expression resolves to null, the transformation still passes.

In some cases this is desirable, but for my use case, stricter validation is required and an exception should be thrown if this is encountered. I would suggest two modes of operation, one which is the current approach and another which I would label the "strict validation/transformation" mode.

I did a very quick scan of the codebase and it seems like this is the point from where the expression should be thrown:
https://github.com/schibsted/jslt/blob/master/src/main/java/com/schibsted/spt/data/jslt/impl/DotExpression.java#L48

The mode flag should either be added during parsing of the JSLT and construction of the Expression or when the apply method is invoked, still haven't made up my mind which one would be best. I guess providing it on the apply method would be more flexible.

Another option would be to add it to the language grammar to make it more explicit.

Add smart merge of objects

Given two objects

{
 "nested": {
    "a": 1,
    "b": 2
  }
}

{
 "nested": {
    "b": "overwrite",
    "c": 3,
  }
}

The merge($a, $b) should produce

{
 "nested": {
    "a": 1,
    "b": "overwrite",
    "c": 3,
  }
}

Note that this is different from the current $a + $b in that + will blindly overwrite all keys producing

{
 "nested": {
    "b": "overwrite",
    "c": 3,
  }
}

where the whole nested key is replaced and not merged

Configure jslt to include keys for null or {} values instead of omitting the key

Currently keys with null or {} or [] value are omitted. Is there a way to configure this behavior or where in code can I patch this so I force it to include empty/null keys as well.

How to merge object to one result ?

I have input Json:

{
  "data": [
    {
      "A": 12,
      "B": 23,
      "C": 15
    },
    {
      "A": 22,
      "B": 13,
      "C": 16
    }
  ]
}

Expected output 1: get the max value of each key.

{
  "A": 22,
  "B": 23,
  "C": 16
}
   

Expected output 2: merge the values of same key to an array?

{
  "A": [12,22],
  "B": [23,13],
  "C": [15,16]
}

Is Jslt able to do those two kinds of transform ?

Unable to access value when key contains special characters

While trying to access a value for a field containing special character using quotes, it returns null
{ "first_name" : "ABC", "last_name" : "XYZ", "city" : "PQR" }

.city returns "PQR" but ."first_name" and ."last_name" returns null.

How to retrieve values for ."first_name" and ."last_name" ?

Grouping by a repeated ID

@larsga I would like to perform the following transform. I apologize if this has already been covered in documentation/another issue. I had a look, but could not find anything.

{
  "details": [
    {
      "childID": "1",
      "mainID": "abc",
    },
    {
      "childID": "2",
      "mainID": "abc",
    },
    {
      "childID": "3",
      "mainID": "def",
    }            
  ]
}

{
  "groupedDetails": [
    {
      "mainID": "abc",
      "childIDs": [
        "1",
        "2"
      ]
    },
    {
      "mainID": "def",
      "childIDs": [
        "3"
      ]
    }
  ]
}

I have a good idea how to do this once I have an array of unique mainIDs, but I'm missing that part.

nested * matcher does not work

I have a large flat JSON object that I wish to convert to nested JSON. The * matcher does not seem to work inside of objects.

Example JSLT:
{
"parent" : .one,
"nested" : {
* : .
}
}

Example input JSON:
{
"one" : "1",
"two" : "2"
}

Actual Result:
{
"parent" : "1"
}

Expected Result:
{
"parent" : "1",
"nested" : {
"two" : 2
}
}

Am I missing something here?

Thanks.

Global variables can't be referenced from functions

The following should return 15, but gives an error because $other is not defined:

let other = 5

def test(obj)
  $obj + $other

test(10)

Removing duplicates while transforming flat nodes to hierarchy.

@larsga I have another unique problem in dealing with nested structures and need your opinion. I have duplicate reference node, I need to figure out how to private the duplicate from being created. I started building a global array and thinking that I need to check that array in flattening to remove duplicates. If you can give it a look and let me know what you think the best option here is then it will be great. Below is current input, expected output, and the current template.

Input

[
{
"chapters": [
{
"name": "a-f"
},
{
"name": "b"
}
]
},
{
"chapters": [
{
"name": "a-f"
},
{
"name": "e"
}
]
},
{
"chapters": [
{
"name": "l-p"
},
{
"name": "m"
}
]
},
{
"chapters": [
{
"name": "v-z"
},
{
"name": "v"
}
]
},
{
"chapters": [
{
"name": "v-z"
},
{
"name": "z"
}
]
}
]

Expected Output

[
{
"name": "a-f"
},
{
"name": "b",
"parents": [
{
"name": "a-f"
}
]
},
{
"name": "e",
"parents": [
{
"name": "a-f"
}
]
},
{
"name": "l-p"
},
{
"name": "m",
"parents": [
{
"name": "l-p"
}
]
},
{
"name": "v-z"
},
{
"name": "v",
"parents": [
{
"name": "v-z"
}
]
},
{
"name": "z",
"parents": [
{
"name": "v-z"
}
]
}
]

Current Template

def flatten(arrayofarrays)
if ($arrayofarrays)
$arrayofarrays[0] + flatten($arrayofarrays[1 : ])
else
[]

let globalTopNodeCache = [for (.) if(.chapters and size(.chapters) > 0) .chapters[0].name]

flatten([for (.)
let chapterZero = if(.chapters and size(.chapters) > 0) .chapters[0] else ""
[{"name" : $chapterZero .name}] + [for (.chapters[1 : ]) {"parents":[{"name":$chapterZero .name}],"name" : .name}]
])

Question about deal with two kinds input Json at once.

I have 2 kinds of json input:

sometimes it is

{
  "A" :{
   "B":"33",
   "C":[]
  }
}

sometimes changes to this

{
  "A" :{
   "B":[],
   "C":"33"
  }
}

Currently, I have two JSLT, to deal with two kinds of input.
one is

{
  "A" : {
    "B":.A.B+"x"
  }
}

the other is

{
  "A" : {
    "C":.A.C+"x"
  }
}

My Question is: Is it possible do some check in the JSLT to deal with two kinds of input at one JSLT ?

JSLT does not produce itself: empty objects

Hi there,

Just getting started with JSLT so may have misunderstood something, but I would have expected this JSLT program to be self-producing:
{"foo": {}}

Instead the playground outputs an empty object:
{ }

The behavior seems to be that if a dictionary value is the empty object anywhere in the JSON document, then the key is omitted from the output. Is this expected?

Add {from,to}-jwt to mirror {from,to}-json

In some cases it's useful to encode and decode jwt tokens, or encrypted jwt tokens (jwe).

JWT is just authenticated (+ optionally encrypted) json, so having support for handling "secure" json and not only stringified would be useful.

Provide a means to change the context node

Input

{
  "a": {
    "b": 1,
    "c": 2, 
    "d": 3
  } 
}

I want to simplify [ .a.b, .a.c, a.d ] to .a | [.b, .c, .d]. The idea is that the context node . inside [.b, .c, .d] no longer points to the input node but to .a instead.

Loss of precision with parse-time/format-time

This program does not yield the intended result:
format-time(parse-time("2017-01-10", "yyyy-MM-dd"), "yyyy-MM-dd")

Instead of returning "2017-01-10", it returns "2017-01-09".

I haven't debugged this but I assume it is due to a numerical instability issue within parse-time.

Add a `has-key()` function

The has-key(obj, 'x') should be equivalent to

is-object($o) and contains("key", $o)

On a first glance it may seems that just .x checks for the existence of key x but

• what if . is an array
• what if . contains key x but the value is null or a falsy value

Can't parse perfectly valid float number

JSLT won't parse 1.484006399998E9. Need to investigate.

Add a filter() function that allow to selected keys and nested keys from an object

Given the input :

{
  "a": 1,
  "b": 2, 
  "c": {
     "d": 3,
     "e": 4
   }
}

I want the filter(myobj, ['.b', '.c.e']) to produce the output

{
  "b": 2, 
  "c": {
     "e": 4
   }
}

Note that is this different from [for (myobj) . if ('.b', '.c.e')] that would produce an array [2, 4]

How to use "*:." in a function?

I have input json like this one :

{
  "A": "CH802053",
  "B": [],
  "C": [
    {
      "D": "F10",
      "E": [
        {
          "id": 1,
          "name": "xx"
        },
        {
          "id": 2,
          "name": "xxx"
        }
      ],
     "F": "asef",
     "G": "xwrew",
     "H": "123"
    }
  ]
}

I have JSLT .

def func1(CObject) {
	"E": [
		for ($CObject.E) size(.)
	]
	// how to keep rest object value here : "D","F","G","H"
	//*:.
}

{
	"C": [ for (.C) func1(.)],
	*: .
}

The result is

{
  "C" : [ {
    "E" : [ 2, 2 ]
  } ],
  "A" : "CH802053",
  "B" : [ ]
}

My question is how can I keep the value of "D","F","G","H" in the "func1", The expression "* : . " give me null pointer here.

Summer holidays

Just a quick notice to let everyone know that until August 13 I will mostly be on summer holidays. Replies will be slow until then.

[request for help] Generating dynamic keys in a JSON object

Hi,

Another language question!

Let's consider the following JSON input:

{
  "filter" : {
    "field" : "aField",
    "terms" : [ "valueA", "valueB" ]
  }
}

I would like to transform into the following snippet :

{
  "filter" : [
    {
       "terms": {
           "aField": [ "valueA", "valueB" ]
       }
    }
  ]
}

So it implies to be able to dynamicaly generate keys in the target JSON based on the value of .filter.field.

So far I've been able to achieve this goal by using the from-json function and generate my JSON as a string like this:

from-json("{
  \"filter\": [
    {
      \"terms\": { 
        \"" + .filter.field + "\": " + .filter.terms + "
      }
    }
  ]
}")

That's not quite elegant though, and would become a pain for more complex JSON structures.

I'm looking for something like:

{
  "filter": [
    {
      "terms": { 
        string(.filter.field) : .filter.terms
      }
    }
  ]
}

however this does not compile. Am I missing something?

Idea: Java functions

Sometimes you may want to do a transform that is easy in Java but hard in jslt. My use case is parsing/converting timestamps of different formats, but you can use your imagination: Decoding base64 data, inner joins, deduplication, etc.

So, my proposal is that jslt should support defining custom functions in Java which can then be referenced from the jslt code.

Implementing this would mitigate many of the other wishlist items since they could be implemented as custom functions.

Is there an easy way to do trimming, padding and replace of strings in jslt?

Hello,

We are using JSLT to achieve dynamic mapping configurations.
But couldn't find a quick way to do string functions like trim, padding, replace etc.

Would you have any pointers for the same?

Substring and/or string replace?

Hello,

Looking at JSLT and trying to fit it to some simple transformations like "take the first two chars of this field" and "replace all spaces with underscores".

Example, would be something like:

{
   "output1": substring(.input1.id, 0, 2),
   "output2": replace(.input2, " ", "_")
}

Is this something anyone is doing right now, and if so are you using an existing built-in function (I see there's regex, but is there anything like replace and substring modeled above), or through your own defined functions?

Thanks in advance.

problem loading a jstl file - Parse error: Can't compare null and 2 at C:\\specs.jstl:7:1

@larsga as a first example I have following in my template. When I test it with http://spt-data-dev-public-web.s3-website-eu-west-1.amazonaws.com/jstl2.html everything works but when I call with Parser.compile with string reader and it fails with an exception "Can't compare null and 2 at C:\specs.jstl:7:1". The problem it seems is that parentSize is not getting evaluated?

JSTL
let parentSize = size(.parents)
if ($parentSize == 1)
{
"size":"Single"

}
else if ($parentSize >= 2)
{
"size": "many"
}
else
{
"size": "0"
}

Input
{
"name": "park",
"parents": [
"home",
"land",
"public"
]
}

Support for compiled JSLT template.

Dear Author,

Could you please look at the possibility of implementing a compiled templates?

Regards,
Pratik Parikh

Float problem

I have input

{
  "value" : 6.777176437128938e+306
}

JSLT

def x(value)
 if ($value >= 7.0)
    2
 else if($value < 7.0)
    1
 else 
    0

{
  "value" : x(.value)
}

And then I get a result as below, which is not correct.
I'm not sure where is wrong here, please help !

{
  "value" : 2
}

Possible resource leakage on BuiltingFunctions (Capture)

I see on the class BuiltingFunctions on the subclass Capture that a static cache exists for JstlPatterns, I suppose this is there to try to optimize capture.

Where I work we are investigating to possibility of using this nice library for our json transformations but on our scenarios the jslts that we use may be on very constant change as they could be injected to the application from an external source we don't control. But while looking at the code one of the concerns we have is that this cache cannot be cleaned and will be kept until a restart.

In my opinion this cache should be somehow bound to the ParseContext and not shared as static so it doesn't leak.

JSLT Transform issue.

Hi,
I am currently trying to create an empty json array and simply trying to add elements, It seems to be able to add only one element. When trying to add the second element it throws an error.

`def addToList(list, obj)
$list + [$obj]

def event(name, value, desc)
{
"name" : $name,
"value" : $value,
"description" : $desc
}

def parse(object)
let events = []
addToList($events, event("switch", "on", "TurnedOn"))
addToList($events, event("mode", "prewash", "Mode Changed to"))

parse(.)`

Are you using JSLT?

I'd like to list users of JSLT in the readme, but to do that I need to know who they are. If you're using JSLT in your software, please post a comment here to let us know.

Forbid duplicate keys in object literals

The following should be detected as invalid JSLT:

{
  "foo" : 24,
  "foo" : 25
}

Allow expressions to compute keys

Issue #55 demonstrates that there might be a need for being able to write objects like this:

{
  .foo : .bar,
  "something" : .else
}

There doesn't seem to be any clear reason why this couldn't be supported.

Adding JSLT extension functions

Hello,

We are looking to extend the jslt functions with custom java functions, as shown here: https://github.com/schibsted/jslt/blob/master/extensions.md, the one piece that is escaping me is what the def and .jslt file should look like as it calls this function for one of the fields. Could anyone be so kind as to add that to the example? Thanks in advance.

Boolean operators in if statement

Hi,

Is it possible to combine(boolean and, or) tests in an if statement? I would need it to test ranges against a numeric field.

I can't find this is the language tutorial or in the exemples.

Upgrade Demo playground: is-decimal() function is not there

When I run is-decimal(1.0) I get

ERROR: com.schibsted.spt.data.jslt.JsltException: No such function: 'is-decimal' at <inline>:1:1

I suppose that the Demo playground is running an old version of the jstl library missing the commit dac7b38

NullPointerException on matcher expression

This expression throws a NullPointerException:

{ "a": "b" }
+
{
  "type" : "Anonymized-View",
  * : .
} 

Input:

{ "test" : "t" }

I would expect the following result:

{"type":"Anonymized-View","test":"t","a":"b"}

Add a hash function that returns a number (integer) instead of a string

The existing sha256-hex() function provides an excellente hashing function that returns a string. There are some use cases where an integer result is more convenient in order to perform:

• modulo(hash-int(.property), 100) == 0 when you need a deterministic way to select a portion of input objects
• range checks on the results ,

I see that there the original proposal for a hash function was hash-code() which indeed returned an integer. that was later changed to the current sha256-hex.

My proposal it to recover the original hash-code() function for those cases where an integer result is more convenient.

Order of operations

Hello,

I'm not sure if this is as designed or a bug.

let x = 1
["h" + $x + 1]

I would expect the output to be h11 but instead the output is h2. Rational, is that h + $x be evaluated first, which creates a string h1 and h1 + 1 would be a new string: h11.

How to solve complex Nested Example

@larsga and others,

I have following am struggling to flatten it, need help to understand what I am doing wrong.

Input
[
{
"movies": [
{
"movie": "The Equalizer II",
"properties": {
"name": "action",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
},
{
"movie": "Mamma mia",
"properties": {
"name": "drama",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
}
],
"knownas": [
{
"movies": "release this week",
"properties": {
"name": "premierlist",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
}
],
"top": {
"properties": {
"name": "new movies",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
}
},
{
"movies": [
{
"movie": "Jurassic World: Fallen Kingdom",
"properties": {
"name": "SiFi",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
},
{
"movie": "incredibles 2",
"properties": {
"name": "family",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
}
],
"knownas": [null],
"top": {
"properties": {
"name": "monthly topper",
"creationDate": "2018-07-20T03:37:18.190909+00:00"
}
}
}
]

output:

[
{
"type": "monthly topper"
},
{
"type" : "new movies"
},
{
"type" : "premierlist"
},
{
"name": "The Equalizer II"
"type" : "action",
"parents" : [{
"type": "new movies"
},{
"type": "premierlist"
}]
},
{
"name": "Mamma mia"
"type" : "drama",
"parents" : [{
"type": "new movies"
},{
"type": "premierlist"
}]
},
{
"name": "Jurassic World: Fallen Kingdom"
"type" : "drama",
"parents" : {
"type": "monthly topper"
}
},
{
"name": "incredibles 2"
"type" : "drama",
"parents" : {
"type": "monthly topper"
}
}
]

currently, I have the following transformation

[for (.[0:]) {"type": .top.properties.name},
[for (.movies){
"name": .movie,
"type": .properties.name
}]
]

Can I import jslt from another jslt ?

Im thinking that create a jslt lib has many def methods, and import it from any other jslt?

1. myLib

def largestValue(arr)
  if (not($arr))
    0
  else
    let temp = largestValue($arr[1 : ])
    if ($temp.value <= $arr[0].value)
      $arr[0]
    else
      $temp

def smallestValue(arr)
  if (not($arr))
   { "value":99999999}
  else
    let temp = smallestValue($arr[1 : ])
    if ($temp.value > $arr[0].value)
      $arr[0]
    else
      $temp

2. Any jslt that uses it

import "http://xxx/methods" as myLib
...
myLib:smallestValue([...])

Is it possible to do that ?

[request for help] Adding properties to existing nodes

Hi,

Let's consider the 2 following JSON documents that schematize a use case I have:

{
    "children": {
       "child": {
           "pseudo_csv": "v1:v2",
           "a_tag": "foo"
       }
   }
}

and

{
    "children": {
       "child": [
         {
           "pseudo_csv": "v1:v2",
           "a_tag": "foo"
         },
         {
           "pseudo_csv": "w1:w2",
           "another_tag": "bar"
         }
      ]
   }
}

When I receive input documents, I can have either a single node or an array for .children.child. Additionnally I can't predict the set of properties in each child, apart from pseudo_csv which is guaranteed to be present ...

I would like to write a JSLT transform that would respectively produce the following outputs :

{
    "child": {
        "prop_1": "v1",
        "prop_2": "v2",
        "a_tag": "foo"
   }
}

and

{
    "child": [
         {
           "prop_1": "v1",
           "prop_2": "v2",
           "a_tag": "foo"
         },
         {
           "prop_1": "w1",
           "prop_2": "w2",
           "another_tag": "bar"
         }
      ]
   }
}

So I'd like to flatten the children.child hierarchy and simply keep one level, and for each child, break a given "pseudo-CSV" property into a set of properties, and add every other property to the document.

I'm actually struggling to write the JSLT. I really need it to be dynamic, I can't use object matching.

Got wrong translate result when pass ObjectNode instead of JsonNode to JSLT.

I have an Input:
{"A":1,"B":2,"C":{"c1":"asf","c2":12}}

Rule:

def applyX(obj)
{
  "c1" : [$obj.c1, $obj.c1],
  "c2" : [$obj.c2, $obj.c2]
}
{
   "A":"new",
   "data":applyX(.C)
}

It works well at http://spt-data-dev-public-web.s3-website-eu-west-1.amazonaws.com/jstl2.html and get correct result.

{
  "A" : "new",
  "data" : {
    "c1" : [ "asf", "asf" ],
    "c2" : [ 12, 12 ]
  }
}

But in my java code it produced wrong result. Because I didn't convert ObjectNode to JsonNode.

Input:
{"A":1,"B":2,"C":{"c1":"asf","c2":12}}
Output:
{"A":"new","data":{"c1":[null,null],"c2":[null,null]}}

In my java code, I have the same input and rule.

import static com.fasterxml.jackson.databind.SerializationFeature.WRITE_DATES_AS_TIMESTAMPS;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import com.schibsted.spt.data.jslt.Expression;
import com.schibsted.spt.data.jslt.Parser;

import java.util.HashMap;
import java.util.Map;

public class JsltTester {

  public static void main(String[] args) {
    final String rule =
        "def applyX(obj)\n"
            + "{\n"
            + "  \"c1\" : [$obj.c1, $obj.c1],\n"
            + "  \"c2\" : [$obj.c2, $obj.c2]\n"
            + "}\n"
            + "{\n"
            + "   \"A\":\"new\",\n"
            + "   \"data\":applyX(.C)\n"
            + "}";
    final JsonNode inputJson = createInput();
    final Expression jslt = Parser.compileString(rule);
    final JsonNode output = jslt.apply(inputJson);

    System.out.println("Input:");
    System.out.println(inputJson);
    System.out.println("Output:");
    System.out.println(output);
  }

  private static JsonNode createInput() {
    final ObjectMapper MAPPER =
        new ObjectMapper().disable(WRITE_DATES_AS_TIMESTAMPS).registerModule(new JavaTimeModule());
    final ObjectNode result = MAPPER.createObjectNode();
    result.put("A", 1);
    result.put("B", 2);
    result.put("C", 3);
    Map<String, Object> cMap = new HashMap<>();
    cMap.put("c1", "asf");
    cMap.put("c2", 12);
    result.putPOJO("C", MAPPER.valueToTree(cMap));
    // ---------- code produce the wrong translate result. ---------- 
    return result;
    //----------- uncomment this return statement, will produce correct result. but why have to convert to jsonNode ?---------- 
    //        try {
    //          return MAPPER.readTree(result.toString());
    //        } catch (IOException e) {
    //          throw new RuntimeException(
    //            "An error occurred when create input json." );
    //        }
  }
}