Slow performance on JVM?

[Omico]

Slow performance on JVM?

[bioball]

By the way there's also an in-language benchmark suite:

// abcd.pkl

a {
  b {
    c {
      d = "Hello World!"
    }
  }
}

// benchmark.pkl
amends "pkl:Benchmark"

import "abcd.pkl"

outputBenchmarks {
  ["abcd"] {
    sourceModule = abcd 
  }
}

Then you can pkl eval benchmark.pkl (or do the equivalent in Java) and get benchmark results!

[holzensp]

This example case is somewhat fundamentally going to underperform. kotlinx-serialization-properties just parses this static data. Pkl brings a lot more plumbing out of the box (and will take ~0.5-1s just to load the stdlib). On the tiny static cases like this, Pkl could/should do better, but will never win.

That said, I'd love it if your pkl-benchmark grows out to significantly cover relevant cases!

[bioball]

Your setup here runs Pkl in the JVM, which means you're also incurring the time it takes for the JVM to bootstrap (e.g. it needs to load classes, fire up our parser, etc)

Performance-wise, Pkl is generally faster if you use the native executables. Here's the same test on my machine using the CLI:

time pkl eval .dan-scripts/test.pkl
a {
  b {
    c {
      d = "Hello World!"
    }
  }
}

________________________________________________________
Executed in  206.59 millis    fish           external
   usr time   73.38 millis   36.00 micros   73.35 millis
   sys time  127.20 millis  907.00 micros  126.29 millis

But, like Phil said, Pkl will never beat a static format, because it is a program that needs to be interpreted, not just parsed.

[Omico]

Just updated my benchmark, seems not that horrible than before.

main: me.omico.pklbenchmark.JavaPropertiesBenchmark.test

Warm-up 1: 44600.802 ops/s
Warm-up 2: 48322.975 ops/s
Warm-up 3: 47936.852 ops/s
Warm-up 4: 48888.251 ops/s
Warm-up 5: 48884.637 ops/s
Iteration 1: 48651.885 ops/s
Iteration 2: 48480.269 ops/s
Iteration 3: 48167.602 ops/s
Iteration 4: 48239.609 ops/s
Iteration 5: 48942.755 ops/s

48496.424 ±(99.9%) 1214.203 ops/s [Average]
  (min, avg, max) = (48167.602, 48496.424, 48942.755), stdev = 315.325
  CI (99.9%): [47282.221, 49710.627] (assumes normal distribution)

main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testOnDemandMap

Warm-up 1: 38855.693 ops/s
Warm-up 2: 43636.701 ops/s
Warm-up 3: 43427.721 ops/s
Warm-up 4: 42766.916 ops/s
Warm-up 5: 42838.192 ops/s
Iteration 1: 43648.165 ops/s
Iteration 2: 43467.138 ops/s
Iteration 3: 43313.115 ops/s
Iteration 4: 42778.440 ops/s
Iteration 5: 42654.035 ops/s

43172.179 ±(99.9%) 1675.063 ops/s [Average]
  (min, avg, max) = (42654.035, 43172.179, 43648.165), stdev = 435.009
  CI (99.9%): [41497.116, 44847.242] (assumes normal distribution)

main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testWithPreParsedMap

Warm-up 1: 3058433.088 ops/s
Warm-up 2: 3247625.592 ops/s
Warm-up 3: 3234467.557 ops/s
Warm-up 4: 3229296.957 ops/s
Warm-up 5: 3247300.579 ops/s
Iteration 1: 3238172.409 ops/s
Iteration 2: 3256747.411 ops/s
Iteration 3: 3215626.194 ops/s
Iteration 4: 3247396.923 ops/s
Iteration 5: 3238568.379 ops/s

3239302.263 ±(99.9%) 58784.722 ops/s [Average]
  (min, avg, max) = (3215626.194, 3239302.263, 3256747.411), stdev = 15266.206
  CI (99.9%): [3180517.542, 3298086.985] (assumes normal distribution)

main: me.omico.pklbenchmark.PklBenchmark.testCreateEvaluatorOnDemand

Warm-up 1: 4612.738 ops/s
Warm-up 2: 8337.182 ops/s
Warm-up 3: 8322.103 ops/s
Warm-up 4: 8431.379 ops/s
Warm-up 5: 8398.744 ops/s
Iteration 1: 8394.138 ops/s
Iteration 2: 8364.193 ops/s
Iteration 3: 8402.756 ops/s
Iteration 4: 8382.008 ops/s
Iteration 5: 8339.333 ops/s

8376.486 ±(99.9%) 97.484 ops/s [Average]
  (min, avg, max) = (8339.333, 8376.486, 8402.756), stdev = 25.316
  CI (99.9%): [8279.002, 8473.970] (assumes normal distribution)

main: me.omico.pklbenchmark.PklBenchmark.testUsePreconfiguredEvaluator

Warm-up 1: 7454.698 ops/s
Warm-up 2: 11526.691 ops/s
Warm-up 3: 11606.931 ops/s
Warm-up 4: 11579.829 ops/s
Warm-up 5: 11595.770 ops/s
Iteration 1: 11544.253 ops/s
Iteration 2: 11502.271 ops/s
Iteration 3: 11538.845 ops/s
Iteration 4: 11554.954 ops/s
Iteration 5: 11506.545 ops/s

11529.374 ±(99.9%) 90.740 ops/s [Average]
  (min, avg, max) = (11502.271, 11529.374, 11554.954), stdev = 23.565
  CI (99.9%): [11438.634, 11620.114] (assumes normal distribution)