Efficient CBOR and JSON (de)serialization for Scala.
See the project website for all documentation.
Efficient CBOR and JSON (de)serialization in Scala
Home Page: https://sirthias.github.io/borer/
License: Mozilla Public License 2.0
Efficient CBOR and JSON (de)serialization for Scala.
See the project website for all documentation.
This is great stuff!
Is there an example of how this can be used directly with Akka?
[info] Updating ProjectRef(uri("file:/home/andriy/Projects/com/github/plokhotnyuk/borer/project/"), "borer-build")...
[warn] module not found: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local: tried
[warn] /home/andriy/.ivy2/local/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/io/bullet/borer-core_2.12/0.9.1-SNAPSHOT/borer-core_2.12-0.9.1-SNAPSHOT.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/andriy/.sbt/preloaded/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/andriy/.sbt/preloaded/io/bullet/borer-core_2.12/0.9.1-SNAPSHOT/borer-core_2.12-0.9.1-SNAPSHOT.pom
[warn] module not found: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local: tried
[warn] /home/andriy/.ivy2/local/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/io/bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/borer-derivation_2.12-0.9.1-SNAPSHOT.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/andriy/.sbt/preloaded/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/andriy/.sbt/preloaded/io/bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/borer-derivation_2.12-0.9.1-SNAPSHOT.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[warn] :: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] io.bullet:borer-core_2.12:0.9.1-SNAPSHOT (/home/andriy/Projects/com/github/plokhotnyuk/borer/project/plugins.sbt#L12-16)
[warn] +- default:borer-build:0.1.0-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[warn] io.bullet:borer-derivation_2.12:0.9.1-SNAPSHOT (/home/andriy/Projects/com/github/plokhotnyuk/borer/project/plugins.sbt#L12-16)
[warn] +- default:borer-build:0.1.0-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[error] sbt.librarymanagement.ResolveException: unresolved dependency: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[error] unresolved dependency: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
[error] at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:332)
[error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:208)
[error] at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:239)
[error] at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:204)
[error] at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[error] at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[error] at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
[error] at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
[error] at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
[error] at xsbt.boot.Using$.withResource(Using.scala:10)
[error] at xsbt.boot.Using$.apply(Using.scala:9)
[error] at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
[error] at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
[error] at xsbt.boot.Locks$.apply0(Locks.scala:31)
[error] at xsbt.boot.Locks$.apply(Locks.scala:28)
[error] at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:199)
[error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:196)
[error] at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:238)
[error] at sbt.internal.librarymanagement.IvyActions$.updateEither(IvyActions.scala:193)
[error] at sbt.librarymanagement.ivy.IvyDependencyResolution.update(IvyDependencyResolution.scala:20)
[error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:56)
[error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:45)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:93)
[error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:68)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$19(LibraryManagement.scala:106)
[error] at scala.util.control.Exception$Catch.apply(Exception.scala:224)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:106)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:89)
[error] at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:149)
[error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:120)
[error] at sbt.Classpaths$.$anonfun$updateTask$5(Defaults.scala:2561)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error] at sbt.std.Transform$$anon$4.work(System.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.Execute.work(Execute.scala:278)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: unresolved dependency: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[error] unresolved dependency: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
https://sirthias.github.io/borer/ redirects to http://decodified.com/borer/ and returns 404
Latest borer-compat-akka
exposes object akkaHttp
which is a great addition!
Can we also have a companion trait akkaHttp
with the same content, but allows users to extend/mix-in and add additional (un)marshalling support as needed?
For example, if we do not fix #62, I would like to add those two (un)marshallers in the extended HttpCodecs
in my app.
utest
, scodec-bits
, and akka-actors
are released for Scala 2.13.0 already
According to the flamegraph report of run with Async Profiler ~95% CPU is spending in call of io/bullet/borer/derivation/MapBasedCodecs$deriveDecoder$.verifyNoDuplicate$1
:
How to reproduce:
Reddit.User
last 10 fields were removed, and run the model benchmark for Borer that should parse reddit-scala.json
:git clone [email protected]:plokhotnyuk/borer.git
cd borer
git checkout unexpected-troughput-drop
sbt clean 'benchmarks/jmh:run -i 5 -wi 5 -f 3 -p fileName=reddit-scala.json BorerModelBenchmark.decode'
And you will get results like this:
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (fileName) Mode Cnt Score Error Units
[info] BorerModelBenchmark.decodeModel reddit-scala.json thrpt 15 159.340 ± 6.725 ops/s
git reset --hard HEAD~1
sbt clean 'benchmarks/jmh:run -i 5 -wi 5 -f 3 -p fileName=reddit-scala.json BorerModelBenchmark.decode'
And the result backs to expected:
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (fileName) Mode Cnt Score Error Units
[info] BorerModelBenchmark.decodeModel reddit-scala.json thrpt 15 2964.494 ± 66.412 ops/s
case class Address[T](x: T)
implicit def addressCodec[T: Encoder: Decoder]: Codec[Address[T]] = Codec.forCaseClass[Address[T]]
The above code snippet fails with the exception:
type mismatch;
[error] found : Address[Nothing] => Option[Nothing]
[error] required: Address[T] => Option[?]
[error] implicit def addressCodec[T: Encoder: Decoder]: Codec[Address[T]] = Codec.forCaseClass[Address[T]]
[error] ^
[error] one error found
scala> io.bullet.borer.Json.decode("1".getBytes).to[Double].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Double] but got [Int]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readDouble(Reader.scala:150)
at io.bullet.borer.Decoder$.$anonfun$forDouble$1(Decoder.scala:56)
at io.bullet.borer.Decoder$.$anonfun$forDouble$1$adapted(Decoder.scala:56)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("1".getBytes).to[Float].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Float] but got [Int]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readFloat(Reader.scala:139)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1(Decoder.scala:55)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1$adapted(Decoder.scala:55)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> import io.bullet.borer._
import io.bullet.borer._
scala> import io.bullet.borer.derivation.MapBasedCodecs._
import io.bullet.borer.derivation.MapBasedCodecs._
scala> case class Foo(a: Int, b: String = "default")
defined class Foo
scala> implicit val Codec(e: Encoder[Foo], d: Decoder[Foo]) = deriveCodec[Foo]
e: io.bullet.borer.Encoder[Foo] = $anon$1@3361e546
d: io.bullet.borer.Decoder[Foo] = $anon$2@1067602b
scala> new String(io.bullet.borer.Json.encode(Foo(1)).toByteArray)
res0: String = {"a":1,"b":"default"}
Can it be some how configured during derivation to get {"a":1}
instead?
Test
case class Wrapper(xs: Array[Byte])
case class TypedWrapper[T](xs: Array[T])
import io.bullet.borer.derivation.MapBasedCodecs._
implicit def wrapperCodec: Codec[Wrapper] = deriveCodec[Wrapper]
implicit def typedWrapperCodec[T: ClassTag: Encoder: Decoder]: Codec[TypedWrapper[T]] = deriveCodec[TypedWrapper[T]]
def decode(bytes: Array[Byte]): Unit = {
println(
Cbor
.decode(bytes)
.withPrintLogging()
.to[Element]
.value
)
}
println("Wrapper codec********")
decode(Cbor.encode(Wrapper("a".getBytes)).toByteArray)
println("TypedWrapper codec********")
decode(Cbor.encode(TypedWrapper("a".getBytes)).toByteArray)
Result
[info] Wrapper codec********
[info] 1: {
[info] 1/1: "xs"
[info] 1/1: -> BYTES[61]
[info] 1: }
[info] 2: END
[info] {StringElem(xs): ByteArrayElem([B@1c1bbc4e)}
[info] TypedWrapper codec********
[info] 1: {
[info] 1/1: "xs"
[info] 1/1: -> [
[info] 1/1: 97
[info] 1/1: ]
[info] 1: }
[info] 2: END
[info] {StringElem(xs): [IntElem(97)]}
Expected
Both the arrays should get encoded as BYTES.
Workaround
type ArrayEnc[T] = Encoder[Array[T]]
type ArrayDec[T] = Decoder[Array[T]]
implicit def typedWrapperCodec[T: ClassTag: ArrayEnc: ArrayDec]: Codec[TypedWrapper[T]] = deriveCodec[TypedWrapper[T]]
scala> io.bullet.borer.Json.decode("-0.900845381675988".getBytes).to[Double].value
res0: Double = 0.900845381675988
It can be reproduced with v0.10.0:
scala> val jsonBytes = "[1,2,3]".getBytes
jsonBytes: Array[Byte] = Array(91, 49, 44, 50, 44, 51, 93)
scala> io.bullet.borer.Json.decode(jsonBytes).to[Array[Byte]].value
res0: Array[Byte] = Array(1, 2, 3)
scala> io.bullet.borer.Json.encode(res0).toByteArray
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support byte strings (Output.ToByteArray index 0)
at io.bullet.borer.json.JsonRenderer.failUnsupported(JsonRenderer.scala:275)
at io.bullet.borer.json.JsonRenderer.onBytes(JsonRenderer.scala:122)
at io.bullet.borer.Writer.writeBytes(Writer.scala:65)
at io.bullet.borer.Encoder$.$anonfun$forByteArray$1(Encoder.scala:107)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:190)
at io.bullet.borer.EncodingSetup$Impl.result(EncodingSetup.scala:138)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:117)
... 36 elided
To reproduce the stacktrace bellow need to move this line to be bellow this derivation and run this test.
java.lang.NullPointerException (input position 35)
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (input position 35)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:101)
at com.github.plokhotnyuk.jsoniter_scala.benchmark.AnyValsReading.borerJson(AnyValsReading.scala:25)
at com.github.plokhotnyuk.jsoniter_scala.benchmark.AnyValsReadingSpec.$anonfun$new$2(AnyValsReadingSpec.scala:9)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.WordSpecLike$$anon$3.apply(WordSpecLike.scala:1075)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at org.scalatest.WordSpec.withFixture(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.invokeWithFixture$1(WordSpecLike.scala:1073)
at org.scalatest.WordSpecLike.$anonfun$runTest$1(WordSpecLike.scala:1085)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286)
at org.scalatest.WordSpecLike.runTest(WordSpecLike.scala:1085)
at org.scalatest.WordSpecLike.runTest$(WordSpecLike.scala:1067)
at org.scalatest.WordSpec.runTest(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.$anonfun$runTests$1(WordSpecLike.scala:1144)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:393)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:370)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:407)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:376)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:458)
at org.scalatest.WordSpecLike.runTests(WordSpecLike.scala:1144)
at org.scalatest.WordSpecLike.runTests$(WordSpecLike.scala:1143)
at org.scalatest.WordSpec.runTests(WordSpec.scala:1881)
at org.scalatest.Suite.run(Suite.scala:1124)
at org.scalatest.Suite.run$(Suite.scala:1106)
at org.scalatest.WordSpec.org$scalatest$WordSpecLike$$super$run(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.$anonfun$run$1(WordSpecLike.scala:1189)
at org.scalatest.SuperEngine.runImpl(Engine.scala:518)
at org.scalatest.WordSpecLike.run(WordSpecLike.scala:1189)
at org.scalatest.WordSpecLike.run$(WordSpecLike.scala:1187)
at org.scalatest.WordSpec.run(WordSpec.scala:1881)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1349)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1343)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1343)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1033)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1011)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1509)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1011)
at org.scalatest.tools.Runner$.run(Runner.scala:850)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
Caused by: java.lang.NullPointerException
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.InputReader.apply(Reader.scala:63)
at io.bullet.borer.DecoderFromApply.$anonfun$from$2(DecoderFromApply.scala:18)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 51 more
I do not see deriveUnaryDecoder
in ArrayBasedCodecs
.
On the other hand, ArrayBasedCodecs.deriveUnaryEncoder
is there.
Any technical reason for the asymmetry? Because of this, currently it is not possible to write unary encoder and decoder separately for a type.
Currently big numbers in Java has a binary representation under hood, and parsing them from the decimal string representation has O(n^2)
complexity.
scala> def timed[A](f: => A): A = { val t = System.currentTimeMillis; val r = f; println(s"Elapsed time (ms): ${System.currentTimeMillis - t}"); r }
timed: [A](f: => A)A
scala> timed(io.bullet.borer.Json.decode(("9" * 1000000).getBytes).to[BigInt].value)
Elapsed time (ms): 13151
scala> timed(io.bullet.borer.Json.decode(("9" * 1000000).getBytes).to[BigDecimal].value)
Elapsed time (ms): 13165
For a contemporary 1Gbit network 10ms of receiving a malicious message with a 1000000-digit number can produce 1300ms of 100% CPU load in a one core. More than 100 cores can be effectively DoS-ed at a full bandwidth rate.
Also, a lot of arithmetic operations has O(n^1.5) complexity, even such "safe" operations as +
for a not validated scale of the input value can DoS much more efficiently by a single message with a 12-digit number:
scala> timed(io.bullet.borer.Json.decode(("1e-100000000").getBytes).to[BigDecimal].value)
Elapsed time (ms): 0
scala> timed(io.bullet.borer.Json.decode(("1e-100000000").getBytes).to[BigDecimal].value + 1)
Elapsed time (ms): 91591
So, parse them with taking in account these worst cases, especially for the systems that are opened for the wild wide world:
O(n^1.5)
complexity, like here.[error] /home/andriy/Projects/com/github/plokhotnyuk/borer/core/src/main/scala/io/bullet/borer/input/FromByteBufferInput.scala:29:30: ambiguous reference to overloaded definition,
[error] both method position in class ByteBuffer of type (x$1: Int)java.nio.ByteBuffer
[error] and method position in class Buffer of type ()Int
[error] match expected type ?
[error] buffer.position(buffer.position - numberOfBytes)
[error] ^
[error] one error found
[error] (core / Compile / compileIncremental) Compilation failed
Following does not work
sealed trait Adt
case class Err(reason: String) extends Adt
case object Ok extends Adt
implicit lazy val errCodec: Codec[Err] = ArrayBasedCodecs.deriveCodecForUnaryCaseClass[Err]
implicit lazy val okCodec: Codec[Ok.type] = deriveCodec[Ok.type]
implicit lazy val adtCodec: Codec[Adt] = deriveCodec[Adt]
Compilation Error
Error:(16, 58) exception during macro expansion:
scala.ScalaReflectionException: object Ok is not a module
Workaround
def singletonCodec[T <: Singleton](x: T): Codec[T] = Codec.implicitly[String].bimap[T](_.toString, _ => x)
implicit lazy val okCodec: Codec[Ok.type] = singletonCodec(Ok)
scala> io.bullet.borer.Json.decode((""""\uD834\uDD1E"""").getBytes).to[String].value
res0: String = 𝄞
scala> io.bullet.borer.Json.decode((""""\uDD1E\uD834"""").getBytes).to[String].value
res1: String = ??
Currently borer only implements "semi-automatic" codec derivation, which means that it can construct codecs for case classes and ADT super-types with a single deriveCodec
line, but requires that codecs for all members (case classes) or sub-types (ADTs) are already implicitly available.
Fully automatic derivation means, that a single deriveCodec[T]
will suffice to have borer construct a codec for T
and all member-/sub-types.
Among other things this requires
Please add a configuration parameter for the Target
methods.
Implementation of it can use own default values, like here:
def encode[T: Encoder](value: T, config: EncodingConfig = EncodingConfig.default): EncodingSetup.JsonApi[T, EncodingConfig] =
new EncodingSetup.Impl(value, Json, config, Receiver.nopWrapper, JsonRenderer)
def decode[T](input: T, config: DecodingConfig = DecodingConfig.default)(implicit w: Input.Wrapper[T]): DecodingSetup.Api[w.In, DecodingConfig] =
We have successfully managed to use single set of user defined (map-based) codecs for a small subset of our domain models and achieve 3 distinct tasks: send them in akka-remote message payloads (CBOR), send some of them to Redis (CBOR) and serve them via http (JSON).
We are very happy with the outcome and looking forward to the possibility of replacing 3 different libraries (chill, scalapb and play-json) with a single library.
One feedback from the team is around nested
encoding of ADTs which becomes apparent mainly during the Json output as:
{
"Dog" : {
"age": 10,
"name": "gabbar"
}
}
Will it be possible to add a configuration which allows it to be encoded flat
as below? Consuming this from say a python library is much easier it seems (cc/ @abrighton)
{
"type": "Dog",
"age": 10,
"name": "gabbar"
}
Parsing:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val obj = io.bullet.borer.Json.decode("""{"n":{"n":{"n":{}}}}""".getBytes).to[Nested].value
println(obj)
// Exiting paste mode, now interpreting.
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (input position 1)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:101)
... 44 elided
Caused by: java.lang.NullPointerException
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.Decoder$$anon$1.$anonfun$withDefaultValue$1(Decoder.scala:188)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 44 more
Serialization:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val bytes = io.bullet.borer.Json.encode(Nested(Some(Nested(None)))).toByteArray
println(new String(bytes))
// Exiting paste mode, now interpreting.
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (Output.ToByteArray index 4)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:131)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:112)
... 36 elided
Caused by: java.lang.NullPointerException
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.Encoder$$anon$1$$anon$2.write(Encoder.scala:177)
at io.bullet.borer.Encoder$$anon$1$$anon$2.write(Encoder.scala:173)
at io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$.rec$1(MapBasedCodecs.scala:55)
at io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$.$anonfun$combine$1(MapBasedCodecs.scala:77)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:180)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:128)
... 37 more
The same code when enclosed in curly brackets defers derivation until first call to decoder/encoder and produces SOE:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
{
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val obj = io.bullet.borer.Json.decode("""{"n":{"n":{"n":{}}}}""".getBytes).to[Nested].value
println(obj)
}
// Exiting paste mode, now interpreting.
java.lang.StackOverflowError
at Nested$2$.nestedTypeclass$macro$1$lzycompute$1(<pastie>:19)
at Nested$2$.nestedTypeclass$macro$1$1(<pastie>:19)
... 1022 elided
scala> io.bullet.borer.Json.decode(("1\u0030").getBytes).to[Int].value
res0: Int = 10
scala> io.bullet.borer.Json.decode(("[true\u002Cfalse]").getBytes).to[Array[Boolean]].value
res1: Array[Boolean] = Array(true, false)
Current code from the master parses floats with rounding where error ~1ULP that is greater than expected ~0.5ULP of java.lang.Float.valueOf()
.
The rounding error can be easy reproduced when parsing string representation of some values with number of digits greater than usually used for floats.
scala> "1.00000017881393432617187499".toFloat
res0: Float = 1.0000001
scala> "1.00000017881393432617187499".toDouble.toFloat
res1: Float = 1.0000002
The detailed explanation is in this comment
The following code can print lot of such numbers after increasing number of iterations:
scala> :paste
// Entering paste mode (ctrl-D to finish)
(1 to 1000).foreach { _ =>
def checkAndPrint(input: String): Unit = {
val actualOutput = io.bullet.borer.Json.decode(input.getBytes).to[Float].value
val expectedOutput = input.toFloat
if (actualOutput != expectedOutput) {
println(s"input = $input, expectedOutput =$expectedOutput, actualOutput = $actualOutput")
}
}
val n = java.util.concurrent.ThreadLocalRandom.current().nextLong()
val x = java.lang.Double.longBitsToDouble(n & ~0xFFFFFFFL)
if (java.lang.Float.isFinite(x.toFloat) && x.toFloat != 0.0) checkAndPrint(x.toString)
}
// Exiting paste mode, now interpreting.
input = 5.726278970996646E-7, expectedOutput =5.726279E-7, actualOutput = 5.7262787E-7
input = -1.502594932252865E35, expectedOutput =-1.5025949E35, actualOutput = -1.502595E35
input = -2.053233379325646E20, expectedOutput =-2.0532335E20, actualOutput = -2.0532333E20
input = 1.511814678325955E20, expectedOutput =1.5118146E20, actualOutput = 1.5118148E20
input = 4.355016683746335E19, expectedOutput =4.3550165E19, actualOutput = 4.355017E19
input = 1.704752798329881E25, expectedOutput =1.7047527E25, actualOutput = 1.7047529E25
input = -5.250234028812652E31, expectedOutput =-5.250234E31, actualOutput = -5.2502343E31
I think it should be documented if there are no other option available.
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> case class A(b: Option[Int] = None, c: Option[String] = None)
defined class A
scala> import io.bullet.borer._
<console>:11: warning: Unused import
import io.bullet.borer._
^
import io.bullet.borer._
scala> import io.bullet.borer.derivation.MapBasedCodecs._
<console>:10: warning: Unused import
import io.bullet.borer._
^
<console>:14: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
import io.bullet.borer.derivation.MapBasedCodecs._
scala> implicit val c = deriveCodec[A]
<console>:12: warning: Unused import
import io.bullet.borer._
^
c: io.bullet.borer.Codec[A] = Codec(io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$$$Lambda$5709/1448885594@76b91f1a,io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$$$Lambda$5732/1761122012@1da255b7)
scala> Json.decode("""{}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res0: A = A(None,None)
scala> Json.decode("""{"b":1}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res1: A = A(Some(1),None)
scala> Json.decode("""{"c":"x"}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res2: A = A(None,Some(x))
scala> Json.decode("""{"b":1,"c":"x"}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res3: A = A(Some(1),Some(x))
scala> Json.decode("""{"b":1,"c":null}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
<console>:10: warning: Unused import
import io.bullet.borer._
^
<console>:13: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
<console>:15: warning: Unused import
import c
^
io.bullet.borer.Borer$Error$InvalidInputData: Expected String or Text Bytes but got Null (input position 7)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:442)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:438)
at io.bullet.borer.InputReader.readString(Reader.scala:239)
at io.bullet.borer.Decoder$.$anonfun$forString$1(Decoder.scala:68)
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.Decoder$$anon$1.$anonfun$withDefaultValue$1(Decoder.scala:188)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 36 elided
Test
val bytes = "abc".getBytes()
Json.encode(bytes).toUtf8String
Error
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support byte strings (Output.ToByteArray index 0)
Related
All of the following works
Json.encode(bytes.head).toUtf8String
Cbor.encode(bytes.head).toByteArray
Cbor.encode(bytes).toByteArray
Workaround
implicit lazy val bytesEnc: Encoder[Array[Byte]] = targetSpecificEnc(
cborEnc = Encoder.forByteArray,
jsonEnc = Encoder.forArray[Byte]
)
def targetSpecificEnc[T](cborEnc: Encoder[T], jsonEnc: Encoder[T]): Encoder[T] = { (writer, value) ⇒
val enc = if (writer.target == Cbor) cborEnc else jsonEnc
enc.write(writer, value)
}
We had to write an analogous target specific decoder for Array[Bytes]
I tried this:
implicit val intArrEnc: Encoder[Array[Int]] = Encoder.forArray[Integer].compose[Array[Int]](_.map(x => x: Integer))
implicit val intArrDec: Decoder[Array[Int]] = Decoder.forArray[Integer].map[Array[Int]](_.map(x => x: Int))
val intArr = Array(1, 2, 3, 4)
val bytes: Array[Byte] = Cbor.encode(intArr).toByteArray
val blob: Array[Int] = Cbor.decode(bytes).to[Array[Int]].value
I get this error:
Exception in thread "main" io.bullet.borer.Borer$Error$General: java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.lang.Integer; ([Ljava.lang.Object; and [Ljava.lang.Integer; are in module java.base of loader 'bootstrap')
at io.bullet.borer.Borer$DecodingSetup.to(Borer.scala:265)
at spikes.Demo$.main(Blob.scala:60)
at spikes.Demo.main(Blob.scala)
Caused by: java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.lang.Integer; ([Ljava.lang.Object; and [Ljava.lang.Integer; are in module java.base of loader 'bootstrap')
at io.bullet.borer.Decoder$DecoderOps$.$anonfun$map$1(Decoder.scala:40)
at io.bullet.borer.Decoder$.$anonfun$apply$1(Decoder.scala:32)
at io.bullet.borer.Borer$DecodingSetup.to(Borer.scala:260)
... 2 more
Test
def getEncoder[T](implicit x: Encoder[Array[T]]): Encoder[Array[T]] = x
val enc: Encoder[Array[java.lang.Byte]] = Encoder.forByteArray.contramap(_.map(x ⇒ x: Byte))
val dec: Decoder[Array[java.lang.Byte]] = Decoder.forByteArray.map(_.map(x ⇒ x: java.lang.Byte))
implicit val codec: Codec[Array[java.lang.Byte]] = Codec(enc, dec)
assert(getEncoder[java.lang.Byte] eq enc)
Code to reproduce:
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
sealed trait Geometry extends Product with Serializable
case class Point(coordinates: (Double, Double)) extends Geometry
case class MultiPoint(coordinates: IndexedSeq[(Double, Double)] = Vector.empty) extends Geometry
case class LineString(coordinates: IndexedSeq[(Double, Double)] = Vector.empty) extends Geometry
case class MultiLineString(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]] = Vector.empty) extends Geometry
case class Polygon(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]] = Vector.empty) extends Geometry
case class MultiPolygon(coordinates: IndexedSeq[IndexedSeq[IndexedSeq[(Double, Double)]]] = Vector.empty) extends Geometry
case class GeometryCollection(geometries: IndexedSeq[Geometry] = Vector.empty) extends Geometry
sealed trait GeoJSON extends Product with Serializable
case class Feature(properties: Map[String, String] = Map.empty, geometry: Geometry, bbox: Option[(Double, Double, Double, Double)] = None) extends GeoJSON
case class FeatureCollection(features: IndexedSeq[GeoJSON] = Vector.empty, bbox: Option[(Double, Double, Double, Double)] = None) extends GeoJSON
implicit val flatAdtEncoding: AdtEncodingStrategy = AdtEncodingStrategy.flat(typeMemberName = "type")
implicit val Codec(geoJsonEnc: Encoder[GeoJSON], geoJsonDec: Decoder[GeoJSON]) = {
implicit lazy val c1: Codec[Geometry] = {
implicit val c11: Codec[Point] = deriveCodec
implicit val c12: Codec[MultiPoint] = deriveCodec
implicit val c13: Codec[LineString] = deriveCodec
implicit val c14: Codec[MultiLineString] = deriveCodec
implicit val c15: Codec[Polygon] = deriveCodec
implicit val c16: Codec[MultiPolygon] = deriveCodec
implicit val c17: Codec[GeometryCollection] = deriveCodec
deriveCodec[Geometry]
}
implicit val c2: Codec[Feature] = deriveCodec
implicit val c3: Codec[FeatureCollection] = deriveCodec
deriveCodec[GeoJSON]
}
// Exiting paste mode, now interpreting.
...
scala> new String(io.bullet.borer.Json.encode[GeoJSON](FeatureCollection(Vector(Feature(geometry = Polygon(Vector(Vector((7.697223,47.543327)))))))).toByteArray)
...
res0: String = {"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[7.697223,47.543327]]]}}]}
scala> io.bullet.borer.Json.decode(res0.getBytes).to[GeoJSON].value
...
io.bullet.borer.Borer$Error$InvalidInputData: Expected Map for decoding an instance of type `GeoJSON` but got Start of unbounded Map (input position 41)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:617)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:613)
at io.bullet.borer.AdtEncodingStrategy$$anon$1.failNoMap$1(AdtEncodingStrategy.scala:175)
at io.bullet.borer.AdtEncodingStrategy$$anon$1.readAdtEnvelopeOpen(AdtEncodingStrategy.scala:234)
at .$anonfun$x$1$4(<console>:32)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at io.bullet.borer.InputReader.rec$1(Reader.scala:520)
at io.bullet.borer.InputReader.readUntilBreak(Reader.scala:521)
at io.bullet.borer.Decoder$.$anonfun$forIterable$1(Decoder.scala:255)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at FeatureCollectionDecoder$1.readObject$9(<console>:31)
at FeatureCollectionDecoder$1.read(<console>:31)
at $anon$18.read(<console>:31)
at $anon$18.read(<console>:31)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at .$anonfun$x$1$4(<console>:32)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:169)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:100)
... 36 elided
By default Scala use MathContext.DECIMAL128
with precision up to 34 decimal digits for BigDecimal
instances, but in current version (0.8.1-SNAPSHOT) the limit is set to 32 and configs cannot be passed as parameters into encode
and decode
methods:
io.bullet.borer.Borer$Error$Overflow: JSON number mantissa longer than configured maximum of 32 digits [input position 1]
In previous version it worked fine:
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
def stringCodec[T](f: String => T): Codec[T] =
Codec((w: Writer, value: T) => w.writeString(value.toString), (r: Reader) => f(r.readString()))
implicit val Codec(charEnc: Encoder[Char], charDec: Decoder[Char]) = stringCodec(_.charAt(0))
case class HasChar(ch: Char)
implicit val Codec(hasCharEnc: Encoder[HasChar], hasCharDec: Decoder[HasChar]) = deriveCodec[HasChar]
io.bullet.borer.Json.decode("""{"ch":"F"}""".getBytes).to[HasChar].value
// Exiting paste mode, now interpreting.
...
res0: HasChar = HasChar(F)
Now it tries to parse chars as ints without taking in account a custom codec provided by the implicit and throws the following error:
io.bullet.borer.Borer$Error$InvalidInputData: Expected Char but got Chars (input position 6)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:536)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:532)
at io.bullet.borer.InputReader.readChar(Reader.scala:85)
at HasCharDecoder$1.readObject$1(<console>:1)
at HasCharDecoder$1.read(<console>:1)
at $anon$2.read(<console>:1)
at $anon$2.read(<console>:1)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:99)
... 36 elided
But for some cases like bellow it picks the custom codec:
scala> io.bullet.borer.Json.decode("""["F"]""".getBytes).to[List[Char]].value
...
res1: List[Char] = List(F)
Currently we do import io.bullet.borer.derivation.MapBasedCodecs._
and then use deriveCodec[T]
for all case classes except the unary case classes for which we use Codec.forCaseClass[T]
.
This works, but is prone to error. We may end up using Codec.forCaseClass[T]
for non-unary case classes, especially during the refactoring, resulting in mix of array-based and map-based encodings.
It will be good to have a dedicated function for the unary case classes which fails to compile if the case class is not unary.
Compilation error looks like:
Error:(15, 51) no arguments allowed for nullary constructor JsonTypeInfo: ()com.fasterxml.jackson.annotation.JsonTypeInfo
Note that 'use', 'property' are not parameter names of the invoked method.
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
Error:(15, 51) no arguments allowed for nullary constructor JsonSubTypes: ()com.fasterxml.jackson.annotation.JsonSubTypes
Note that 'value' is not a parameter name of the invoked method.
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
Code to reproduce:
import com.fasterxml.jackson.annotation.JsonSubTypes.Type
import com.fasterxml.jackson.annotation.{JsonSubTypes, JsonTypeInfo}
import scala.collection.immutable.IndexedSeq
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes(Array(
new Type(value = classOf[Point], name = "Point"),
new Type(value = classOf[MultiPoint], name = "MultiPoint"),
new Type(value = classOf[LineString], name = "LineString"),
new Type(value = classOf[MultiLineString], name = "MultiLineString"),
new Type(value = classOf[Polygon], name = "Polygon"),
new Type(value = classOf[MultiPolygon], name = "MultiPolygon"),
new Type(value = classOf[GeometryCollection], name = "GeometryCollection")))
sealed trait Geometry extends Product with Serializable
case class Point(coordinates: (Double, Double)) extends Geometry
case class MultiPoint(coordinates: IndexedSeq[(Double, Double)]) extends Geometry
case class LineString(coordinates: IndexedSeq[(Double, Double)]) extends Geometry
case class MultiLineString(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]]) extends Geometry
case class Polygon(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]]) extends Geometry
case class MultiPolygon(coordinates: IndexedSeq[IndexedSeq[IndexedSeq[(Double, Double)]]]) extends Geometry
case class GeometryCollection(geometries: IndexedSeq[Geometry]) extends Geometry
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes(Array(
new Type(value = classOf[Feature], name = "Feature"),
new Type(value = classOf[FeatureCollection], name = "FeatureCollection")))
sealed trait GeoJSON extends Product with Serializable
case class Feature(properties: Map[String, String], geometry: Geometry) extends GeoJSON
case class FeatureCollection(features: IndexedSeq[GeoJSON]) extends GeoJSON
implicit val c1: Codec[Point] = deriveCodec[Point]
implicit val c2: Codec[MultiPoint] = deriveCodec[MultiPoint]
implicit val c3: Codec[LineString] = deriveCodec[LineString]
implicit val c4: Codec[MultiLineString] = deriveCodec[MultiLineString]
implicit val c5: Codec[Polygon] = deriveCodec[Polygon]
implicit val c6: Codec[MultiPolygon] = deriveCodec[MultiPolygon]
implicit val c7: Codec[GeometryCollection] = deriveCodec[GeometryCollection]
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
implicit val c9: Codec[Feature] = deriveCodec[Feature]
implicit val c10: Codec[FeatureCollection] = deriveCodec[FeatureCollection]
implicit val c11: Codec[GeoJSON] = deriveCodec[GeoJSON]
scala> io.bullet.borer.Json.decode(("[" + "1," * 10000 + "l" + "]").getBytes).to[Array[Int]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:367)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.readInt(Reader.scala:104)
at io.bullet.borer.Decoder$.$anonfun$forInt$1(Decoder.scala:53)
at io.bullet.borer.Decoder$.$anonfun$forInt$1$adapted(Decoder.scala:53)
at io.bullet.borer.InputReader.read(Reader.scala:311)
at io.bullet.borer.Decoder$.$anonfun$forArray$2(Decoder.scala:181)
at io.bullet.borer.InputReader.rec$3(Reader.scala:340)
at io.bullet.borer.InputReader.readUntilBreak(Reader.scala:341)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:181)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.encode(BigDecimal(100)).toByteArray
res0: Array[Byte] = Array(49, 48, 48)
scala> io.bullet.borer.Json.encode(BigDecimal(100.1)).toByteArray
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support CBOR tags [Output.ToByteArray index 0]
at io.bullet.borer.json.JsonRenderer.failUnsupported(JsonRenderer.scala:279)
at io.bullet.borer.json.JsonRenderer.onTag(JsonRenderer.scala:221)
at io.bullet.borer.Writer.writeTag(Writer.scala:67)
at io.bullet.borer.Encoder$.$anonfun$forJBigDecimal$1(Encoder.scala:130)
at io.bullet.borer.Encoder$EncoderOps$.$anonfun$contramap$1(Encoder.scala:80)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:165)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:113)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:101)
... 36 elided
W/A is to add a custom encoder:
scala> implicit val bigDecimalEnc: io.bullet.borer.Encoder[BigDecimal] = io.bullet.borer.Encoder.apply[BigDecimal]((w, x) => w.writeNumberString(x.bigDecimal.toString))
bigDecimalEnc: io.bullet.borer.Encoder[BigDecimal] = $$Lambda$6643/1040047064@37e72351
scala> io.bullet.borer.Json.encode(BigDecimal(100.1)).toByteArray
res1: Array[Byte] = Array(49, 48, 48, 46, 49)
scala> io.bullet.borer.Json.decode(("[0,::,1]").getBytes).to[Array[Int]].value
res0: Array[Int] = Array(0, 1)
scala> io.bullet.borer.Json.decode(("[false:true]").getBytes).to[Array[Boolean]].value
res1: Array[Boolean] = Array(false, true)
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[Long]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[BigInt]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[BigDecimal]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
This is while upgrading to 0.9.0
Test
case class Box(id: String)
object Box {
def apply(): Box = new Box("default")
}
implicit val boxCodec: Codec[Box] = Codec.forCaseClass[Box]
val array = Cbor.encode(Box("abc")).toByteArray
Cbor.decode(array).to[Box].value
Error
io.bullet.borer.Borer$Error$InvalidInputData: Expected Array Header (0) but got Text (input position 0)
[info] at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:442)
[info] at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:438)
[info] at io.bullet.borer.InputReader.readArrayHeader(Reader.scala:295)
[info] at io.bullet.borer.InputReader.readArrayHeader(Reader.scala:292)
[info] at io.bullet.borer.DecoderFromApply.$anonfun$from$1(DecoderFromApply.scala:15)
[info] at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
[info] at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
scala> new String(io.bullet.borer.Json.encode(1.1999999f).toByteArray)
res0: String = 1.1999999284744263
I would recommend to peek and adopt the Ryu algorithm for serialization of both Float
and Double
types - it is much performant and some values can be serialized using less digits than Java's toString
.
Often we want slightly different codecs for Json vs Cbor. We have defined the helpers like these:
def targetSpecificEnc[T](cborEnc: Encoder[T], jsonEnc: Encoder[T]): Encoder[T] = { (writer, value) ⇒
val enc = if (writer.target == Cbor) cborEnc else jsonEnc
enc.write(writer, value)
}
def targetSpecificDec[T](cborDec: Decoder[T], jsonDec: Decoder[T]): Decoder[T] = { reader ⇒
val dec = if (reader.target == Cbor) cborDec else jsonDec
dec.read(reader)
}
Which we use to define Instant codecs like these:
implicit lazy val instantEnc: Encoder[Instant] = CborHelpers.targetSpecificEnc(
cborEnc = Encoder.forTuple2[Long, Long].contramap(x => (x.getEpochSecond, x.getNano)),
jsonEnc = Encoder.forString.contramap(_.toString)
)
implicit lazy val instantDec: Decoder[Instant] = CborHelpers.targetSpecificDec(
cborDec = Decoder.forTuple2[Long, Long].map(t => Instant.ofEpochSecond(t._1, t._2)),
jsonDec = Decoder.forString.map(Instant.parse)
)
I hope this is the right way to go about doing this. If so, it will be good to have these helpers in the library.
Test
sealed trait Animal
case class Tiger(name: String) extends Animal
object Animal {
import io.bullet.borer.derivation.MapBasedCodecs._
implicit val tigerCodec: Codec[Tiger] = deriveCodec[Tiger]
implicit val animalCodec: Codec[Animal] = deriveCodec[Animal]
}
val tiger = Tiger("king")
val byteArray = Cbor.encode(tiger).toByteArray
Cbor.decode(byteArray).to[Animal].value
Exception
io.bullet.borer.Borer$Error$UnexpectedDataItem:
Unexpected data item:
Expected [Array with 2 elements for decoding an instance of type [csw.params.core.formats.Animal]] but got [Map Header (1)]
Workaround
object Animal {
import io.bullet.borer.derivation.MapBasedCodecs._
implicit lazy val animalCodec: Codec[Animal] = {
implicit val tigerCodec: Codec[Tiger] = deriveCodec[Tiger]
deriveCodec[Animal]
}
implicit lazy val tigerCodec: Codec[Tiger] = Codec(
animalCodec.encoder.compose(identity),
animalCodec.decoder.map(_.asInstanceOf[Tiger])
)
}
since publishing to oss.sonatype.org is increasingly becoming a major hassle due to severe upload performance issues on their oss platform.
Test
Below code compiles, but throws RuntimeException during the run
package demo
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
sealed trait A
sealed trait B extends A
sealed trait C extends A
case class D(a: Int) extends B with C
object DiamondApp {
def main(args: Array[String]): Unit = {
implicit val a: Codec[A] = deriveCodec[A]
}
}
Exception during run
java.lang.RuntimeException: @key collision: At least two subtypes of `demo.A` share the same type id `D`
Any possible workaround?
We have an ADT like this, for which we have written the codec only for the base type and it compiles successfully.
sealed trait A
case class B(a: Int) extends A
case class C(a: Int) extends A
implicit val a: Codec[A] = deriveCodec[A]
If we want to serialize/deserialize the values as (compile-time) type A
, it works fine.
The docs suggest that we should write similar codecs for B
and C
too, but I guess that is only if we want to use specific compile-time types for ser/de. Is this correct? Is there any performance difference?
Maybe, something to clarify in the docs.
scala> "1.1999999".toFloat
res0: Float = 1.1999999
scala> io.bullet.borer.Json.decode("1.1999999".getBytes).to[Float].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Float] but got [Double]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readFloat(Reader.scala:139)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1(Decoder.scala:55)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1$adapted(Decoder.scala:55)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
Workaround:
import java.lang.{Byte ⇒ JByte}
implicit val javaByteArrayEnc: Encoder[Array[JByte]] = Encoder.forByteArray.compose(javaArray ⇒ javaArray.map(x ⇒ x: Byte))
implicit val javaByteArrayDec: Decoder[Array[JByte]] = Decoder.forByteArray.map(scalaArray ⇒ scalaArray.map(x ⇒ x: JByte))
Test
val input = "0" * 1022
val payload: Array[Byte] = input.getBytes("utf-8")
val byteBuffer = Cbor.encode(payload).to[ByteBuffer].result
val value = Cbor.decode(byteBuffer).to[Array[Byte]].value
val output = new String(value, "utf8")
output shouldBe input
Exception
io.bullet.borer.Borer$Error$InvalidInputData: Expected End of Input but got Int (input position 1025)
Related
using val input = "0" * 1021
or less works. Also, using Array[Byte] instead of ByteBuffer works for large arrays as well.
Akka-http handles (un)marshalling of a few types like Option[T]
and Either[A, B]
in a special way.
Akka-http always picks up its own (un)marshallers for them ignoring defaultBorerUnmarshaller
and defaultBorerMarshaller
provided by borer's akkaHttp
.
Inn our project we copy the following in the immediate lexical scope to give them a higher priority.
liftMarshaller
from LowPriorityToResponseMarshallerImplicits
andmessageUnmarshallerFromEntityUnmarshaller
from LowerPriorityGenericUnmarshallers
Will it make sense to add them akkaHttp
of the borer-compat-akka
module?
def bimap[From: Encoder: Decoder, To](to: From ⇒ To, from: To ⇒ From): Codec[To] = Codec(
implicitly[Encoder[From]].contramap(from),
implicitly[Decoder[From]].map(to)
)
GenCodec
with CborInput/CborOutput
from AVSystems scala-commons.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.