Efficient CBOR and JSON (de)serialization in Scala
Home Page: https://sirthias.github.io/borer/
License: Mozilla Public License 2.0
Efficient CBOR and JSON (de)serialization for Scala.
See the project website for all documentation.
![mergify[bot] avatar](https://avatars.githubusercontent.com/in/10562?v=4 "mergify[bot]")
Test
sealed trait Animal
case class Tiger(name: String) extends Animal
object Animal {
import io.bullet.borer.derivation.MapBasedCodecs._
implicit val tigerCodec: Codec[Tiger] = deriveCodec[Tiger]
implicit val animalCodec: Codec[Animal] = deriveCodec[Animal]
}
val tiger = Tiger("king")
val byteArray = Cbor.encode(tiger).toByteArray
Cbor.decode(byteArray).to[Animal].valueException
io.bullet.borer.Borer$Error$UnexpectedDataItem:
Unexpected data item:
Expected [Array with 2 elements for decoding an instance of type [csw.params.core.formats.Animal]] but got [Map Header (1)]Workaround
object Animal {
import io.bullet.borer.derivation.MapBasedCodecs._
implicit lazy val animalCodec: Codec[Animal] = {
implicit val tigerCodec: Codec[Tiger] = deriveCodec[Tiger]
deriveCodec[Animal]
}
implicit lazy val tigerCodec: Codec[Tiger] = Codec(
animalCodec.encoder.compose(identity),
animalCodec.decoder.map(_.asInstanceOf[Tiger])
)
}scala> import io.bullet.borer._
import io.bullet.borer._
scala> import io.bullet.borer.derivation.MapBasedCodecs._
import io.bullet.borer.derivation.MapBasedCodecs._
scala> case class Foo(a: Int, b: String = "default")
defined class Foo
scala> implicit val Codec(e: Encoder[Foo], d: Decoder[Foo]) = deriveCodec[Foo]
e: io.bullet.borer.Encoder[Foo] = $anon$1@3361e546
d: io.bullet.borer.Decoder[Foo] = $anon$2@1067602b
scala> new String(io.bullet.borer.Json.encode(Foo(1)).toByteArray)
res0: String = {"a":1,"b":"default"}
Can it be some how configured during derivation to get {"a":1} instead?
scala> io.bullet.borer.Json.encode(BigDecimal(100)).toByteArray
res0: Array[Byte] = Array(49, 48, 48)
scala> io.bullet.borer.Json.encode(BigDecimal(100.1)).toByteArray
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support CBOR tags [Output.ToByteArray index 0]
at io.bullet.borer.json.JsonRenderer.failUnsupported(JsonRenderer.scala:279)
at io.bullet.borer.json.JsonRenderer.onTag(JsonRenderer.scala:221)
at io.bullet.borer.Writer.writeTag(Writer.scala:67)
at io.bullet.borer.Encoder$.$anonfun$forJBigDecimal$1(Encoder.scala:130)
at io.bullet.borer.Encoder$EncoderOps$.$anonfun$contramap$1(Encoder.scala:80)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:165)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:113)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:101)
... 36 elided
W/A is to add a custom encoder:
scala> implicit val bigDecimalEnc: io.bullet.borer.Encoder[BigDecimal] = io.bullet.borer.Encoder.apply[BigDecimal]((w, x) => w.writeNumberString(x.bigDecimal.toString))
bigDecimalEnc: io.bullet.borer.Encoder[BigDecimal] = $$Lambda$6643/1040047064@37e72351
scala> io.bullet.borer.Json.encode(BigDecimal(100.1)).toByteArray
res1: Array[Byte] = Array(49, 48, 48, 46, 49)
since publishing to oss.sonatype.org is increasingly becoming a major hassle due to severe upload performance issues on their oss platform.
According to the flamegraph report of run with Async Profiler ~95% CPU is spending in call of io/bullet/borer/derivation/MapBasedCodecs$deriveDecoder$.verifyNoDuplicate$1:
How to reproduce:
Reddit.User last 10 fields were removed, and run the model benchmark for Borer that should parse reddit-scala.json:git clone [email protected]:plokhotnyuk/borer.git
cd borer
git checkout unexpected-troughput-drop
sbt clean 'benchmarks/jmh:run -i 5 -wi 5 -f 3 -p fileName=reddit-scala.json BorerModelBenchmark.decode'
And you will get results like this:
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (fileName) Mode Cnt Score Error Units
[info] BorerModelBenchmark.decodeModel reddit-scala.json thrpt 15 159.340 ± 6.725 ops/s
git reset --hard HEAD~1
sbt clean 'benchmarks/jmh:run -i 5 -wi 5 -f 3 -p fileName=reddit-scala.json BorerModelBenchmark.decode'
And the result backs to expected:
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (fileName) Mode Cnt Score Error Units
[info] BorerModelBenchmark.decodeModel reddit-scala.json thrpt 15 2964.494 ± 66.412 ops/s
scala> io.bullet.borer.Json.decode(("[0,::,1]").getBytes).to[Array[Int]].value
res0: Array[Int] = Array(0, 1)
scala> io.bullet.borer.Json.decode(("[false:true]").getBytes).to[Array[Boolean]].value
res1: Array[Boolean] = Array(false, true)
def bimap[From: Encoder: Decoder, To](to: From ⇒ To, from: To ⇒ From): Codec[To] = Codec(
implicitly[Encoder[From]].contramap(from),
implicitly[Decoder[From]].map(to)
)scala> new String(io.bullet.borer.Json.encode(1.1999999f).toByteArray)
res0: String = 1.1999999284744263
I would recommend to peek and adopt the Ryu algorithm for serialization of both Float and Double types - it is much performant and some values can be serialized using less digits than Java's toString.
By default Scala use MathContext.DECIMAL128 with precision up to 34 decimal digits for BigDecimal instances, but in current version (0.8.1-SNAPSHOT) the limit is set to 32 and configs cannot be passed as parameters into encode and decode methods:
io.bullet.borer.Borer$Error$Overflow: JSON number mantissa longer than configured maximum of 32 digits [input position 1]
Test
val input = "0" * 1022
val payload: Array[Byte] = input.getBytes("utf-8")
val byteBuffer = Cbor.encode(payload).to[ByteBuffer].result
val value = Cbor.decode(byteBuffer).to[Array[Byte]].value
val output = new String(value, "utf8")
output shouldBe inputException
io.bullet.borer.Borer$Error$InvalidInputData: Expected End of Input but got Int (input position 1025)Related
using val input = "0" * 1021 or less works. Also, using Array[Byte] instead of ByteBuffer works for large arrays as well.
Workaround:
import java.lang.{Byte ⇒ JByte}
implicit val javaByteArrayEnc: Encoder[Array[JByte]] = Encoder.forByteArray.compose(javaArray ⇒ javaArray.map(x ⇒ x: Byte))
implicit val javaByteArrayDec: Decoder[Array[JByte]] = Decoder.forByteArray.map(scalaArray ⇒ scalaArray.map(x ⇒ x: JByte))
Currently borer only implements "semi-automatic" codec derivation, which means that it can construct codecs for case classes and ADT super-types with a single deriveCodec line, but requires that codecs for all members (case classes) or sub-types (ADTs) are already implicitly available.
Fully automatic derivation means, that a single deriveCodec[T] will suffice to have borer construct a codec for T and all member-/sub-types.
Among other things this requires
Latest borer-compat-akka exposes object akkaHttp which is a great addition!
Can we also have a companion trait akkaHttp with the same content, but allows users to extend/mix-in and add additional (un)marshalling support as needed?
For example, if we do not fix #62, I would like to add those two (un)marshallers in the extended HttpCodecs in my app.
This is while upgrading to 0.9.0
Test
case class Box(id: String)
object Box {
def apply(): Box = new Box("default")
}
implicit val boxCodec: Codec[Box] = Codec.forCaseClass[Box]
val array = Cbor.encode(Box("abc")).toByteArray
Cbor.decode(array).to[Box].valueError
io.bullet.borer.Borer$Error$InvalidInputData: Expected Array Header (0) but got Text (input position 0)
[info] at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:442)
[info] at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:438)
[info] at io.bullet.borer.InputReader.readArrayHeader(Reader.scala:295)
[info] at io.bullet.borer.InputReader.readArrayHeader(Reader.scala:292)
[info] at io.bullet.borer.DecoderFromApply.$anonfun$from$1(DecoderFromApply.scala:15)
[info] at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
[info] at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)scala> io.bullet.borer.Json.decode((""""\uD834\uDD1E"""").getBytes).to[String].value
res0: String = 𝄞
scala> io.bullet.borer.Json.decode((""""\uDD1E\uD834"""").getBytes).to[String].value
res1: String = ??
Test
case class Wrapper(xs: Array[Byte])
case class TypedWrapper[T](xs: Array[T])
import io.bullet.borer.derivation.MapBasedCodecs._
implicit def wrapperCodec: Codec[Wrapper] = deriveCodec[Wrapper]
implicit def typedWrapperCodec[T: ClassTag: Encoder: Decoder]: Codec[TypedWrapper[T]] = deriveCodec[TypedWrapper[T]]
def decode(bytes: Array[Byte]): Unit = {
println(
Cbor
.decode(bytes)
.withPrintLogging()
.to[Element]
.value
)
}
println("Wrapper codec********")
decode(Cbor.encode(Wrapper("a".getBytes)).toByteArray)
println("TypedWrapper codec********")
decode(Cbor.encode(TypedWrapper("a".getBytes)).toByteArray)Result
[info] Wrapper codec********
[info] 1: {
[info] 1/1: "xs"
[info] 1/1: -> BYTES[61]
[info] 1: }
[info] 2: END
[info] {StringElem(xs): ByteArrayElem([B@1c1bbc4e)}
[info] TypedWrapper codec********
[info] 1: {
[info] 1/1: "xs"
[info] 1/1: -> [
[info] 1/1: 97
[info] 1/1: ]
[info] 1: }
[info] 2: END
[info] {StringElem(xs): [IntElem(97)]}Expected
Both the arrays should get encoded as BYTES.
Workaround
type ArrayEnc[T] = Encoder[Array[T]]
type ArrayDec[T] = Decoder[Array[T]]
implicit def typedWrapperCodec[T: ClassTag: ArrayEnc: ArrayDec]: Codec[TypedWrapper[T]] = deriveCodec[TypedWrapper[T]]
utest, scodec-bits, and akka-actors are released for Scala 2.13.0 already
scala> io.bullet.borer.Json.decode(("[" + "1," * 10000 + "l" + "]").getBytes).to[Array[Int]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:367)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.readInt(Reader.scala:104)
at io.bullet.borer.Decoder$.$anonfun$forInt$1(Decoder.scala:53)
at io.bullet.borer.Decoder$.$anonfun$forInt$1$adapted(Decoder.scala:53)
at io.bullet.borer.InputReader.read(Reader.scala:311)
at io.bullet.borer.Decoder$.$anonfun$forArray$2(Decoder.scala:181)
at io.bullet.borer.InputReader.rec$3(Reader.scala:340)
at io.bullet.borer.InputReader.readUntilBreak(Reader.scala:341)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:181)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
case class Address[T](x: T)
implicit def addressCodec[T: Encoder: Decoder]: Codec[Address[T]] = Codec.forCaseClass[Address[T]]
The above code snippet fails with the exception:
type mismatch;
[error] found : Address[Nothing] => Option[Nothing]
[error] required: Address[T] => Option[?]
[error] implicit def addressCodec[T: Encoder: Decoder]: Codec[Address[T]] = Codec.forCaseClass[Address[T]]
[error] ^
[error] one error found
To reproduce the stacktrace bellow need to move this line to be bellow this derivation and run this test.
java.lang.NullPointerException (input position 35)
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (input position 35)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:101)
at com.github.plokhotnyuk.jsoniter_scala.benchmark.AnyValsReading.borerJson(AnyValsReading.scala:25)
at com.github.plokhotnyuk.jsoniter_scala.benchmark.AnyValsReadingSpec.$anonfun$new$2(AnyValsReadingSpec.scala:9)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.WordSpecLike$$anon$3.apply(WordSpecLike.scala:1075)
at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
at org.scalatest.WordSpec.withFixture(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.invokeWithFixture$1(WordSpecLike.scala:1073)
at org.scalatest.WordSpecLike.$anonfun$runTest$1(WordSpecLike.scala:1085)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286)
at org.scalatest.WordSpecLike.runTest(WordSpecLike.scala:1085)
at org.scalatest.WordSpecLike.runTest$(WordSpecLike.scala:1067)
at org.scalatest.WordSpec.runTest(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.$anonfun$runTests$1(WordSpecLike.scala:1144)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:393)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:370)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:407)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:381)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:376)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:458)
at org.scalatest.WordSpecLike.runTests(WordSpecLike.scala:1144)
at org.scalatest.WordSpecLike.runTests$(WordSpecLike.scala:1143)
at org.scalatest.WordSpec.runTests(WordSpec.scala:1881)
at org.scalatest.Suite.run(Suite.scala:1124)
at org.scalatest.Suite.run$(Suite.scala:1106)
at org.scalatest.WordSpec.org$scalatest$WordSpecLike$$super$run(WordSpec.scala:1881)
at org.scalatest.WordSpecLike.$anonfun$run$1(WordSpecLike.scala:1189)
at org.scalatest.SuperEngine.runImpl(Engine.scala:518)
at org.scalatest.WordSpecLike.run(WordSpecLike.scala:1189)
at org.scalatest.WordSpecLike.run$(WordSpecLike.scala:1187)
at org.scalatest.WordSpec.run(WordSpec.scala:1881)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1349)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1343)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1343)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1033)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1011)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1509)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1011)
at org.scalatest.tools.Runner$.run(Runner.scala:850)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
Caused by: java.lang.NullPointerException
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.InputReader.apply(Reader.scala:63)
at io.bullet.borer.DecoderFromApply.$anonfun$from$2(DecoderFromApply.scala:18)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 51 more
I do not see deriveUnaryDecoder in ArrayBasedCodecs.
On the other hand, ArrayBasedCodecs.deriveUnaryEncoder is there.
Any technical reason for the asymmetry? Because of this, currently it is not possible to write unary encoder and decoder separately for a type.
We have successfully managed to use single set of user defined (map-based) codecs for a small subset of our domain models and achieve 3 distinct tasks: send them in akka-remote message payloads (CBOR), send some of them to Redis (CBOR) and serve them via http (JSON).
We are very happy with the outcome and looking forward to the possibility of replacing 3 different libraries (chill, scalapb and play-json) with a single library.
One feedback from the team is around nested encoding of ADTs which becomes apparent mainly during the Json output as:
{
"Dog" : {
"age": 10,
"name": "gabbar"
}
}Will it be possible to add a configuration which allows it to be encoded flat as below? Consuming this from say a python library is much easier it seems (cc/ @abrighton)
{
"type": "Dog",
"age": 10,
"name": "gabbar"
}[error] /home/andriy/Projects/com/github/plokhotnyuk/borer/core/src/main/scala/io/bullet/borer/input/FromByteBufferInput.scala:29:30: ambiguous reference to overloaded definition,
[error] both method position in class ByteBuffer of type (x$1: Int)java.nio.ByteBuffer
[error] and method position in class Buffer of type ()Int
[error] match expected type ?
[error] buffer.position(buffer.position - numberOfBytes)
[error] ^
[error] one error found
[error] (core / Compile / compileIncremental) Compilation failed
We have an ADT like this, for which we have written the codec only for the base type and it compiles successfully.
sealed trait A
case class B(a: Int) extends A
case class C(a: Int) extends A
implicit val a: Codec[A] = deriveCodec[A]If we want to serialize/deserialize the values as (compile-time) type A, it works fine.
The docs suggest that we should write similar codecs for B and C too, but I guess that is only if we want to use specific compile-time types for ser/de. Is this correct? Is there any performance difference?
Maybe, something to clarify in the docs.
scala> "1.1999999".toFloat
res0: Float = 1.1999999
scala> io.bullet.borer.Json.decode("1.1999999".getBytes).to[Float].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Float] but got [Double]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readFloat(Reader.scala:139)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1(Decoder.scala:55)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1$adapted(Decoder.scala:55)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[Long]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[BigInt]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("[3326704554664441231]".getBytes).to[Array[BigDecimal]].value
io.bullet.borer.Borer$Error$InvalidJsonData: Invalid JSON syntax
at io.bullet.borer.json.JsonParser.io$bullet$borer$json$JsonParser$$failInvalidJsonSyntax(JsonParser.scala:417)
at io.bullet.borer.json.JsonParser.result$1(JsonParser.scala:98)
at io.bullet.borer.json.JsonParser.parseNumberString$1(JsonParser.scala:105)
at io.bullet.borer.json.JsonParser.breakOutToNumberString$2(JsonParser.scala:146)
at io.bullet.borer.json.JsonParser.parseNumber$1(JsonParser.scala:164)
at io.bullet.borer.json.JsonParser.pull(JsonParser.scala:366)
at io.bullet.borer.InputReader.pull(Reader.scala:346)
at io.bullet.borer.InputReader.pullIfTrue(Reader.scala:354)
at io.bullet.borer.InputReader.tryReadArrayStart(Reader.scala:259)
at io.bullet.borer.Decoder$.$anonfun$forArray$1(Decoder.scala:179)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
Akka-http handles (un)marshalling of a few types like Option[T] and Either[A, B] in a special way.
Akka-http always picks up its own (un)marshallers for them ignoring defaultBorerUnmarshaller and defaultBorerMarshaller provided by borer's akkaHttp.
Inn our project we copy the following in the immediate lexical scope to give them a higher priority.
liftMarshaller from LowPriorityToResponseMarshallerImplicits andmessageUnmarshallerFromEntityUnmarshaller from LowerPriorityGenericUnmarshallersWill it make sense to add them akkaHttp of the borer-compat-akka module?
scala> io.bullet.borer.Json.decode("-0.900845381675988".getBytes).to[Double].value
res0: Double = 0.900845381675988
This is great stuff!
Is there an example of how this can be used directly with Akka?
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> case class A(b: Option[Int] = None, c: Option[String] = None)
defined class A
scala> import io.bullet.borer._
<console>:11: warning: Unused import
import io.bullet.borer._
^
import io.bullet.borer._
scala> import io.bullet.borer.derivation.MapBasedCodecs._
<console>:10: warning: Unused import
import io.bullet.borer._
^
<console>:14: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
import io.bullet.borer.derivation.MapBasedCodecs._
scala> implicit val c = deriveCodec[A]
<console>:12: warning: Unused import
import io.bullet.borer._
^
c: io.bullet.borer.Codec[A] = Codec(io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$$$Lambda$5709/1448885594@76b91f1a,io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$$$Lambda$5732/1761122012@1da255b7)
scala> Json.decode("""{}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res0: A = A(None,None)
scala> Json.decode("""{"b":1}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res1: A = A(Some(1),None)
scala> Json.decode("""{"c":"x"}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res2: A = A(None,Some(x))
scala> Json.decode("""{"b":1,"c":"x"}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
res3: A = A(Some(1),Some(x))
scala> Json.decode("""{"b":1,"c":null}""".getBytes).to[A].value
<console>:15: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
<console>:10: warning: Unused import
import io.bullet.borer._
^
<console>:13: warning: Unused import
import io.bullet.borer.derivation.MapBasedCodecs._
^
<console>:15: warning: Unused import
import c
^
io.bullet.borer.Borer$Error$InvalidInputData: Expected String or Text Bytes but got Null (input position 7)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:442)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:438)
at io.bullet.borer.InputReader.readString(Reader.scala:239)
at io.bullet.borer.Decoder$.$anonfun$forString$1(Decoder.scala:68)
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.Decoder$$anon$1.$anonfun$withDefaultValue$1(Decoder.scala:188)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 36 elided
scala> io.bullet.borer.Json.decode("1".getBytes).to[Double].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Double] but got [Int]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readDouble(Reader.scala:150)
at io.bullet.borer.Decoder$.$anonfun$forDouble$1(Decoder.scala:56)
at io.bullet.borer.Decoder$.$anonfun$forDouble$1$adapted(Decoder.scala:56)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
scala> io.bullet.borer.Json.decode("1".getBytes).to[Float].value
io.bullet.borer.Borer$Error$UnexpectedDataItem: Unexpected data item: Expected [Float] but got [Int]
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:373)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:369)
at io.bullet.borer.InputReader.readFloat(Reader.scala:139)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1(Decoder.scala:55)
at io.bullet.borer.Decoder$.$anonfun$forFloat$1$adapted(Decoder.scala:55)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:177)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:109)
... 36 elided
Currently big numbers in Java has a binary representation under hood, and parsing them from the decimal string representation has O(n^2) complexity.
scala> def timed[A](f: => A): A = { val t = System.currentTimeMillis; val r = f; println(s"Elapsed time (ms): ${System.currentTimeMillis - t}"); r }
timed: [A](f: => A)A
scala> timed(io.bullet.borer.Json.decode(("9" * 1000000).getBytes).to[BigInt].value)
Elapsed time (ms): 13151
scala> timed(io.bullet.borer.Json.decode(("9" * 1000000).getBytes).to[BigDecimal].value)
Elapsed time (ms): 13165
For a contemporary 1Gbit network 10ms of receiving a malicious message with a 1000000-digit number can produce 1300ms of 100% CPU load in a one core. More than 100 cores can be effectively DoS-ed at a full bandwidth rate.
Also, a lot of arithmetic operations has O(n^1.5) complexity, even such "safe" operations as + for a not validated scale of the input value can DoS much more efficiently by a single message with a 12-digit number:
scala> timed(io.bullet.borer.Json.decode(("1e-100000000").getBytes).to[BigDecimal].value)
Elapsed time (ms): 0
scala> timed(io.bullet.borer.Json.decode(("1e-100000000").getBytes).to[BigDecimal].value + 1)
Elapsed time (ms): 91591
So, parse them with taking in account these worst cases, especially for the systems that are opened for the wild wide world:
O(n^1.5) complexity, like here.In previous version it worked fine:
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
def stringCodec[T](f: String => T): Codec[T] =
Codec((w: Writer, value: T) => w.writeString(value.toString), (r: Reader) => f(r.readString()))
implicit val Codec(charEnc: Encoder[Char], charDec: Decoder[Char]) = stringCodec(_.charAt(0))
case class HasChar(ch: Char)
implicit val Codec(hasCharEnc: Encoder[HasChar], hasCharDec: Decoder[HasChar]) = deriveCodec[HasChar]
io.bullet.borer.Json.decode("""{"ch":"F"}""".getBytes).to[HasChar].value
// Exiting paste mode, now interpreting.
...
res0: HasChar = HasChar(F)
Now it tries to parse chars as ints without taking in account a custom codec provided by the implicit and throws the following error:
io.bullet.borer.Borer$Error$InvalidInputData: Expected Char but got Chars (input position 6)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:536)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:532)
at io.bullet.borer.InputReader.readChar(Reader.scala:85)
at HasCharDecoder$1.readObject$1(<console>:1)
at HasCharDecoder$1.read(<console>:1)
at $anon$2.read(<console>:1)
at $anon$2.read(<console>:1)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:99)
... 36 elided
But for some cases like bellow it picks the custom codec:
scala> io.bullet.borer.Json.decode("""["F"]""".getBytes).to[List[Char]].value
...
res1: List[Char] = List(F)
Current code from the master parses floats with rounding where error ~1ULP that is greater than expected ~0.5ULP of java.lang.Float.valueOf().
The rounding error can be easy reproduced when parsing string representation of some values with number of digits greater than usually used for floats.
scala> "1.00000017881393432617187499".toFloat
res0: Float = 1.0000001
scala> "1.00000017881393432617187499".toDouble.toFloat
res1: Float = 1.0000002
The detailed explanation is in this comment
The following code can print lot of such numbers after increasing number of iterations:
scala> :paste
// Entering paste mode (ctrl-D to finish)
(1 to 1000).foreach { _ =>
def checkAndPrint(input: String): Unit = {
val actualOutput = io.bullet.borer.Json.decode(input.getBytes).to[Float].value
val expectedOutput = input.toFloat
if (actualOutput != expectedOutput) {
println(s"input = $input, expectedOutput =$expectedOutput, actualOutput = $actualOutput")
}
}
val n = java.util.concurrent.ThreadLocalRandom.current().nextLong()
val x = java.lang.Double.longBitsToDouble(n & ~0xFFFFFFFL)
if (java.lang.Float.isFinite(x.toFloat) && x.toFloat != 0.0) checkAndPrint(x.toString)
}
// Exiting paste mode, now interpreting.
input = 5.726278970996646E-7, expectedOutput =5.726279E-7, actualOutput = 5.7262787E-7
input = -1.502594932252865E35, expectedOutput =-1.5025949E35, actualOutput = -1.502595E35
input = -2.053233379325646E20, expectedOutput =-2.0532335E20, actualOutput = -2.0532333E20
input = 1.511814678325955E20, expectedOutput =1.5118146E20, actualOutput = 1.5118148E20
input = 4.355016683746335E19, expectedOutput =4.3550165E19, actualOutput = 4.355017E19
input = 1.704752798329881E25, expectedOutput =1.7047527E25, actualOutput = 1.7047529E25
input = -5.250234028812652E31, expectedOutput =-5.250234E31, actualOutput = -5.2502343E31
I think it should be documented if there are no other option available.
GenCodec with CborInput/CborOutput from AVSystems scala-commons.It can be reproduced with v0.10.0:
scala> val jsonBytes = "[1,2,3]".getBytes
jsonBytes: Array[Byte] = Array(91, 49, 44, 50, 44, 51, 93)
scala> io.bullet.borer.Json.decode(jsonBytes).to[Array[Byte]].value
res0: Array[Byte] = Array(1, 2, 3)
scala> io.bullet.borer.Json.encode(res0).toByteArray
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support byte strings (Output.ToByteArray index 0)
at io.bullet.borer.json.JsonRenderer.failUnsupported(JsonRenderer.scala:275)
at io.bullet.borer.json.JsonRenderer.onBytes(JsonRenderer.scala:122)
at io.bullet.borer.Writer.writeBytes(Writer.scala:65)
at io.bullet.borer.Encoder$.$anonfun$forByteArray$1(Encoder.scala:107)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:190)
at io.bullet.borer.EncodingSetup$Impl.result(EncodingSetup.scala:138)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:117)
... 36 elided
Compilation error looks like:
Error:(15, 51) no arguments allowed for nullary constructor JsonTypeInfo: ()com.fasterxml.jackson.annotation.JsonTypeInfo
Note that 'use', 'property' are not parameter names of the invoked method.
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
Error:(15, 51) no arguments allowed for nullary constructor JsonSubTypes: ()com.fasterxml.jackson.annotation.JsonSubTypes
Note that 'value' is not a parameter name of the invoked method.
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
Code to reproduce:
import com.fasterxml.jackson.annotation.JsonSubTypes.Type
import com.fasterxml.jackson.annotation.{JsonSubTypes, JsonTypeInfo}
import scala.collection.immutable.IndexedSeq
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes(Array(
new Type(value = classOf[Point], name = "Point"),
new Type(value = classOf[MultiPoint], name = "MultiPoint"),
new Type(value = classOf[LineString], name = "LineString"),
new Type(value = classOf[MultiLineString], name = "MultiLineString"),
new Type(value = classOf[Polygon], name = "Polygon"),
new Type(value = classOf[MultiPolygon], name = "MultiPolygon"),
new Type(value = classOf[GeometryCollection], name = "GeometryCollection")))
sealed trait Geometry extends Product with Serializable
case class Point(coordinates: (Double, Double)) extends Geometry
case class MultiPoint(coordinates: IndexedSeq[(Double, Double)]) extends Geometry
case class LineString(coordinates: IndexedSeq[(Double, Double)]) extends Geometry
case class MultiLineString(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]]) extends Geometry
case class Polygon(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]]) extends Geometry
case class MultiPolygon(coordinates: IndexedSeq[IndexedSeq[IndexedSeq[(Double, Double)]]]) extends Geometry
case class GeometryCollection(geometries: IndexedSeq[Geometry]) extends Geometry
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes(Array(
new Type(value = classOf[Feature], name = "Feature"),
new Type(value = classOf[FeatureCollection], name = "FeatureCollection")))
sealed trait GeoJSON extends Product with Serializable
case class Feature(properties: Map[String, String], geometry: Geometry) extends GeoJSON
case class FeatureCollection(features: IndexedSeq[GeoJSON]) extends GeoJSON
implicit val c1: Codec[Point] = deriveCodec[Point]
implicit val c2: Codec[MultiPoint] = deriveCodec[MultiPoint]
implicit val c3: Codec[LineString] = deriveCodec[LineString]
implicit val c4: Codec[MultiLineString] = deriveCodec[MultiLineString]
implicit val c5: Codec[Polygon] = deriveCodec[Polygon]
implicit val c6: Codec[MultiPolygon] = deriveCodec[MultiPolygon]
implicit val c7: Codec[GeometryCollection] = deriveCodec[GeometryCollection]
implicit val c8: Codec[Geometry] = deriveCodec[Geometry]
implicit val c9: Codec[Feature] = deriveCodec[Feature]
implicit val c10: Codec[FeatureCollection] = deriveCodec[FeatureCollection]
implicit val c11: Codec[GeoJSON] = deriveCodec[GeoJSON]
Parsing:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val obj = io.bullet.borer.Json.decode("""{"n":{"n":{"n":{}}}}""".getBytes).to[Nested].value
println(obj)
// Exiting paste mode, now interpreting.
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (input position 1)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:101)
... 44 elided
Caused by: java.lang.NullPointerException
at io.bullet.borer.InputReader.read(Reader.scala:352)
at io.bullet.borer.Decoder$$anon$1.$anonfun$withDefaultValue$1(Decoder.scala:188)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.fillArgsAndConstruct$1(MapBasedCodecs.scala:207)
at io.bullet.borer.derivation.MapBasedCodecs$deriveDecoder$.$anonfun$combine$2(MapBasedCodecs.scala:215)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:160)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:98)
... 44 more
Serialization:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val bytes = io.bullet.borer.Json.encode(Nested(Some(Nested(None)))).toByteArray
println(new String(bytes))
// Exiting paste mode, now interpreting.
io.bullet.borer.Borer$Error$General: java.lang.NullPointerException (Output.ToByteArray index 4)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:131)
at io.bullet.borer.EncodingSetup$Impl.toByteArray(EncodingSetup.scala:112)
... 36 elided
Caused by: java.lang.NullPointerException
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.Encoder$$anon$1$$anon$2.write(Encoder.scala:177)
at io.bullet.borer.Encoder$$anon$1$$anon$2.write(Encoder.scala:173)
at io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$.rec$1(MapBasedCodecs.scala:55)
at io.bullet.borer.derivation.MapBasedCodecs$deriveEncoder$.$anonfun$combine$1(MapBasedCodecs.scala:77)
at io.bullet.borer.Writer.write(Writer.scala:86)
at io.bullet.borer.EncodingSetup$Impl.render(EncodingSetup.scala:180)
at io.bullet.borer.EncodingSetup$Impl.bytes(EncodingSetup.scala:128)
... 37 more
The same code when enclosed in curly brackets defers derivation until first call to decoder/encoder and produces SOE:
> console
[info] Starting scala interpreter...
Welcome to Scala 2.12.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> :paste
// Entering paste mode (ctrl-D to finish)
{
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
case class Nested(n: Option[Nested] = None)
object Nested {
implicit val Codec(encoder: Encoder[Nested], decoder: Decoder[Nested]) = deriveCodec[Nested]
}
val obj = io.bullet.borer.Json.decode("""{"n":{"n":{"n":{}}}}""".getBytes).to[Nested].value
println(obj)
}
// Exiting paste mode, now interpreting.
java.lang.StackOverflowError
at Nested$2$.nestedTypeclass$macro$1$lzycompute$1(<pastie>:19)
at Nested$2$.nestedTypeclass$macro$1$1(<pastie>:19)
... 1022 elided
https://sirthias.github.io/borer/ redirects to http://decodified.com/borer/ and returns 404
Often we want slightly different codecs for Json vs Cbor. We have defined the helpers like these:
def targetSpecificEnc[T](cborEnc: Encoder[T], jsonEnc: Encoder[T]): Encoder[T] = { (writer, value) ⇒
val enc = if (writer.target == Cbor) cborEnc else jsonEnc
enc.write(writer, value)
}
def targetSpecificDec[T](cborDec: Decoder[T], jsonDec: Decoder[T]): Decoder[T] = { reader ⇒
val dec = if (reader.target == Cbor) cborDec else jsonDec
dec.read(reader)
}Which we use to define Instant codecs like these:
implicit lazy val instantEnc: Encoder[Instant] = CborHelpers.targetSpecificEnc(
cborEnc = Encoder.forTuple2[Long, Long].contramap(x => (x.getEpochSecond, x.getNano)),
jsonEnc = Encoder.forString.contramap(_.toString)
)
implicit lazy val instantDec: Decoder[Instant] = CborHelpers.targetSpecificDec(
cborDec = Decoder.forTuple2[Long, Long].map(t => Instant.ofEpochSecond(t._1, t._2)),
jsonDec = Decoder.forString.map(Instant.parse)
)I hope this is the right way to go about doing this. If so, it will be good to have these helpers in the library.
Test
Below code compiles, but throws RuntimeException during the run
package demo
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
sealed trait A
sealed trait B extends A
sealed trait C extends A
case class D(a: Int) extends B with C
object DiamondApp {
def main(args: Array[String]): Unit = {
implicit val a: Codec[A] = deriveCodec[A]
}
}Exception during run
java.lang.RuntimeException: @key collision: At least two subtypes of `demo.A` share the same type id `D`
Any possible workaround?
Please add a configuration parameter for the Target methods.
Implementation of it can use own default values, like here:
def encode[T: Encoder](value: T, config: EncodingConfig = EncodingConfig.default): EncodingSetup.JsonApi[T, EncodingConfig] =
new EncodingSetup.Impl(value, Json, config, Receiver.nopWrapper, JsonRenderer)
def decode[T](input: T, config: DecodingConfig = DecodingConfig.default)(implicit w: Input.Wrapper[T]): DecodingSetup.Api[w.In, DecodingConfig] =
Following does not work
sealed trait Adt
case class Err(reason: String) extends Adt
case object Ok extends Adt
implicit lazy val errCodec: Codec[Err] = ArrayBasedCodecs.deriveCodecForUnaryCaseClass[Err]
implicit lazy val okCodec: Codec[Ok.type] = deriveCodec[Ok.type]
implicit lazy val adtCodec: Codec[Adt] = deriveCodec[Adt]Compilation Error
Error:(16, 58) exception during macro expansion:
scala.ScalaReflectionException: object Ok is not a module
Workaround
def singletonCodec[T <: Singleton](x: T): Codec[T] = Codec.implicitly[String].bimap[T](_.toString, _ => x)
implicit lazy val okCodec: Codec[Ok.type] = singletonCodec(Ok)scala> io.bullet.borer.Json.decode(("1\u0030").getBytes).to[Int].value
res0: Int = 10
scala> io.bullet.borer.Json.decode(("[true\u002Cfalse]").getBytes).to[Array[Boolean]].value
res1: Array[Boolean] = Array(true, false)
[info] Updating ProjectRef(uri("file:/home/andriy/Projects/com/github/plokhotnyuk/borer/project/"), "borer-build")...
[warn] module not found: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local: tried
[warn] /home/andriy/.ivy2/local/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/io/bullet/borer-core_2.12/0.9.1-SNAPSHOT/borer-core_2.12-0.9.1-SNAPSHOT.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/andriy/.sbt/preloaded/io.bullet/borer-core_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/andriy/.sbt/preloaded/io/bullet/borer-core_2.12/0.9.1-SNAPSHOT/borer-core_2.12-0.9.1-SNAPSHOT.pom
[warn] module not found: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local: tried
[warn] /home/andriy/.ivy2/local/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/io/bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/borer-derivation_2.12-0.9.1-SNAPSHOT.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /home/andriy/.sbt/preloaded/io.bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////home/andriy/.sbt/preloaded/io/bullet/borer-derivation_2.12/0.9.1-SNAPSHOT/borer-derivation_2.12-0.9.1-SNAPSHOT.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[warn] :: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] io.bullet:borer-core_2.12:0.9.1-SNAPSHOT (/home/andriy/Projects/com/github/plokhotnyuk/borer/project/plugins.sbt#L12-16)
[warn] +- default:borer-build:0.1.0-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[warn] io.bullet:borer-derivation_2.12:0.9.1-SNAPSHOT (/home/andriy/Projects/com/github/plokhotnyuk/borer/project/plugins.sbt#L12-16)
[warn] +- default:borer-build:0.1.0-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[error] sbt.librarymanagement.ResolveException: unresolved dependency: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[error] unresolved dependency: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
[error] at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:332)
[error] at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:208)
[error] at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:239)
[error] at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:204)
[error] at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[error] at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[error] at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
[error] at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
[error] at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
[error] at xsbt.boot.Using$.withResource(Using.scala:10)
[error] at xsbt.boot.Using$.apply(Using.scala:9)
[error] at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
[error] at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
[error] at xsbt.boot.Locks$.apply0(Locks.scala:31)
[error] at xsbt.boot.Locks$.apply(Locks.scala:28)
[error] at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:199)
[error] at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:196)
[error] at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:238)
[error] at sbt.internal.librarymanagement.IvyActions$.updateEither(IvyActions.scala:193)
[error] at sbt.librarymanagement.ivy.IvyDependencyResolution.update(IvyDependencyResolution.scala:20)
[error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:56)
[error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:45)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:93)
[error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:68)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$19(LibraryManagement.scala:106)
[error] at scala.util.control.Exception$Catch.apply(Exception.scala:224)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:106)
[error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:89)
[error] at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:149)
[error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:120)
[error] at sbt.Classpaths$.$anonfun$updateTask$5(Defaults.scala:2561)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error] at sbt.std.Transform$$anon$4.work(System.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.Execute.work(Execute.scala:278)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (update) sbt.librarymanagement.ResolveException: unresolved dependency: io.bullet#borer-core_2.12;0.9.1-SNAPSHOT: not found
[error] unresolved dependency: io.bullet#borer-derivation_2.12;0.9.1-SNAPSHOT: not found
Currently we do import io.bullet.borer.derivation.MapBasedCodecs._ and then use deriveCodec[T] for all case classes except the unary case classes for which we use Codec.forCaseClass[T].
This works, but is prone to error. We may end up using Codec.forCaseClass[T] for non-unary case classes, especially during the refactoring, resulting in mix of array-based and map-based encodings.
It will be good to have a dedicated function for the unary case classes which fails to compile if the case class is not unary.
Test
def getEncoder[T](implicit x: Encoder[Array[T]]): Encoder[Array[T]] = x
val enc: Encoder[Array[java.lang.Byte]] = Encoder.forByteArray.contramap(_.map(x ⇒ x: Byte))
val dec: Decoder[Array[java.lang.Byte]] = Decoder.forByteArray.map(_.map(x ⇒ x: java.lang.Byte))
implicit val codec: Codec[Array[java.lang.Byte]] = Codec(enc, dec)
assert(getEncoder[java.lang.Byte] eq enc)Code to reproduce:
scala> :paste
// Entering paste mode (ctrl-D to finish)
import io.bullet.borer._
import io.bullet.borer.derivation.MapBasedCodecs._
sealed trait Geometry extends Product with Serializable
case class Point(coordinates: (Double, Double)) extends Geometry
case class MultiPoint(coordinates: IndexedSeq[(Double, Double)] = Vector.empty) extends Geometry
case class LineString(coordinates: IndexedSeq[(Double, Double)] = Vector.empty) extends Geometry
case class MultiLineString(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]] = Vector.empty) extends Geometry
case class Polygon(coordinates: IndexedSeq[IndexedSeq[(Double, Double)]] = Vector.empty) extends Geometry
case class MultiPolygon(coordinates: IndexedSeq[IndexedSeq[IndexedSeq[(Double, Double)]]] = Vector.empty) extends Geometry
case class GeometryCollection(geometries: IndexedSeq[Geometry] = Vector.empty) extends Geometry
sealed trait GeoJSON extends Product with Serializable
case class Feature(properties: Map[String, String] = Map.empty, geometry: Geometry, bbox: Option[(Double, Double, Double, Double)] = None) extends GeoJSON
case class FeatureCollection(features: IndexedSeq[GeoJSON] = Vector.empty, bbox: Option[(Double, Double, Double, Double)] = None) extends GeoJSON
implicit val flatAdtEncoding: AdtEncodingStrategy = AdtEncodingStrategy.flat(typeMemberName = "type")
implicit val Codec(geoJsonEnc: Encoder[GeoJSON], geoJsonDec: Decoder[GeoJSON]) = {
implicit lazy val c1: Codec[Geometry] = {
implicit val c11: Codec[Point] = deriveCodec
implicit val c12: Codec[MultiPoint] = deriveCodec
implicit val c13: Codec[LineString] = deriveCodec
implicit val c14: Codec[MultiLineString] = deriveCodec
implicit val c15: Codec[Polygon] = deriveCodec
implicit val c16: Codec[MultiPolygon] = deriveCodec
implicit val c17: Codec[GeometryCollection] = deriveCodec
deriveCodec[Geometry]
}
implicit val c2: Codec[Feature] = deriveCodec
implicit val c3: Codec[FeatureCollection] = deriveCodec
deriveCodec[GeoJSON]
}
// Exiting paste mode, now interpreting.
...
scala> new String(io.bullet.borer.Json.encode[GeoJSON](FeatureCollection(Vector(Feature(geometry = Polygon(Vector(Vector((7.697223,47.543327)))))))).toByteArray)
...
res0: String = {"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[7.697223,47.543327]]]}}]}
scala> io.bullet.borer.Json.decode(res0.getBytes).to[GeoJSON].value
...
io.bullet.borer.Borer$Error$InvalidInputData: Expected Map for decoding an instance of type `GeoJSON` but got Start of unbounded Map (input position 41)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:617)
at io.bullet.borer.InputReader.unexpectedDataItem(Reader.scala:613)
at io.bullet.borer.AdtEncodingStrategy$$anon$1.failNoMap$1(AdtEncodingStrategy.scala:175)
at io.bullet.borer.AdtEncodingStrategy$$anon$1.readAdtEnvelopeOpen(AdtEncodingStrategy.scala:234)
at .$anonfun$x$1$4(<console>:32)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at io.bullet.borer.InputReader.rec$1(Reader.scala:520)
at io.bullet.borer.InputReader.readUntilBreak(Reader.scala:521)
at io.bullet.borer.Decoder$.$anonfun$forIterable$1(Decoder.scala:255)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at FeatureCollectionDecoder$1.readObject$9(<console>:31)
at FeatureCollectionDecoder$1.read(<console>:31)
at $anon$18.read(<console>:31)
at $anon$18.read(<console>:31)
at io.bullet.borer.InputReader.read(Reader.scala:516)
at .$anonfun$x$1$4(<console>:32)
at io.bullet.borer.DecodingSetup$Impl.decodeFrom(DecodingSetup.scala:169)
at io.bullet.borer.DecodingSetup$Impl.value(DecodingSetup.scala:100)
... 36 elided
I tried this:
implicit val intArrEnc: Encoder[Array[Int]] = Encoder.forArray[Integer].compose[Array[Int]](_.map(x => x: Integer))
implicit val intArrDec: Decoder[Array[Int]] = Decoder.forArray[Integer].map[Array[Int]](_.map(x => x: Int))
val intArr = Array(1, 2, 3, 4)
val bytes: Array[Byte] = Cbor.encode(intArr).toByteArray
val blob: Array[Int] = Cbor.decode(bytes).to[Array[Int]].valueI get this error:
Exception in thread "main" io.bullet.borer.Borer$Error$General: java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.lang.Integer; ([Ljava.lang.Object; and [Ljava.lang.Integer; are in module java.base of loader 'bootstrap')
at io.bullet.borer.Borer$DecodingSetup.to(Borer.scala:265)
at spikes.Demo$.main(Blob.scala:60)
at spikes.Demo.main(Blob.scala)
Caused by: java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [Ljava.lang.Integer; ([Ljava.lang.Object; and [Ljava.lang.Integer; are in module java.base of loader 'bootstrap')
at io.bullet.borer.Decoder$DecoderOps$.$anonfun$map$1(Decoder.scala:40)
at io.bullet.borer.Decoder$.$anonfun$apply$1(Decoder.scala:32)
at io.bullet.borer.Borer$DecodingSetup.to(Borer.scala:260)
... 2 more
Test
val bytes = "abc".getBytes()
Json.encode(bytes).toUtf8StringError
io.bullet.borer.Borer$Error$Unsupported: The JSON renderer doesn't support byte strings (Output.ToByteArray index 0)Related
All of the following works
Json.encode(bytes.head).toUtf8String
Cbor.encode(bytes.head).toByteArray
Cbor.encode(bytes).toByteArrayWorkaround
implicit lazy val bytesEnc: Encoder[Array[Byte]] = targetSpecificEnc(
cborEnc = Encoder.forByteArray,
jsonEnc = Encoder.forArray[Byte]
)
def targetSpecificEnc[T](cborEnc: Encoder[T], jsonEnc: Encoder[T]): Encoder[T] = { (writer, value) ⇒
val enc = if (writer.target == Cbor) cborEnc else jsonEnc
enc.write(writer, value)
}We had to write an analogous target specific decoder for Array[Bytes]
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.