virtuslab/scala-yaml

Scala Yaml

Scala Yaml is a dependency-free Scala library that allows you to work with YAML.

The goal of this project is to create an idiomatic library which:

• provides compile time type safety
• semiautomatically derives codec instances
• has explicit error handling and helpful error messages
• cross compiles to Scala.js and Scala Native (waiting for Scala 3 support in SN)

Please see the guide for more information

Help us create our roadmap!

Take part in our discussions, post your ideas, vote for feature requests, and have a real impact on how our next milestone will look like!

Usage

Add as a dependency:

• via sbt

libraryDependencies += "org.virtuslab" %% "scala-yaml" % "<version>"

• via scala-cli

//> using dep org.virtuslab::scala-yaml:<version>

and use it in your code!

import org.virtuslab.yaml.*

case class Address(city: String, zipcode: String) derives YamlCodec
case class Person(name: String, age: Int, address: Address) derives YamlCodec

val yaml = s"""name: John Wick
              |age: 40
              |address:
              |  city: Anywhere
              |  zipcode: 12-345
              |""".stripMargin

val decoded = yaml.as[Person]
// Either[YamlError, Person] = Right(Person(John Wick,40,Address(Anywhere,12-345)))

case class Activity(kind: String, distance: Seq[Double]) derives YamlCodec

val activity = Activity("running", Seq(5.37, 4.98, 5.73))
val encoded  = activity.asYaml
//kind: running
//distance: 
//  - 5.37
//  - 4.98
//  - 5.73

scala-yaml's Issues

Support nested types in YamlEncoder

For example, encoding Person would produce malformed output.

  case class Address(city: String, street: String) derives YamlEncoder
  case class Person(name: String, address: Address) derives YamlEncoder

jar for 0.0.5 version for scala 2.13 is empty

please see jar file published for 2.13:0.0.5 is empty (weights 300B - only META-INF dir)

https://repo1.maven.org/maven2/org/virtuslab/scala-yaml_2.13/0.0.5/scala-yaml_2.13-0.0.5.jar

Provide a way to test scala-yaml with the official YAML test suite

YAML test suite

Integration test fails even in clean checkout

sbt:scala-yaml> integration/test
org.virtuslab.yaml.K8sConfigSpec:
==> X org.virtuslab.yaml.K8sConfigSpec.initializationError  0.001s java.lang.NullPointerException: null
    at org.virtuslab.yaml.YamlRunnerSpec.<init>(YamlRunnerSpec.scala:14)
    at org.virtuslab.yaml.K8sConfigSpec.<init>(K8sConfigSpec.scala:3)
Error in test - /Users/tisue/scala-yaml/integration-tests/target/scala-2.13/test-classes/yaml/test-suite/9DXL.tml
Events differ - /Users/tisue/scala-yaml/integration-tests/target/scala-2.13/test-classes/yaml/test-suite/RXY3.tml
Error in test - /Users/tisue/scala-yaml/integration-tests/target/scala-2.13/test-classes/yaml/test-suite/M5DY.tml
Error in test - /Users/tisue/scala-yaml/integration-tests/target/scala-2.13/test-classes/yaml/test-suite/SF5V.tml
Error in test - /Users/tisue/scala-yaml/integration-tests/target/scala-2.13/test-classes/yaml/test-suite/Q9WF.tml
...

this came up in the context of scala/community-build#1713

Parsing key with empty value

in-yaml:

---
- { "single line", a: b}
- { "multi
  line", a: b}

Should be parsed to key "single line" with null value

Tokenize tags and mapping keys with proper order

Similar to the #87.

See related testcase:

scala-yaml/yaml/shared/src/test/scala/org/virtuslab/yaml/tokenizer/TagSuite.scala

Lines 75 to 94 in 5b97381

	// todo Tag token should be placed after MappingKey
	test("verbatim tag".ignore) {
	val yaml = """|---
	|!<tag:yaml.org,2002:str> foo :
	| !<!bar> baz""".stripMargin
	val tokens = List(
	DocumentStart,
	MappingStart,
	MappingKey,
	Tag(TagValue.Verbatim("!<tag:yaml.org,2002:str>")),
	Scalar("foo"),
	MappingValue,
	Tag(TagValue.Verbatim("!<!bar>")),
	Scalar("baz"),
	BlockEnd
	)
	assertTokenEquals(yaml.tokens, tokens)
	}

All non-printable characters must be escaped

I am comparing scala-yaml parser with Circe JSON

Char.MinValue.toString.as[String].foreach(println) // prints nothing
> ""
println(Char.MinValue.toString.asJson) // circe lib code is called, ".asJson"
> \u0000

According to YAML specification, all non-printable characters must be escaped. Is it a bug in scala-yaml?

Don't convert `!!null` implicitly into empty strings

// using scala 3.0.2
// using lib org.virtuslab::scala-yaml:0.0.4

import org.virtuslab.yaml.*

implicit class EitherThrowOps[E <: YamlError, T](private val either: Either[E, T]) {
  def orThrow: T =
    either match {
      case Left(e)  => throw new RuntimeException(e.msg)
      case Right(t) => t
    }
}

case class Foo(key1: String, key2: String) derives YamlDecoder
case class Bar(key1: Int, key2: Int) derives YamlDecoder

val foo =
  """|key1: ""
     |key2: !!null
     |""".stripMargin.as[Foo].orThrow

println(foo.key2.size) // << works

val bar =
  """|key1: 1
     |key2: !!null
     |""".stripMargin.as[Bar].orThrow // << throws because "" cannot be parsed into Int

println(bar.key2)

String types should not be silently converted to `Integer` or `Boolean`

If a Yaml value is a String but the actual value appears to be parsable as integer or boolean, scala-yaml automatically translates it into java.lang.Integer and java.lang.Boolean. This changes the yaml schema silently and unexpectedly. With typical kubernetes Yaml where values such as "2" or "true" are of type String , the Yaml cannot be parsed after scala-yaml serializes the data.

Example code:

    val yaml1 = """  revision: "2"  """.as[Any].toOption.get
   // We expect `revision` to be of type `String` and value `"2"`, instead have an integer:
    assert(yaml1.asInstanceOf[Map[String, Any]].apply("revision").isInstanceOf[java.lang.Integer])

    val yaml2 = """  enabled: "true"  """.as[Any].toOption.get
   // We expect `enabled` to be of type `String` and value `"true"`, instead have a Boolean:
    assert(yaml2.asInstanceOf[Map[String, Any]].apply("enabled").isInstanceOf[java.lang.Boolean])

I expected it to be a String in each case, since "2" and "true" are just strings.

Not sure if this is a feature that can be turned off, please advise.

Add API for extracting data from nodes

Discussed in #56

^{Originally posted by pikinier20 August 19, 2021}

• Add an API to retrieve a value of a field with desired name

Examples:

case class Example(a: Int, b: String)
val yamlNode: Node = Node.MappingNode(
Node.KeyValueNode(Node.ScalarNode("a"), Node.ScalarNode(2)),
Node.KeyValueNode(Node.ScalarNode("b"), Node.ScalarNode("asd")
)
for {
a <- yamlNode.get("a").as[Int]
b <- yamlNode.get("b").as[String]
} yield Example(a, b)

Allow incremental processing of documents

According to the spec YAML was designed to support incremental interfaces that include both input (“getNextEvent()”) and output (“sendNextEvent()”) one-pass interfaces.

However, the current implementation allows only batched processing of whole documents. This change would be also very handy when working with the official test suite because it has negative tests, for instance this one. Those test cases contain a list of expected events before an error occurrence.

Support explicit document end (three dots)

https://yaml.org/spec/1.2/spec.html#id2760395

Cannot construct `Option` instance

// using scala 3.1.0
// using lib org.virtuslab::scala-yaml:0.0.4

import org.virtuslab.yaml.*

case class Foo(int: Int, string: String) derives YamlCodec

val foo =
"""|- int: 1
   |  string: "1"
   |""".stripMargin.as[List[Option[Foo]]]

println(foo)

yields

Left(ConstructError(Could't construct scala.Option from CoreSchemaTag(tag:yaml.org,2002:map)
  string: "1"
  ^
))

Parsing final break style in folded scalar

in-yaml

        -----BEGIN CERTIFICATE-----
        0MTk0MVoXDenkKThvP7IS9q
        +Dzv5hG392KWh5f8xJNs4LbZyl901MeReiLrPH3w=
        -----END CERTIFICATE----


kind: v1

Expected events:

+STR
+DOC
+MAP
=VAL :certificate
=VAL >-----BEGIN CERTIFICATE-----\n\n\n\n0MTk0MVoXDenkKThvP7IS9q +Dzv5hG392KWh5f8xJNs4LbZyl901MeReiLrPH3w= -----END CERTIFICATE----\n\n\n
=VAL :kind
=VAL :v1
-MAP
-DOC
-STR

Implement YamlEncoder & YamlDecoder instances for primitive types

StackOverflow when parsing folded value

in-yaml

>
 folded
 text

Expected events:

+STR
+DOC
=VAL >folded text\n
-DOC
-STR

Comprehensive testing using yaml-test-suite

Hi, quick question. Would you consider adding tests based on https://github.com/yaml/yaml-test-suite to make parser and printers more robust?

Add error handling to the ParserImpl

Parser accepts invalid `foo: - bar`

//> using scala "3.1.3"
//> using lib "org.virtuslab::scala-yaml::0.0.4"
//> using lib "org.snakeyaml:snakeyaml-engine:2.3"


val myYaml = """foo: - bar"""

def virtusYaml() =
  import org.virtuslab.yaml._
  println(myYaml.asNode)

def snakeYaml() =
  import org.snakeyaml.engine.v2.api._
  val load = new Load(LoadSettings.builder().build())
  println(load.loadFromString(myYaml))

@main def main =
  virtusYaml()
  println()
  snakeYaml()

Right(MappingNode(Map(ScalarNode(foo,CoreSchemaTag(tag:yaml.org,2002:str),Some(Range(Position(0,0,0),Vector(foo: - bar),Some(Position(3,0,3))))) -> SequenceNode(List(ScalarNode(bar,CoreSchemaTag(tag:yaml.org,2002:str),Some(Range(Position(7,0,7),Vector(foo: - bar),Some(Position(10,0,10)))))),CoreSchemaTag(tag:yaml.org,2002:seq),Some(Range(Position(7,0,7),Vector(foo: - bar),Some(Position(10,0,10)))))),CoreSchemaTag(tag:yaml.org,2002:map),Some(Range(Position(0,0,0),Vector(foo: - bar),Some(Position(3,0,3))))))

Exception in thread "main" 
sequence entries are not allowed here
 in reader, line 1, column 6:
    foo: - bar
         ^

        at org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchBlockEntry(ScannerImpl.java:737)

scala 2.13 is not relased

It seems that there is a version for 2.12 and 3 but not 2.13

Add API/dsl which will provide a convenient way to construct nodes

Discussed in #56

^{Originally posted by pikinier20 August 19, 2021}

• Add a DSL to construct Nodes

Examples:

yamlObj(
  "field1" -> true,
  "field2" -> 2,
  "field3" -> yamlObj(
    "field4" - "asd"
  )
)

StringReader should maintain current position when reading

Fix parsing new line for literal style

in-yaml

certificate: |-
        -----BEGIN CERTIFICATE-----
        0MTk0MVoXDenkKThvP7IS9q
        +Dzv5hG392KWh5f8xJNs4LbZyl901MeReiLrPH3w=
        -----END CERTIFICATE----

Expected events:

+STR
+DOC
+MAP
=VAL :certificate
=VAL |-----BEGIN CERTIFICATE-----\n0MTk0MVoXDenkKThvP7IS9q\n+Dzv5hG392KWh5f8xJNs4LbZyl901MeReiLrPH3w=\n-----END CERTIFICATE----
-MAP
-DOC
-STR

Publish for 2.12/2.13?

Thrilled to discover this project yesterday! It scratches a real itch :)

I understand the codec derivation features rely on Scala 3, but I assume it should be straightforward to cross-compile the parser and Node AST for Scala 2.12/2.13? The parser and Node AST is sufficient to enable a circe-yaml frontend for example.

Happy to do the work to split into version-specific sources if 👍

Discussion on the importance of 2.12 (and 2.13) support is in:

• circe/circe#1818

I noticed that the advice in #102 (comment) is to use CrossVersion.for2_13Use3 but the consensus seems to be to avoid this technique for libraries. See also:

• https://gist.github.com/djspiewak/b8f2b4547951442102488964bc351cf9#5-dont-rely-on-withdottycompat
• scala-js/scala-js-dom#451

Thanks!

Support complex keys in mappings

Currently, we assume that the mapping node consists of (scalar node, node) pairs, but YAML spec allows complex keys to be used (AFAIK it could be arbitrary YAML node) in mappings.

Hexadecimal values or `.inf` fails to get parsed

Yaml values such as 0xFFFF or -.inf are recognized as Integer and Double but parsing fails:

scala> println("0xFFFF".as[Int])
Left(ConstructError(Cannot parse 0xFFFF as Int
at 0:0, expected Int
0xFFFF
^ ))

scala> println("-.inf".as[Double])
Left(ConstructError(Cannot parse -.inf as Double
at 0:0, expected Double
-.inf
^ ))

Add module name to the documentation

Currently the name of the module is neither in the Readme nor the microsite. The only way to know that the module name is scala-yaml is by looking into the build.sbt or searching for the artefact in the Maven Repository.

It would be helpful if it was documented somewhere.

Integrate with official yaml test suite

Scala yaml should have functionality that will allow you to see what percentage of tests pass. This will prevent regression.

Tests should be run by a task in sbt.

Tests - https://github.com/yaml/yaml-test-suite

Parsing anchors with keys

For

{
  &a a : &b b,
  seq: [*a, *b]
}

tokenizer should return ..., MappingKey, Anchor("a"), Scalar(a), ..., but currently it returns ..., Anchor("a"), MappingKey, Scalar(a), .... Returning this in the proposed order will allow us to support such constructions without any changes in the parser.

Parsing multiline string with escaped new line

If I have attribute

description: |-
  Hi
  my name
  is John

I expect the value will be

Hi
my name
is John

and not

Hi\nmy name\nis John

The new lines should not be escaped "\\n".

Tested with version 0.0.8

Use default arguments when field is not provided

Upickle and some other json libraries uses default arguments when field is not provided. It'll highly improve readability of yaml when you don't need to provide optional fields, and will give Devs much more flexibility.

//> using scala "3.2.2"
//> using dep "org.virtuslab::scala-yaml:0.0.6"

import org.virtuslab.yaml.*

case class Foo(id:String, name:String = "optional") derives YamlCodec

val example1 = 
	"id: 0112\n" + 
	"name: Tottally not fake name\n"
	
val example2 = 
	"id: 0113\n"

println(">" + example1.as[Foo]) //> Right(Foo(0112,Tottally not fake name))
println(">" + example2.as[Foo]) //> Left(ConstructError(Key name doesn't exist in parsed document)) 
// expected                     //> Right(Foo(0113,optional))

Backslashes in single-quoted YAML strings should not be escaped

Single-quoted YAML strings should not escape \ characters.
For example,

$ scala-cli repl --dep org.virtuslab::scala-yaml:0.0.8
Welcome to Scala 3.3.1 (17.0.9, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                                                      
scala> import org.virtuslab.yaml.*
                                                                                                                                                                                                                                      
scala> raw"'hello\there'".as[String] match
     |  case Right(s) => println(s)
     |  case Left(e) => println(s"error ${e.msg}")
     | 
hello\\there
                                                                                                                                                                                                                                      
scala> 

should produce hello\there, not hello\\there. See https://yaml.org/spec/1.2.2/#single-quoted-style:

The single-quoted style is specified by surrounding “'” indicators. Therefore, within a single-quoted scalar, such characters need to be repeated. This is the only form of escaping performed in single-quoted scalars. In particular, the “\” and “"” characters may be freely used.

Add `YamlEncoder` for `Option`

// using scala 3.1.0
// using lib org.virtuslab::scala-yaml:0.0.4

import org.virtuslab.yaml.*

case class Foo(field: Option[String]) derives YamlCodec

running via scala-cli run . yields compilation error

Compiling project (Scala 3.1.0, JVM)
[error] ./example2.sc:6:47: no implicit argument of type org.virtuslab.yaml.YamlEncoder[Option[String]] was found
[error] case class Foo(field: Option[String]) derives YamlCodec
[error]                                               ^
Error compiling project (Scala 3.1.0, JVM)

Publish scala-yaml for scala-js

Support for more codecs (`Duration`, `java.time.LocalDate`, etc ...)

Hi

Would be cool to have codecs for objects such as scala.concurrent.Duration, java.time.LocalDate, java.time.Instant.
For the Duration, I would get inspired by pureconfig: https://github.com/pureconfig/pureconfig/blob/7d15967d12b3bee0fc7bbd13325ebb787a8a3a31/core/src/main/scala/pureconfig/DurationUtils.scala

Thanks

Maven Jar for 2.13 (0.0.5) seems to be empty

Integrated from https://mvnrepository.com/artifact/org.virtuslab/scala-yaml_2.13/0.0.5 , but Jar is empty (only Manifest).

Don't require `!!null` for `Option[T]`

I see there is some work to add a YamlEncoder for Option in https://github.com/VirtusLab/scala-yaml/pull/127/files but even in that situation when using an Option[T] you still have to use !!null inside your yml file which is clunky.

For example if you have something like this:

//> using scala "3.2.1"
//> using lib "org.virtuslab::scala-yaml:0.0.6"

import scala.io.Source
import Encoders.given_YamlDecoder_Option
import Encoders.given_YamlEncoder_Option
import org.virtuslab.yaml.YamlCodec
import org.virtuslab.yaml.YamlEncoder
import org.virtuslab.yaml.Node
import org.virtuslab.yaml.YamlDecoder
import org.virtuslab.yaml.Node.ScalarNode
import org.virtuslab.yaml.*

@main def run() =

  val names = Source.fromFile("./example.yml").mkString
  val parsed = names.as[Seq[Name]]
  parsed match
    case Left(e) =>
      println(e.msg)
    case Right(value) =>
      print(s"got ${value} names")

final case class Name(
    name: String,
    other: Option[String]
) derives YamlCodec

object Encoders:
  given [T](using encoder: YamlEncoder[T]): YamlEncoder[Option[T]] = {
    case Some(t) => encoder.asNode(t)
    case None    => Node.ScalarNode("!!null")
  }

  // I know this is weird, but just trying to see if there is any way to get this to be correct if the key doesn't exist.
  given [T](using decoder: YamlDecoder[T]): YamlDecoder[Option[T]] =
    YamlDecoder { node => Right(None) }

And you have a file like this:

- name: My name
  other: My other name

- name: Their name

It will fail with:

❯ scli run .
Key other doesn't exist in parsed document

Unless I'm doing something wrong, I wish the missing key could just be decoded into a None. Is there a way to accomplish this with a custom decoder?

Add cross building

Manual about cross-building.

Scala3doc is published for the test module?

Hi! I found this repo and wanted to see the documentation. I was surprised to see this:

None of this looks like library types:

I suppose there might be some publishing mix-up.

Update: interestingly enough, there are some other pages being generated (example: https://virtuslab.github.io/scala-yaml/org/virtuslab/yaml/internal.html), but they're not visible via navigation.

Handle anchors & aliases when composing nodes from events

See todo in ComposerImpl