zio/zio-schema

Compositional, type-safe schema definitions, which enable auto-derivation of codecs and migrations.

ZIO Schema

ZIO Schema is a ZIO-based library for modeling the schema of data structures as first-class values.

Introduction

ZIO Schema helps us to solve some of the most common problems in distributed computing, such as serialization, deserialization, and data migration.

It turns a compiled-time construct (the type of a data structure) into a runtime construct (a value that can be read, manipulated, and composed at runtime). A schema is a structure of a data type. ZIO Schema reifies the concept of structure for data types. It makes a high-level description of any data type and makes them first-class values.

Creating a schema for a data type helps us to write codecs for that data type. So this library can be a host of functionalities useful for writing codecs and protocols like JSON, Protobuf, CSV, and so forth.

What Problems Does ZIO Schema Solve?

With schema descriptions that can be automatically derived for case classes and sealed traits, ZIO Schema will be going to provide powerful features for free:

1. Metaprogramming without macros, reflection, or complicated implicit derivations.
   1. Creating serialization and deserialization codecs for any supported protocol (JSON, Protobuf, etc.)
   2. Deriving standard type classes (Eq, Show, Ordering, etc.) from the structure of the data
   3. Default values for data types
2. Automate ETL (Extract, Transform, Load) pipelines
   1. Diffing: diffing between two values of the same type
   2. Patching: applying a diff to a value to update it
   3. Migration: migrating values from one type to another
3. Computations as data: Not only we can turn types into values, but we can also turn computations into values. This opens up a whole new world of possibilities concerning distributed computing.

When our data structures need to be serialized, deserialized, persisted, or transported across the wire, then ZIO Schema lets us focus on data modeling and automatically tackle all the low-level, messy details for us.

ZIO Schema is used by a growing number of ZIO libraries, including ZIO Flow, ZIO Redis, ZIO SQL and ZIO DynamoDB.

Installation

In order to use this library, we need to add the following lines in our build.sbt file:

libraryDependencies += "dev.zio" %% "zio-schema"          % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-avro"     % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-bson"     % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-json"     % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-msg-pack" % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-protobuf" % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-thrift"   % "1.0.1"
libraryDependencies += "dev.zio" %% "zio-schema-zio-test" % "1.0.1"

// Required for the automatic generic derivation of schemas
libraryDependencies += "dev.zio" %% "zio-schema-derivation" % "1.0.1"
libraryDependencies += "org.scala-lang" % "scala-reflect"  % scalaVersion.value % "provided"

Example

In this simple example first, we create a schema for Person and then run the diff operation on two instances of the Person data type, and finally, we encode a Person instance using Protobuf protocol:

import zio._
import zio.stream._
import zio.schema.codec.{BinaryCodec, ProtobufCodec}
import zio.schema.{DeriveSchema, Schema}

import java.io.IOException

final case class Person(name: String, age: Int)

object Person {
  implicit val schema: Schema[Person]    = DeriveSchema.gen
  val protobufCodec: BinaryCodec[Person] = ProtobufCodec.protobufCodec
}

object Main extends ZIOAppDefault {
  def run: ZIO[Any, IOException, Unit] =
    ZStream
      .succeed(Person("John", 43))
      .via(Person.protobufCodec.streamEncoder)
      .runCollect
      .flatMap(x =>
        Console.printLine(s"Encoded data with protobuf codec: ${toHex(x)}")
      )

  def toHex(chunk: Chunk[Byte]): String =
    chunk.map("%02X".format(_)).mkString
}

Here is the output of running the above program:

Encoded data with protobuf codec: 0A044A6F686E102B

Resources

• Zymposium - ZIO Schema by John A. De Goes, Adam Fraser, and Kit Langton (May 2021)
• ZIO SCHEMA: A Toolkit For Functional Distributed Computing by Dan Harris (Functional Scala 2021)
• Creating Declarative Query Plans With ZIO Schema by Dan Harris (ZIO World 2022)
• Describing Data...with free applicative functors (and more) by Kris Nuttycombe (Scala World) on the idea behind the xenomorph library

Documentation

Learn more on the ZIO Schema homepage!

Contributing

For the general guidelines, see ZIO contributor's guide.

TL;DR

Before you submit a PR, make sure your tests are passing, and that the code is properly formatted

sbt prepare

sbt test

Code of Conduct

See the Code of Conduct

Support

Come chat with us on .

License

License

zio-schema's Issues

Ambiguous implicit in build.sbt for unusedCompileDependenciesFilter

On a fresh pull of this project, I was getting an error from sbt while trying to load the project:

  [success] Total time: 6 s, completed Jul 9, 2021, 4:32:48 PM
  /Users/mrudolph/Projects/alterationx10/zio-schema/build.sbt:52: error: No implicit for Remove.Value[explicitdeps.ModuleFilter, sbt.librarymanagement.ModuleFilter] found,
  so sbt.librarymanagement.ModuleFilter cannot be removed from explicitdeps.ModuleFilter
  unusedCompileDependenciesFilter -= moduleFilter("org.scala-js", "scalajs-library")
  ^
  sbt.compiler.EvalException: Type error in expression
  [error] sbt.compiler.EvalException: Type error in expression
  [error] Use 'last' for the full log.

I believe the issue if from the sbt-explicit-dependencies plugin, along with something that's bringing in the sbt-updates plugin as well: see corresponding issue: cb372/sbt-explicit-dependencies#33

The general rundown is:

Summary: Ambiguous implicit import from same code in two projects...
Solution: Explicit import of ambiguous implicit.

PR incoming...

Derive ZIO Test Gen[R, A] from Schema[A]

Inside a new top-level project called zio-schema-zio-test, we can provide a derivation of ZIO Test's Gen for any data type that has a Schema[A].

package zio.schema

object DeriveGen {
  def gen[A](implicit schema: Schema[A]): Gen[Any, A] = ???
}

Add Schema[java.time.ChronoUnit]

This one can probably be derived and does not need to be added to the standard types.

Add CI/CD

As part of the hackathon this project was created from zio-web but without ci/cd.

Task will include

1. adding circle ci setup like other projects
2. updating zio-web to resolve from sonotype instead of from git

Patch value with Diff

This is a placeholder to gather thoughts but the initial idea is to:

Once #78 is implemented, we can add a method to the Schema[A] trait:

trait Schema[A] { self =>
  def patch(diff: Diff): Either[String, A => Either[String,A]]
}

which returns a Left(error message) if the diff cannot be applied to the schema structure (e.g. if I try to apply a Diff.Number to a Schema.Record and otherwise returns a function which will apply the specified diff to value of type A (possibly failing).

And associated syntax

implicit class DiffOps[A: Schema](a: A) {
   def patch(diff): Either[String,A] = Schema[A].patch(diff).flatMap(_.apply(a))
}

Laws:

1. Given a: A then a.patch(Diff.Identical) == Right(a)
2. Given a1: A and a2: A we should have a1.patch(a1.diff(a2)) == Right(a2)
3. Given a1: A and a2: A we should have

val diff1 = a1.diff(a2)
val diff2 = a2.diff(a1)
a1.patch(diff1).flatMap(_.diff(diff2)) == Right(a1)

Some questions:

1. Should the patching be able to fail if the the structure is correct. That is, should the signature be def patch(diff: Diff): Either[String, A => A] instead?

Enhance codec with non-streaming methods

Currently, Codec is defined in a stream-oriented way:

trait Codec {
  def encoder[A](schema: Schema[A]): ZTransducer[Any, Nothing, A, Byte]
  def decoder[A](schema: Schema[A]): ZTransducer[Any, String, Byte, A]
}

This is ideal for many applications, but imposes undue performance penalty on non-streaming code.

To solve this, we should enhance the design of codec as follows:

trait Codec {
  def encode[A](schema: Schema[A]): A => Chunk[Byte]
  def decode[A](schema: Schema[A]): Chunk[Byte] => Either[String, A]

  def encoder[A](schema: Schema[A]): ZTransducer[Any, Nothing, A, Byte]
  def decoder[A](schema: Schema[A]): ZTransducer[Any, String, Byte, A]
}

Specialize enumerations 1 - N

Similar to #2 but for enumerations. Remains to be seen exactly how to do this.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update peter-evans/create-pull-request action to v4.2.4
• Update actions/checkout action to v3.6.0
• Update actions/setup-java action to v3.13.0
• Update dependency org.scalameta:sbt-scalafmt to v2.5.2
• Update hmarr/auto-approve-action action to v2.4.0
• Update actions/checkout action to v4
• Update actions/setup-node action to v4
• Update dependency ubuntu to v22
• Update hmarr/auto-approve-action action to v3
• Update olafurpg/setup-scala action to v14
• Update peter-evans/create-pull-request action to v5

Detected dependencies

github-actions

.github/workflows/auto-approve.yml

• hmarr/auto-approve-action v2.1.0
• ubuntu 20.04

.github/workflows/ci.yml

• actions/checkout v3.0.0
• olafurpg/setup-scala v13
• coursier/cache-action v6
• actions/checkout v3.0.0
• olafurpg/setup-scala v13
• coursier/cache-action v6
• actions/checkout v3.0.0
• olafurpg/setup-scala v13
• ubuntu 20.04
• ubuntu 20.04
• ubuntu 20.04

.github/workflows/release-drafter.yml

• release-drafter/release-drafter v5
• ubuntu 20.04

.github/workflows/site.yml

• actions/checkout v3.3.0
• actions/setup-java v3.9.0
• actions/checkout v3.3.0
• actions/setup-java v3.9.0
• actions/setup-node v3
• actions/checkout v3.3.0
• actions/setup-java v3.9.0
• peter-evans/create-pull-request v4.2.3

npm

docs/package.json

sbt

build.sbt

project/BuildHelper.scala

• dev.zio:zio-test 2.0.19
• dev.zio:zio-test-sbt 2.0.19
• org.typelevel:kind-projector 0.13.2
• org.scalamacros:paradise 2.1.1
• com.github.vovapolu:scaluzzi 0.1.23

project/build.properties

• sbt/sbt 1.9.7

project/plugins.sbt

• org.scalameta:sbt-scalafmt 2.4.5
• ch.epfl.scala:sbt-scalafix 0.11.1
• com.github.cb372:sbt-explicit-dependencies 0.3.1
• org.scala-js:sbt-scalajs 1.14.0
• org.portable-scala:sbt-scalajs-crossproject 1.3.2
• com.eed3si9n:sbt-buildinfo 0.11.0
• com.github.sbt:sbt-ci-release 1.5.12
• org.portable-scala:sbt-scala-native-crossproject 1.3.2
• org.scala-native:sbt-scala-native 0.4.16
• pl.project13.scala:sbt-jmh 0.4.6
• dev.zio:zio-sbt-website 0.3.10
• org.snakeyaml:snakeyaml-engine 2.7

---

• Check this box to trigger a request for Renovate to run again on this repository

Implement Protobuf code generator

Given a protobuf, it should be possible to generate both the Scala data structure, as well as the Schema corresponding to this data structure.

Support native schema for Map

Currently there is no native support for maps. However, some protocols and formats have a native representation for maps (e.g. DynamoDB). Consequently, there is a strong advantage to having a native map type, distinct from Sequence.

• Add new Schema.Map case class, which stores a schema for key, stores a schema for value, and extends Schema[Map[K, V]].
• Support new schema type for all existing codecs and other operations that match on schema

@thinkharderdev Contender for ZIO Hackathon??

Specialize caseClass1 - caseClassN

Currently, all the Schema.caseClass methods are written in terms of Record. This is semantically correct, but not optimally performant, because it involves packing and unpacking into maps.

To maximize performance of codecs generated with Schema, we should add CaseClass subtypes of Schema for various common arities of case classes (1 - 22???). Then as a convenience, we could have a method that converts these case classes to a Schema.Record, for cases where performance does not matter that much (e.g. Schema.CaseClass2#toRecord.

Scala 3 Support

Add support for Scala 3.

The main issue to solve here is refactoring the macro derivation since Scala 3 macros are quite different from what is available in Scala 2. We can potentially leverage Magnolia 2.0 here but will need to investigate whether that will suffice.

Add Schema[None.type]

Since None.type is isomorphic to Unit, we should just derive a schema for it from the unit schema.

Add Schema[Nil.type]

Nil.type has no information so we can always describe it with a schema. It could be useful to serialize / deserialize empty lists, regardless of their element types. So we can add a (derived) schema for the type.

Add Schema#toJsonSchema

JSON Schema is a protocol to describe the structure of JSON values.

It would be great if we could take an existing schema for a user-defined data type, and convert it into a JSON schema, in such a way that the JSON codec could read and write data in that format.

trait Schema[A] {
  def toJsonSchema: String
}

Probably, we should implement this as a method on MetaSchema, and then delegate to it from Schema, because a MetaSchema is more than sufficient to document the structure of JSON values.

Implement a way to generate an Ordering[A] given a Schema[A]

Add the following method to Schema[A]:

def ordering: Ordering[A] 

Support for recursive types

The current encoding of case classes has a field for each field of the case class, that returns a tuple of String and Schema. But if the type is recursive or if indirectly it contains classes that have circular dependencies, it will blow up with stack overflow. Example: https://scastie.scala-lang.org/mSCrN0pQRjOw9pBZNzMSAQ

In Caliban I dealt with this by making the field information lazy (() => FieldInfo instead of FieldInfo).

Introduce type for Schema to capture structure of algebraic data type

Just placing this here to think about, it's not fully fleshed out, but I think we need a way to extract lenses / prisms from schema, and that means having a way to capture the structure of ADTs.

trait Schema[A] {
  type Structure
}
type SchemaStructure[Structure0, A] = Schema[A] { type Structure = Structure0 }

final case class Person(name: String, age: Int)

SchemaStructure[String :*: Int :*: Unit, Person]

Proper handling of recursive data types

The current implementation of JsonCodec, ProtobufCodec, DynamicValue and MetaSchema do not handle recursive data types such as

  sealed trait Json
  case object JNull                                extends Json
  case class JString(s: String)                    extends Json
  case class JNumber(l: Double)                    extends Json
  case class JObject(fields: List[(String, Json)]) extends Json

There are currently (ignored) test cases for these issues for reference.

We probably need to introduce a fix point operator to handle this (and other cases that will arise in any codec implemented from a Schema)

Add codec for UTF Strings

It would be useful to have simple codec that work with utf strings

Allow pattern matching on Schema for tuples, eithers

Tuples and eithers (products/sums) are so core to schema that we should bake them in and allow pattern matching against them, rather than encoding them atop Record / Enum.

Add Schema.tuple5 - Schema.tuple22

There are implicit def tuple2 all the way up to implicit def tuple4 inside the companion object of Schema. The pattern can be continued all the way up to tuples of size 22.

e.g.:

implicit def tuple5[A, B, C, D, E](implicit a: Schema[A], b: Schema[B], c: Schema[C], d: Schema[D], e: Schema[E]): Schema[(A, B, C, D, E)] = ???

...

implicit def tuple22[A, B, C, ...](...): Schema[(A, B, C, D, ...)] = ???

Add `defaultValue` to Schema trait

A Schema#defaultValue: Either[String, A] method can be added to the trait Schema, which computes a "default value" for the specified Schema.

e.g.:

trait Schema[A]
  def defaultValue: Either[String, A]
}

This default value can be computed as follows:

1. For records, compute the default value of their fields.
2. For enumerations, pick the first case.
3. For primitives, pick whatever seems natural (e.g. 0 or 0.0 or empty string).

Possibly, we can enhance this later with a @defaultValue annotation.

Add support for Scala.js

Currently, the build is only configured to build for JVM. We can add JS support easily by making it a cross-build, as per ZIO and other projects in the ZIO ecosystem.

Rationalize handling of nested Optional schemas

Currently if we have a nested Optional schema such as Schema.Optional(Schema.Optional(Schema.primtive(StandardType.StringType)) we cannot distinguish between the JSONprotobuf encodings of Some(None) and None which will both be encoded to null in JSON (and similarly for protobuf).

As a first order issue this will cause test failures when we do a "round-trip" encoding/decoding of Some(None which will encode to null and then decode to just plain None instead of Some(None).

A couple of options for handling this:

1. Use a type constraint to prevent "nesting" of optional schemas

trait Schema[A] {. self =>
  def optional(implicit ev: A <:!< Option[Any]): Schema.Optional[Option[A]]
} 

This generalized type constraint doesn't exist in Scala so we'd have to create it

2. Allow nested optional schemas but make it so Some(None) == None in the specs.

Add a generic data representation

We should have a generic data representation (Generic) that mirrors the Schema hierarchy, such that we can convert an A and a Schema[A] into a Generic.

Something like:

sealed trait Schema[A] {
  def toGeneric(value A): Generic = ???

  def fromGeneric(generic: Generic): Either[String, A] = ???
}

where:

sealed trait Generic
object Generic {
  final case class Integer(value: Int) extends Generic
  final case class Record(values: Map[String, Generic]) extends Generic
  final case class Enumeration(value: (String, Generic)) extends Generic
  ...
}

As a law: schema.fromGeneric(schema.toGeneric(a)) === Right(a).

It would be ideal (for #35) if there is an Ordering for Generic, as well as (of course) consistent equals/hashCode.

Rename implicits with the same name in the companion object of Schema

I recently discovered Scala cannot choose between two or more implicit methods with the same name.

To solve this problem, we should give every implicit method in the companion object of Schema a different name. The name can reflect the arity of the operator.

For example, zip2 instead of zip.

Add support for BigInteger and BigDecimal as StandardTypes

The omission of BigInteger and BigDecimal seems to be accidental.

Support for Annotations on all types

Annotation support is needed on all types not just on Record.

Currently records support annotations, but it would be useful to also support annotations on all Schema types.

https://github.com/zio/zio-schema/blob/main/zio-schema/shared/src/main/scala/zio/schema/Schema.scala#L218

Implement Enum4 - Enum22

Implement sealed trait hierarchies for arity up to 22. This should extend the existing implementation for Schema.Enum1,Schema.Enum2,Schema.Enum3 up to Schema.Enum22 and keep Schema.EnumN for higher arities.

Define valid equality/hashCode on Schema

Schema must have well-defined equality and hashCode methods.

Note that due to the presence of Schema#transform, which stores functions in the ADT, we will have to use reference equality on functions to check for equality and to compute hash code. Reference equality implies equality, but is not necessary for equality, which means that two schemas could in fact be equal, but equals will return false. That's a known and acceptable limitation of this approach.

We should have a test suite that demonstrates equal schemas compare as equal and hash the same.

Add support for latest version of Magnolia for deriving Schemas

A particular goal is taking advantage of the Scala 3 support that is currently in-progress.

Add support for flat buffer codecs

FlatBuffer should have their own ZIO Schema Codec.

Add Schema.Fail

Schema.Fail will be the schema that always fails. It represents the absence of schema information for a given type.

Something like:

object Schema {
  final case class Fail[A](message: String) extends Schema[A]
  def fail[A](message: String): Schema[A] = SchemaFail(message)
}

Protobuf Codec: Fails to decode lists of lists

Currently, we fail to decode empty lists of lists. Issue can replicated with the following test case

  case class WithPiggyback(
    conversationId: Long,
    message: Chunk[Byte],
    gossip: List[Chunk[Byte]]
  )

  object WithPiggyback {
    implicit val schema: Schema[WithPiggyback] = DeriveSchema.gen

    implicit val gen: Gen[Random with Sized, WithPiggyback] =
      for {
        conversationId <- Gen.anyLong
        message        <- Gen.chunkOf(Gen.anyByte)
        gossip         <- Gen.listOf(Gen.chunkOf(Gen.anyByte))
      } yield WithPiggyback(conversationId, message, gossip)
  }

      testM("record with array fields") {
        checkM(WithPiggyback.gen) { value =>
          encodeAndDecodeNS(Schema[WithPiggyback], value, print = true).map { ed =>
            assert(ed)(equalTo(value))
          }
        }
      }

The issue seems to be in composing Decoders where flatMap errors out if there are no bytes remaining in the chunk after running the "first" decoder. This causes an issue when the final field in a Record is a collection. In that case, the key will be decoded as a length delimited field with zero length. As expected there are no more bytes to decode since the list is empty but when flatMaping the Decoder will error out.

Add Schema[List] for pattern matching

Add intelligent transformation of data between two schemas

Given Schema[A], and Schema[B], we should be able to produce Either[String, A => Either[String, B]], either alone, or with hints.

trait Schema[A] {
  ...
  def migrate[B](newSchema: Schema[B]): Either[String, A => Either[String, B]] = ???
}

A simple case is Schema[B] having smaller records and bigger enumerations, but the names of terms and relative position of structure is unchanged, in which case, the transition may always succeed.

Hints would allow more sophisticated transformations (including those requiring default values), as would an optimizer that can detect unambiguous field renames.

Upgrade to ZIO 2.0

ZIO 2.0 is at Milestone 4, with an RC expected in the next few weeks.
https://github.com/zio/zio/releases/tag/v2.0.0-M4

The API is nearly stable at this point, so any early migration work against this version should pay off towards the official 2.0 release.

The progress is being tracked here:
zio/zio#5470

The Stream Encoding work in progress is the only area where the API might still change before the RC.

We are actively working on a ScalaFix rule that will cover the bulk of the simple API changes:
https://github.com/zio/zio/blob/series/2.x/scalafix/rules/src/main/scala/fix/Zio2Upgrade.scala
We highly recommend starting with that, and then working through any remaining compilation errors :)

To assist with the rest of the migration, we have created this guide:
https://zio.dev/howto/migrate/zio-2.x-migration-guide/

If you would like assistance with the migration from myself or other ZIO contributors, please let us know!

Implement JsonCodec using ZIO JSON

Moving this issue here from https://github.com/zio/zio-web/issues/73

Add support for annotations into Schema

Using macros, we should pick up and store annotations on case classes, enumerations (sealed traits), and the individual terms that make up them (fields in case classes, cases of enumerations).

This way, without macros, users of Schema can peer into and alter behavior based on annotations.

Add Schema[Left[A, B]] and Schema[Right[A, B]]

A schema for Left[A, B] can be implemented trivially just by transforming a Schema[A]. Same for Right[A, B].

object Schema {
  implicit def schemaLeft[A, B](implicit schemaA: Schema[A]): Schema[Left[A, Nothing]] = schemaA.transform(Left(_), _.value)
  implicit def schemaRight[A, B](implicit schemaB: Schema[B]): Schema[Right[Nothing, B]] = schemaB.transform(Right(_), _.value)
}

Alternate implementation: using Schema.Fail for the side that is known to not exist.

Serialize type class

import zio.schema._

sealed trait DummyExpr[A] {
  def eval(value: DummyExpr[A]): A
}

object DummyExpr {
  case class Dummy(predicate: DummyExpr[Boolean]) extends DummyExpr[Double] {
    override def eval(value: DummyExpr[Double]): Double = ???
  }
}

object main extends App {
  val schema = DeriveSchema.gen[DummyExpr[Double]]
}

Compilation for this fails with error magnolia: could not find any direct subtypes of trait DummyExpr.
It is very hard to understand the issue from the error. Can we have some working examples around this?

I am using zio-schema 0.0.6 version

Implemented automatic diff

We should be able to use a Schema[A] to automatically diff values of type A.

To do this we should add a method like so

trait Schema[A] { self =>
  def diff(thisValue: A, thatValue: A): Diff =  DiffAlgorithm.fromSchema(self)(thisValue,thatValue)
}

Where we have DiffAlgorithm and Diff that look something like

trait DiffAlgorithm[A] { self =>
  def apply(thisValue: A, thatValue: A): Diff
}

object DiffAlgorithm {
  def fromSchema[A](schema: Schema[A]): DiffAlgorithm[A] = ???
}

sealed trait Diff
object Diff {
  final case object Identical extends Diff
  final case class Number[A : Numeric](distance: A) extends Diff
  final case class Tuple(leftDifference: Diff, rightDifference: Diff) extends Diff
  // etc
}

Implement Avro codec

We need to implement two codecs for Avro:

1. An AvroCodec to serialize/deserialize the Avro binary serialization format
2. An AvroSchemaCodec to serialize a Schema[A] to an Avro JSON schema and deserialize an Avro JSON schema to a Schema.GenericRecord

This should be implement in a zio-schema-avro sub-project

Reimplementation of Scala 2 macro derivation

The current magnolia based Schema derivation is not sufficient for usable type inference of the Accessors type member . With the current implementation, users either need to explicitly annotate the Schema type:

implicit val schema: Schema.CaseClass1[String,Foo] = DeriveSchema.gen[Foo]

which is suboptimal since the type signatures can be very long.

This can be solved by refactoring the macro derivation to use a whitebox macro.

Alphabetically sort all method names in the companion object of Schema

All method names in the companion object of Schema should be sorted alphabetically.

Separate codecs into subprojects

Individual codecs (JSON/protocol buffers/etc) should be implemented in subprojetts and published as separate modules which can be included as separate dependencies on as as-needed basis.

Based on the current implementations we should break things down into the following sub-projects:

• zio-schema (container the core Schema types, traits and macros)
• zio-schema-json (contains the JsonCodec implementation)
• zio-schema-protobuf (contains the ProtobufCodec implementation)
• zio-schema-zio-test (contains Gen derivation for Schemas)

Future codec implementations should be added as subproject zio-schema-<codec>

Shift to use ordered maps for Schema.Record and Schema.Enumeration

Currently, we are using a Map for the keys in records and enumerations, which means that we cannot rely on the key ordering. Yet for some protocols, such as grpc, the ordering is very important: it must be stable and ideally reflected by case class / subtype ordering. In order to accommodate this, we need to switch to a map type that allows linear ordering of keys.

Scala has SortedMap, which could be used, but is not really ideal (it wants to order based on a property of the keys, but that would force us to store (Int, String) in the map, which is unpleasant to work with). Possibly we could use an ordinary map, together with an array that maintains key ordering; or we could create our own OrderedMap type.

Implement Thrift Codec

We should implement a codec to serialize/deserialize a Schema to the Apache Thrift binary serialization format.

This should be added in the zio-schema-thrift subproject

Add Schema#serializable

Add a new method to Schema:

sealed trait Schema[A] {
  def serializable: Schema[Schema[_]]
}

This will return a Schema for the original Schema, but without the transform nodes -- i.e. without any functions inside of them.

This will allow you to serialize a schema across the wire (or deserialize it), as long as you're willing to "forget" which Scala types are being used to model the built-in types.