andersfugmann/ppx_protocol_conv

Pluggable serialization and deserialization of ocaml data strucures based on type_conv

Ppx Protocol Conv

Ppx protocol conv (de)serializers using deriving, which allows for plugable (de)serializers. Api.

This page contains an simple overview of functionality provided. More information is available in the wiki pages

Table of contents

1. Features
2. Examples
3. Drivers
4. Custom drivers
5. Not supported

Features

The ppx supports the following features:

• records
• recursive and non-recursive types
• variants
• polymophic variants
• All primitive types (except nativeint)

The following drivers exists

• Json which serializes to Yojson.Safe.t
• Jsonm which serializes to Ezjsonm.value
• Msgpack which serializes to Msgpck.t
• Yaml which serializes to Yaml.t
• Xml_light which serializes to Xml.xml list
• Xmlm which serializes to Ezxmlm.node

Examples

open Protocol_conv_json
type a = {
  x: int;
  y: string [@key "Y"]
  z: int list [@default [2;3]]
} [@@deriving protocol ~driver:(module Json)]

type b = A of int
       | B of int [@key "b"]
       | C
[@@deriving protocol ~driver:(module Json)]

will generate the functions:

val a_to_json: a -> Json.t
val a_of_json_exn: Json.t -> a
val a_of_json: Json.t -> (a, exn) result

val b_to_json: a -> Json.t
val b_of_json_exn: Json.t -> a
val b_of_json: Json.t -> (b, exn) result

a_to_json { x=42; y:"really"; z:[6;7] }

Evaluates to

[ "x", `Int 42; "Y", `String "really"; "z", `List [ `Int 6; `Int 7 ] ] (* Yojson.Safe.json *)

to_protocol deriver will generate serilisation of the type. of_protocol deriver generates de-serilisation of the type, while protocol deriver will generate both serilisation and de-serilisation functions.

Attributes

Record label names can be changed using [@key <string>]

Variant and polymorphic variant constructors names can be changed using the [@name <string>] attribute.

If a record field is not present in the input when deserialising, as default value can be assigned using [@default <expr>]. If the value to be serialized matches the default value, the field will be omitted (Some drivers allow disabling this functonality. Comparrison uses polymorphic compare, so be careful.

Signatures

The ppx also handles signature, but disallows [@key ...], [@default ...] and [@name] .... as these does not impact signatures.

Drivers

Drivers specify concrete serialization and deserialization. Users of the library can elect to implement their own driver see custom drivers, or use predefined drivers:

• Json which serializes to Yojson.Safe.t
• Jsonm which serializes to Ezjsonm.value
• Msgpack which serializes to Msgpck.t
• Yaml which serializes to Yaml.t
• Xml_light which serializes to Xml.xml list
• Xmlm which serializes to Ezxmlm.node

Custom drivers

It is easy to provide custom drivers by implementing the signature:

include Protocol_conv.Runtime.Driver with
  type t = ...

See the drivers directory for examples on how to implemented new drivers. Submissions of new drivers are more than welcome.

Not supported

• Generalised algebraic datatypes
• Extensible types
• Extensible polymorphic variants

ppx_protocol_conv's Issues

Support for base >= v0.14

See ocaml/opam-repository#16711

Failed to deserialize msgpck

Hi!

I am currently trying to deserialize a msgpacked message. I defined my object in the following way

type member = {
    name : string; [@key "Name"]
    ...
  } [@@deriving protocol ~driver:(module Msgpack)]

  type t = {
    members : member list; [@key "Members"]
  } [@@deriving protocol ~driver:(module Msgpack)]

  let to_msgpack = t_to_msgpack
  let from_msgpack = t_of_msgpack

When the data arrives I print it with

    print_endline (Msgpck.show members);

And it prints well, but after that I try to get the object back from Msgpck.t to Members.t with

    let deserialized = Members.from_msgpack members in
    print_endline "finished";

But it never prints anything. Is it possible that the deserialization process gets stuck somewhere?
I am new to ocaml, so it is possible that I messed up something.

Thanks in advance,
Daniel

The of_json functions are raising an exception instead of returning an error

This concerns the Ppx_protocol_conv_json package.

yaml 3.0.0 changes output format

Yaml 3.0.0 now detects special strings (like true or 1.0) and quotes them so that they can be reconstructed as strings and not interpreted as floats or bools.

This breaks the test cases in ppx_protocol_conv_yaml, so I've added an upper bound in ocaml/opam-repository#19204 for this. The yaml 3.0.0 release is in ocaml/opam-repository#19202

Upgrade from ppx_type_conv to ppxlib

ppx_type_conv and the other ppx libraries have been merged into ppxlib and deprecated. At this point the latest ppxlib (0.3) is incompatible with the last version of ppx_type_conv and thus causes a downgrade when trying to install ppx_protocol_conv.

(ez)xmlm for xml parsing

xml-lite has, unfortunately, been effectively unsupported for a long while. It would good to use something that's more actively supported.

More fingraned user control over structures

Use GADT's to describe variants, tuples and records. This allows the user code to do heavy optimization and startup evaluation to produce faster code.

Support for Msgpck.Bytes

Hi, thanks for the great library!

I was wondering if you could add support for deserializing fields of type Msgpck.Bytes into OCaml fields (maybe of type Bytes?). I am communicating with a service whose response is correctly deserialized into Msgpck.Bytes (see here) but the automatic conversion provided by ppx_protocol_conv is unable to handle this value.

Doesn't seem to be working -- followed instructions in README to no avail

I followed the README, but no de/serializer functions are getting generated. I'm attaching a transcript.

(0) ocaml 4.07.1 installed via opam

(1) versions of (what I think are) relevant packages, freshly installed via opam

The following actions will be performed:

• install yojson 1.7.0
• install ppx_derivers 1.0 [required by ppxlib]
• install ocaml-migrate-parsetree 1.2.0 [required by ppxlib]
• install ppxlib 0.5.0 [required by ppx_protocol_conv]
• install ppx_protocol_conv 3.1.3 [required by ppx_protocol_conv_json]
• install ppx_protocol_conv_json 3.1.3

(2) transcript of toplevel interaction
#use "topfind";;

• : unit = ()
  Findlib has been successfully loaded. Additional directives:
  #require "package";; to load a package
  #list;; to list the available packages
  #camlp4o;; to load camlp4 (standard syntax)
  #camlp4r;; to load camlp4 (revised syntax)
  #predicates "p,q,...";; to set these predicates
  Topfind.reset();; to force that packages will be reloaded
  #thread;; to enable threads

• : unit = ()

#require "ppx_protocol_conv_json";;

/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv/runtime: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv/runtime/protocol_conv.cma: loaded
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv/driver: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv/driver/ppx_protocol_driver.cma: loaded
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/easy-format: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/easy-format/easy_format.cma: loaded
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/biniou: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/biniou/biniou.cma: loaded
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/yojson: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/yojson/yojson.cma: loaded
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv_json: added to search path
/home/chet/Hack/Ocaml/4.07.1/ocaml-base-compiler.4.07.1/lib/ppx_protocol_conv_json/protocol_conv_json.cma: loaded

open Protocol_conv

open Protocol_conv_json
;;

#   type a = {

x: int;
y: string [@key "Y"]
} [@@deriving protocol ~driver:(module Json) ~flags:(`Mangle Json.mangle)]
;;

    type a = { x : int; y : string; }

type b = A of int
| B of int [@key "b"]
| C
[@@deriving protocol ~driver:(module Json)] ;;

    type b = A of int | B of int | C

Support result type

Type result has been part of Pervasives since 4.03.0.

Support ppxlib 0.10

Currently installing ppx_protocol_conv together with ppxlib 0.10 (and thus base >= 0.13) is not possible.

Ppx_protocol_driver does not support bytes

Special case when type is called `t`

I greatly enjoy the (common) case when a module is working on a specific type t then the derivers in ppx_deriving would not add t_to_hest and t_of_hest but rather to_hest and of_hest, avoiding some boilerplate.

It would be great if this module could at least create an alias of of_hest to t_of_hest and to_hest to t_to_hest, which would keep existing code running but add some convenience.

Split drivers out into separate packages

With jbuilder and maybe topkg it is pretty easy to split out subpackages (as the Mirage project is documented. It would be nice if all the derivers which introduce dependencies (xml-light, msgpck, yojson) could be split into their own respective sub-derivers, so one can just depend on the parts that are required.

I'd look into it myself, but I thought I might as well create a tracking issue for this, so here it is.

Make the conversion functions return a result instead of exception

I started to change the code of merlin-lsp from ppx_deriving_yojson to ppx_protocol_conv_json. It is mostly mechanical work that does not create problems. And this new ppx adds some nice features. But there is one annoying point. The of_yojson functions of ppx_deriving_yojson are returning a result rather. ppx_protocol_conv creates functions that raise exceptions. In general, I think that it is preferable to use the result type.

Would it be possible to follow the same behavior than ppx_deriving_yojson? With eventually an optional way to generate the of_json_exn function if necessary?

If it makes sense for you, I could try to contribute to the change.

ocaml/merlin#948
https://github.com/ocaml-ppx/ppx_deriving_yojson#syntax

Feature request: [@default ...]

I come from ppx_deriving_yojson. When I switched to Ezjsonm, I discovered this project, and I must admit it find it really nice (in particular, the fact that there is support for other serialisation formats like Yaml). There is just one feature that I find really cool and that I miss here: the [@default ...] attribute. It allows to write:

type t =
    { bla : string [@default ""] ;
      blo: int list [@default [42]] ;
      blu: float }
[@@deriving protocol ~driver:(module ...)]

The fields bla and blo become optional in the input serialised value and, in case they're not there, the default is used. I don't know how easy it would be to add, or if you'd accept PR. I could give it a go.

Using a deriver causes base and ppxlib to be linked

Consider this minimal example from the readme:

open Protocol_conv_json
type a = {
  x: int;
  y: string [@key "Y"];
  z: int list [@default [2;3]]
} [@@deriving protocol ~driver:(module Json)]

(executable
 (name test)
 (preprocess
  (pps ppx_protocol_conv_json))
 (libraries ppx_protocol_conv_json))

The resulting binary is 31 MB and contains both base (probably due to the inline tests) and ppxlib (including OCaml's compiler libs) to be linked. Checked by running nm on the executable, for example nm _build/default/test.exe | grep camlBase.

Defining Map fields

I have an OCaml type defined like this:

type t = {
  messages : (string * string) list; [@key "Messages"]
} [@@deriving protocol ~driver:(module Msgpack)]

I would like to treat messages as Msgpck.Map but apparently ppx_protocol_conv treats it as Msgpck.List. When I try to unmarshal such a value, I get list expected, {}, which suggests that the underlying Msgpck structure is a Map, but this lib tries to read it as a List.

My question is: how to define fields as Maps instead of List?

Adapt to ppxlib 0.22.0

It's currently constrained to < "0.18.0":

% opam show ppx_protocol_conv |grep ppxlib
              "ppxlib" {>= "0.9.0" & < "0.18.0"}