ergl/pony-protobuf

Pony implementation / compiler of Google protocol buffers

pony-protobuf

A protocol buffers library and compiler for Pony.

Status

pony-protobuf is an alpha-level package. Expect API breakage. For now, pony-protobuf only supports proto2 files. The generated Pony files have a run-time dependency of this package.

What works

• Message definitions (also messages in messages).
• Scalar types.
• Importing other files (although namespaces are flat, see point below about packages).
• Nested types.
• The packed and default options for fields (but not for bytes, see #2).
• oneof fields.
• Skipping over unknown message fields.
• Merging of messages (simply call parse_from_stream multiple times).

What doesn't work (yet)

• map<_,_> syntax
• "proto3" syntax
• Message extensions.
• Generating descriptor metadata
• Preserving unknown types when parsing (proto2, proto3 3.5+).
• groups (proto2). Although deprecated, the current library doesn't know how to handle these. In the future it will ignore any groups when marshalling.
• Default definitions for byte fields.
• The allow_alias enum option.
• JSON serialization.
• Service definitions.
• Proper package namespaces. Importing packages is supported, but the generated code assumes that the types will be present in the same Pony package.

Installation

• Install corral
• corral add github.com/ergl/pony-protobuf.git --version 0.2.3
• corral fetch to fetch your dependencies
• use "protobuf" to include this package
• corral run -- ponyc to compile your application

Nit: the above will include the protobuf library in your application. To use the compiler, you will need to build from source (a binary download / homebrew package might be nice).

Compiler Usage

The compiler is implemented as a protoc plugin. If you don't have protoc installed, see the install notes.

If you have downloaded the repo, run make plugin. This will generate a protoc-gen-pony file in build/release that you can use with protoc, as such:

protoc --pony_out=<out_dir> --plugin=build/release/protoc-gen-pony path/to/proto/file.proto

For more protoc options, see protoc --help. If you have protoc-gen-pony in your PATH, you can omit the --plugin flag.

Examples

Take a peek at the examples directory. It contains auto-generated Pony files. In general, the generated code looks like this:

use "protobuf"

// <snip>

class AddressBook is ProtoMessage
  var person: Array[Person] = Array[Person]

  fun compute_size(): U32 =>
    var size: U32 = 0
    for v in person.values() do
      size = size + FieldSize.inner_message(1, v)
    end
    size

  fun ref parse_from_stream(reader: ProtoReader) ? =>
    while reader.size() > 0 do
      match reader.read_field_tag()?
      | (1, DelimitedField) =>
        let v: Person = Person
        v.parse_from_stream(reader.pop_embed()?)?
        person.push(v)
      | (_, let typ: TagKind) => reader.skip_field(typ)?
      end
    end

  fun write_to_stream(writer: ProtoWriter) =>
    for v in person.values() do
      writer.write_tag(1, DelimitedField)
      // TODO: Don't recompute size here, it's wasteful
      writer.write_varint[U32](v.compute_size())
      v.write_to_stream(writer)
    end

  fun is_initialized(): Bool =>
    for v in person.values() do
      if not v.is_initialized() then
        return false
      end
    end
    true

Mapping of protocol buffer types to Pony

Protobuf type	Pony type
bool	Bool
double	F32
float	F64
int32, sint32, fixed32	I32
int64, sint64, fixed64	I64
uint32	U32
uint64	U64
string	String val
bytes	Array[U8] ref
enum	Primitives (see below)
oneof	(ChosenField, value): union type of tuples and primitives (see below)
map<_,_>	Not supported (yet)
groups	Not supported (no plans)

Repeated fields are represented as Array[T] ref, and optional types are represented as (T | None). Required types (proto2) are also represented as (T | None), to discern between uninitialized types and default values. Users are encouraged to call Message.is_initialized() to ensure that the message contains all required types.

Enums

Since Pony lacks the concept of C-style enums, pony-protobuf opts to represent them as primitive types, one per enum value. A type alias is also generated to represent the valid values, along with an utility primitive to translate between the numeric representation and the primitive type. Here's an example from addressbook.pony:

enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
}

primitive PhoneTypeMOBILE is ProtoEnumValue
  fun as_i32(): I32 => 0
  fun string(): String => "PhoneTypeMOBILE"

primitive PhoneTypeHOME is ProtoEnumValue
  fun as_i32(): I32 => 1
  fun string(): String => "PhoneTypeHOME"

primitive PhoneTypeWORK is ProtoEnumValue
  fun as_i32(): I32 => 2
  fun string(): String => "PhoneTypeWORK"

type PhoneType is (
  PhoneTypeMOBILE
  | PhoneTypeHOME
  | PhoneTypeWORK
)

primitive PhoneTypeBuilder is ProtoEnum
  fun from_i32(value: I32): (PhoneType | None) =>
    match value
    | 0 => PhoneTypeMOBILE
    | 1 => PhoneTypeHOME
    | 2 => PhoneTypeWORK
    else
      None
    end

Another option to represent enum types would be to have a single primitive, with one function per enum field, like so:

primitive PhoneType
  fun field_MOBILE(): I32 => 0
  fun field_HOME(): I32 => 1
  fun field_WORK(): I32 => 2

This option is shorter, doesn't pollute the namespace, but doesn't offer any type-checking affordances to the user. That's the reason I opted for the more verbose alternative. In the future, I might change this.

oneof fields

In protobuf, a oneof field represents a set of many optional fields, where at most once field is set at any time. To distiniguish between different fields, the generated Pony code uses a union type, where each element is a tuple of the form (Name, value). Each Name is represented by a Pony primitive. Here's an example from oneof.pony:

message SampleMessage {
  oneof test_oneof {
    int32 field_a = 1;
    int32 field_b = 2;
    string name = 3;
    SubMessage sub_message = 4;
  }
}

primitive SampleMessageFieldAField
primitive SampleMessageFieldBField
primitive SampleMessageNameField
primitive SampleMessageSubMessageField

class SampleMessage is ProtoMessage
  var test_oneof: (
    None
    | (SampleMessageFieldAField, I32)
    | (SampleMessageFieldBField, I32)
    | (SampleMessageNameField, String)
    | (SampleMessageSubMessageField, SubMessage) = None

  // ...

API Documentation

API documentation

Performance

Baby steps.

pony-protobuf's Issues

Implement skipping groups in parse_from_stream

Avoid recomputing field size when marshalling nested messages

Since nested messages are length-delimited, we need to compute their size when marshalling. But we could cache their size to avoid recomputing it every time.

Avoid extra copying when (un)marshalling

1. Currently ProtoWriter.write_bytes needs to copy the entire array, since the original field is Array[U8] ref, and the argument for Writer.write takes a BytSeq

2. ProtoReader.pop_embed will allocate a new array when parsing nested messages. Ideally, we'd do like rust-protobuf: use configurable "limits" that allow us to pass the original ProtoReader to nested messages, without worrying that they will read more bytes than they should.

Rethink how custom default values are treated.

Right now we're treating custom default values (proto2) as default assignments to the fields:

// proto
message Message
    optional int32 foo = 1 [default = 10];
}

// pony
class Message
  var foo: (I32 | None) = 10

For oneof fields, if several options have a custom default value:

// proto
message Message {
    oneof test {
        int32 foo = 1 [default = 10];
        int32 bar = 2 [default = 20];
    }
}

// pony
class Message
  var test: ((MessageFooField, I32) | (MessageBarField, I32) | None) = // ???

we have to choose which default value to use, but ideally we'd support all. Right now we're doing LWW, meaning, last default value (in field number order) wins.

Don't generate is_initialized if there are no required fields in nested messages

Right now we generate is_initialized if the message contains nested messages, even if those nested messages don't have. The compiler already has this information when generating nested messages, so it should tell parent messages.

Implement default values for `byte` messages

Default values for byte fields are represented using C-escaped values. Right now we ignore any default byte values, but we should implement parsing for these types. See golang/protobuf#427

Improve generator error reporting

Right now error reporting when generating Pony code is absent. If we encounter some internal errors, we should abort the entire codegen process, instead of silently dropping some statements from the final program.