Giter Site home page Giter Site logo

dproto's Introduction

D Protocol Buffers


Build Status Coverage Status DUB DUB license

Protocol buffers are a language-agnostic way of specifying message structures to allow communication and serialization.

dproto is designed to enable mixing protocol buffer files into your D code at compile time.

Inspiration and a good portion of the original parser is adapted from square/protoparser


Options

dproto supports altering behavior via protobuf options:

Option Meaning Example Default
dproto_reserved_fmt The format for renaming reserved D keywords as fields. "%s_" will convert version to version_ "%s_"

Examples

Further info

Examples can be found in import/dproto/dproto.d and in examples/.

Simple Example

import std.stdio;
import dproto.dproto;

mixin ProtocolBufferFromString!"
	message Person {
	  required string name = 1;
	  required int32 id = 2;
	  optional string email = 3;

	  enum PhoneType {
		MOBILE = 0;
		HOME = 1;
		WORK = 2;
	  }

	  message PhoneNumber {
		required string number = 1;
		optional PhoneType type = 2 [default = HOME];
	  }

	  repeated PhoneNumber phone = 4;
	}
";


int main()
{
	Person person;
	person.name = "John Doe";
	person.id = 1234;
	person.email = "[email protected]";

	ubyte[] serializedObject = person.serialize();

	Person person2 = Person(serializedObject);
	writeln("Name: ", person2.name);
	writeln("E-mail: ", person2.email);
	return 0;
}

More Complex Example

import dproto.dproto;

mixin ProtocolBufferFromString!"
	enum PhoneType {
	  MOBILE = 0;
	  HOME = 0;
	  WORK = 2;
	}

	message Person {
	  required string name = 1;
	  required int32 id = 2;
	  optional string email = 3;

	  message PhoneNumber {
		required string number = 1;
		optional PhoneType type = 2 [default = HOME];
	  }

	  repeated PhoneNumber phone = 4;
	}

	message AddressBook {
	  repeated Person person = 1;
	}
";


int main()
{
	Person t;
	t.name = "Max Musterman";
	t.id = 3;
	t.email = "[email protected]";

	Person.PhoneNumber pn1;
	pn1.number = "0123456789";
	pn1.type = PhoneType.WORK;

	Person.PhoneNumber pn2;
	pn2.number = "0123456789";

	t.phone = [pn1, pn2];
	AddressBook addressbook;
	addressbook.person ~= t;
	addressbook.person ~= t;

	ubyte[] serializedObject = addressbook.serialize();

	AddressBook addressbook2 = AddressBook(serializedObject);
	assert(addressbook2.person.length == 2);
	foreach(t2; addressbook2.person)
	{
		assert(t2.name == "Max Musterman");
		assert(t2.id == 3);
		assert(t2.email == "[email protected]");
		assert(t2.email !is null);
		assert(t2.phone[0].number == "0123456789");
		assert(t2.phone[0].type == PhoneType.WORK);
		assert(t2.phone[1].number == "0123456789");
		assert(t2.phone[1].type == PhoneType.HOME);
		assert(t2.phone[1].type == PhoneType.MOBILE);
		assert(t2.phone.length == 2);
	}
	version(DigitalMars)
	{
		assert(addressbook2.person[0] == addressbook.person[1]);
	}
	return 0;
}

Services

Generate interfaces for service definitions.

import dproto.dproto;

mixin ProtocolBufferInterface!"
	message ServiceRequest {
		string request = 1;
	}
	message ServiceResponse {
		string response = 1;
	}
	service TestService {
		rpc TestMethod (ServiceRequest) returns (ServiceResponse);
	}
";

class ServiceImplementation : TestService {
	ServiceResponse TestMethod(ServiceRequest input) {
		ServiceResponse output;
		output.response = "received: " ~ input.request;
		return output;
	}
}

dproto's People

Contributors

clawconduce avatar denizzzka avatar elvisxzhou avatar geod24 avatar kelet avatar martinnowak avatar msoucy avatar redstar avatar shigekikarita avatar timotheecour avatar webdrake avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dproto's Issues

package.d

Support should be added for DMD 2.064+ style packages, changing dproto.d to package.d.

Until 2.064 gets regular adoption, it's probably a good idea to make package.d just public import dproto.d

option to convert from a D struct to/from a Proto when this makes sense

would be nice to support conversions, when possible:

eg:

Person1 proto;
Person2 s;

proto.deserialize("...");
s.readFrom(proto);

Person1 proto2;
proto2.readFrom(s);
assert(proto2==proto);
message Person1 {
      required string name = 1;
      required int32 id = 2;
      optional string email = 3;

      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }

      message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
      }

      repeated PhoneNumber phone = 4;
    }
struct Person2{
    string name;
    int d;
    string email;

      enum PhoneType {
        MOBILE = 0,
        HOME = 1,
        WORK = 2,
      }

      struct PhoneNumber {
        string number;
        PhoneType type = PhoneType.HOME;
      }
      PhoneNumber[] phone;
    }
}

opGet() needed on RepeatedBuffer for use with std.algorithm.remove

The title is overly specific, but I'm not sure of the concrete implications. The comments line is what I'd expect, whereas the noncommented line before it is what works currently.

import std.algorithm;
import dproto.dproto;

mixin ProtocolBufferFromString!"
    message Msg
    {
        repeated uint32 aggr = 1;
    }
";

void main()
{
    Msg msg;
    msg.aggr = [1, 2, 3, 4, 5];
    // msg.aggr = msg.aggr.remove(0);
    msg.aggr = msg.aggr.opGet().remove(0);
    assert(msg.aggr == [2, 3, 4, 5]);
}

Error message with the commented line instead of the noncommented line:

Compiling...
source\app.d(15): Error: template std.algorithm.remove does not match any function template declaration.
 Candidates are:
c:\D\dmd2\windows\bin\..\..\src\phobos\std\algorithm.d(7562):        std.algorithm.remove(SwapStrategy s
 = SwapStrategy.stable, Range, Offset...)(Range range, Offset offset) if (s != SwapStrategy.stable && isBidirectionalRange!Range && hasLength!Range && Offset.length >= 1)
c:\D\dmd2\windows\bin\..\..\src\phobos\std\algorithm.d(7630):        std.algorithm.remove(SwapStrategy s
 = SwapStrategy.stable, Range, Offset...)(Range range, Offset offset) if (s == SwapStrategy.stable && isForwardRange!Range && Offset.length >= 1)
c:\D\dmd2\windows\bin\..\..\src\phobos\std\algorithm.d(7742):        std.algorithm.remove(alias pred, SwapStrategy s = SwapStrategy.stable, Range)(Range range) if (isBidirectionalRange!Range)
source\app.d(15): Error: template std.algorithm.remove(SwapStrategy s = SwapStrategy.stable, Range, Offset...)(Range range, Offset offset) if (s != SwapStrategy.stable && isBidirectionalRange!Range && hasLength!Range && Offset.length >= 1) cannot deduce template function from argument types !()(RepeatedBuffer!(1, "uint32", uint, false, false), int)
Error executing command run: DMD compile run failed with exit code 1

strange thing: assign changes an arguments

I am sligtly changed example with file (added multiple phones):

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  repeated PhoneNumber phone = 4;
}

enum PhoneType {
    MOBILE = 0;
    HOME = 0;
    WORK = 2;
}

message PhoneNumber { 
    required string number = 1;
    optional PhoneType type = 2 [default = MOBILE];
}
import std.stdio;
import dproto.dproto;

mixin ProtocolBuffer!"person.proto";


int main()
{
    PhoneNumber phone;
    phone.number = "45345";

    Person person;
    person.name = "John Doe";
    person.id = 1234;
    person.email = "[email protected]";
    person.phone ~= phone; // look here!
    person.phone ~= phone; // look here!

    ubyte[] serializedObject = person.serialize();

    Person person2 = Person(serializedObject);
    writeln("Name: ", person2.name);
    writeln("E-mail: ", person2.email);
    writeln("Phones: ", phone); // but now it contains two different values
    return 0;
}
$ dub
Target dproto 1.2.3 is up to date. Use --force to rebuild.
Building dproto_simple_file ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dproto_simple_file 
Name: John Doe
E-mail: [email protected]
Phones: PhoneNumber("45345", 0) # <====== here

errors 1.2.0

~/.dub/packages/dproto-1.1.1/examples/simple % dub
Target dproto 1.1.1 is up to date. Use --force to rebuild.
Building dproto_simple ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dproto_simple
Name: John Doe
E-mail: [email protected]

but

:~/.dub/packages/dproto-1.2.0/examples/simple % dub
WARNING: A deprecated branch based version specification is used for the dependency dproto. Please use numbered versions instead. Also note that you can still use the dub.selections.json file to override a certain dependency to use a branch instead.
Target dproto ~master is up to date. Use --force to rebuild.
Building dproto_simple ~master configuration "application", build type debug.
Compiling using dmd...
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(49): Error: template instance appender!(ubyte[]) template 'appender' is not defined
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(73): Error: template instance appender!(ubyte[]) template 'appender' is not defined
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(51): Error: template instance isOutputRange!(R, ubyte) template 'isOutputRange' is not defined
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(51): Error: template instance isOutputRange!(R, ubyte) template 'isOutputRange' is not defined
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/buffers.d(317): Error: template instance dproto.buffers.RepeatedBuffer!(4LU, "PhoneNumber", PhoneNumber, false, false).RepeatedBuffer.serializeTo!(Appender!(ubyte[], ubyte)) error instantiating
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(72):        instantiated from here: RepeatedBuffer!(4LU, "PhoneNumber", PhoneNumber, false, false)
../../../../../../../../home/igor/.dub/packages/dproto-master/import/dproto/dproto.d-mixin-43(102): Error: template instance isInputRange!R template 'isInputRange' is not defined
FAIL .dub/build/application-debug-posix.freebsd-x86_64-dmd_2066-D58341A42DC7432308F148A8247E9EF7/ dproto_simple executable
Error executing command run:
dmd failed with exit code 1.

Error with uint32: cannot modify const expression src

This code causes complilation errors on DMD 2.067.1 (tested only on this version)

mixin ProtocolBufferFromString!"   message Person {   required uint32 id = 1;  }  ";
\dproto-1.3.0\import\dproto\serialize.d(242): Error: cannot modify const expression src
\dproto-1.3.0\import\dproto\serialize.d(433): Error: template instance dproto.serialize.toVarint!(Appender!(ubyte[]), const(uint)) error instantiating
\dproto-1.3.0\import\dproto\buffers.d(214):        instantiated from here: writeProto!("uint32", Appender!(ubyte[]))
\dproto-1.3.0\import\dproto\dproto.d-mixin-38(43):        instantiated from here: serializeTo!(Appender!(ubyte[]))
\dproto-1.3.0\import\dproto\dproto.d-mixin-38(41):        instantiated from here: serializeTo!(Appender!(ubyte[]))
\dproto-1.3.0\import\dproto\dproto.d-mixin-38(39): Error: template instance dproto.buffers.RequiredBuffer!(1LU, "uint32", uint, false) error instantiating

slicing problem when assigning value

"pb" is struct generated by dproto from .proto file:

alias PubKey = ubyte[33];

PubKey pubKey(in PubKey pk) { pb.pubKey = pk /*.dup*/; return pk; }
source/ecdsa.d(28): Error: cannot implicitly convert expression (pk) of type const(ubyte[33]) to RequiredBuffer!(2LU, "bytes", ubyte[], false)

But if "pk" isn't const and no .dup has been used pb.pubKey will contain wrong value. I.e., dproto wants mutable value, prepared individually for it? I do not understand what's the problem.

(copied from #51 (comment))

Comments in proto file

I noticed while trying to debug #43 that there's an issue with comments not properly up showing when using dprotoc

Extension support

Parser should also handle extensions properly

This will involve searching for known structures that match the given name...

enum Default Values aren't standard conform

I had to rearrange File 1 to File 2:

//  File 1 (https://developers.google.com/protocol-buffers/docs/overview):

message Person {

    required string name = 1;
    required int32 id = 2;
    optional string email = 3;

    enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
    }

    message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
    }

    repeated PhoneNumber phone = 4;
}
// File 2

enum PhoneType {
    MOBILE = 0;
    HOME = 0;
    WORK = 2;
}

message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;

    message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = PhoneType.HOME];
    }

    repeated PhoneNumber phone = 4;
}

Ambiguous definition of `dproto.serialize.readProto` for "sint32" and "sint64"

../../.dub/packages/dproto-2.0.1/dproto/import/dproto/attributes.d(186,27): Error: dproto.serialize.readProto called with argument types (ubyte[]) matches both:
../../.dub/packages/dproto-2.0.1/dproto/import/dproto/serialize.d(396,12):     dproto.serialize.readProto!("sint64", ubyte[]).readProto(ref ubyte[] src)
and:
../../.dub/packages/dproto-2.0.1/dproto/import/dproto/serialize.d(403,12):     dproto.serialize.readProto!("sint64", ubyte[]).readProto(ref ubyte[] src)
../../.dub/packages/dproto-2.0.1/dproto/import/dproto/attributes.d(173,31): Error: template instance dproto.attributes.putSingleProtoVal!("sint64", long, ubyte[]) error instantiating
../../.dub/packages/dproto-2.0.1/dproto/import/dproto/attributes.d(89,28):        instantiated from here: putProtoVal!("sint64", long, ubyte[])
../../.dub/packages/dproto-2.0.1/dproto/import/dproto/attributes.d(54,14):        instantiated from here: deserialize!(ubyte[])
src/dosmpbf/proto.d(254,28):        instantiated from here: __ctor!(ubyte[])
dmd failed with exit code 1.

Sample Protocol Buffer:

message HeaderBBox {
    required sint64 left = 1;
    required sint64 right = 2;
    required sint64 top = 3;
    required sint64 bottom = 4;
}

Example usage:

HeaderBBox headerBBox;

headerBBox.left = 10;
headerBBox.right = 5;
headerBBox.top = -32;
headerBBox.bottom = -24;

auto hbb = headerBBox.serialize();
headerBBox = HeaderBBox(hbb);  // Error here

assert(headerBBox.left == 10);
assert(headerBBox.right == 5);
assert(headerBBox.top == -32);
assert(headerBBox.bottom == -24);

Problem function definitions in import/dproto/serialize.d:

BuffType!T readProto(string T, R)(ref R src)
    if(isProtoInputRange!R && (T == "sint32" || T == "sint64"))
{
    return src.readVarint().fromZigZag().to!(BuffType!T)();
}

BuffType!T readProto(string T, R)(ref R src)
    if(isProtoInputRange!R && T.msgType == "int32".msgType)
{
    return src.readVarint().to!(BuffType!T)();
}

questions

I have a few questions.

enum s=message Foo {...};

//how to use that?
ProtocolBuffer!s;

//for this:
mixin(ProtocolBufferString!s)
inspecting code produced with writeln shows stuff like: /// @todo: Figure out how to handle this stuff and other comments.
so my question is: how ready is it? what's done and left to be be done?

I couldn't see any test examples?

do you handle Importing Definitions? (cf https://developers.google.com/protocol-buffers/docs/proto: importing definitions)

enum within a Message in *.proto doesn't work

I had to rearrange File 1 to File 2:

//  File 1 (https://developers.google.com/protocol-buffers/docs/overview):

message Person {

    required string name = 1;
    required int32 id = 2;
    optional string email = 3;

    enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
    }

    message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
    }

    repeated PhoneNumber phone = 4;
}
// File 2

enum PhoneType {
    MOBILE = 0;
    HOME = 0;
    WORK = 2;
}

message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;

    message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = PhoneType.HOME];
    }

    repeated PhoneNumber phone = 4;
}

dmd 2.067: all versions since 1.1.0 are broken for this compiler?

Building dproto 1.2.1 configuration "library", build type unittest.
Running dmd...
../.dub/packages/dproto-1.2.1/import/dproto/buffers.d(253): Error: cannot implicitly convert expression ([]) of type Person[] to inout(Person[])
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d-mixin-45(107): Error: template instance dproto.buffers.RepeatedBuffer!(1LU, "Person", Person, false, false) error instantiating
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d(278): Error: mixin dproto.dproto.__unittestL276_13.ProtocolBufferFromString!"\x0a\x09\x09message Person {\x0a\x09\x09  required string name = 1;\x0a\x09\x09  required int32 id = 2;\x0a\x09\x09  optional string email = 3;\x0a\x0a\x09\x09  enum PhoneType {\x0a\x09\x09\x09MOBILE = 0;\x0a\x09\x09\x09HOME = 0;\x0a\x09\x09\x09WORK = 2;\x0a\x09\x09  }\x0a\x0a\x09\x09  message PhoneNumber {\x0a\x09\x09\x09required string number = 1;\x0a\x09\x09\x09optional PhoneType type = 2 [default = HOME];\x0a\x09\x09  }\x0a\x0a\x09\x09  repeated PhoneNumber phone = 4;\x0a\x09\x09}\x0a\x0a\x09\x09message AddressBook {\x0a\x09\x09  repeated Person person = 1;\x0a\x09\x09}\x0a\x09" error instantiating
../.dub/packages/dproto-1.2.1/import/dproto/buffers.d(253): Error: cannot implicitly convert expression ([]) of type Person[] to inout(Person[])
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d-mixin-45(107): Error: template instance dproto.buffers.RepeatedBuffer!(1LU, "Person", Person, false, false) error instantiating
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d(343): Error: mixin dproto.dproto.__unittestL341_14.ProtocolBufferFromString!"\x0aenum PhoneType {\x0a  MOBILE = 0;\x0a  HOME = 0;\x0a  WORK = 2;\x0a}\x0a\x0amessage Person {\x0a  required string name = 1;\x0a  required int32 id = 2;\x0a  optional string email = 3;\x0a\x0a  message PhoneNumber {\x0a\x09required string number = 1;\x0a\x09optional PhoneType type = 2 [default = HOME];\x0a  }\x0a\x0a  repeated PhoneNumber phone = 4;\x0a}\x0a\x0amessage AddressBook {\x0a  repeated Person person = 1;\x0a}\x0a\x09" error instantiating
../.dub/packages/dproto-1.2.1/import/dproto/buffers.d(253): Error: cannot implicitly convert expression ([]) of type Person[] to inout(Person[])
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d-mixin-45(107): Error: template instance dproto.buffers.RepeatedBuffer!(1LU, "Person", Person, false, false) error instantiating
../.dub/packages/dproto-1.2.1/import/dproto/dproto.d(411): Error: mixin dproto.dproto.__unittestL409_15.ProtocolBufferFromString!"\x0amessage Person {\x0a  required string name = 1;\x0a  required int32 id = 2;\x0a  optional string email = 3;\x0a\x0a  enum PhoneType {\x0a\x09MOBILE = 0;\x0a\x09HOME = 0;\x0a\x09WORK = 2;\x0a  }\x0a\x0a  message PhoneNumber {\x0a\x09required string number = 1;\x0a\x09optional PhoneType type = 2 [default = HOME];\x0a  }\x0a\x0a  repeated PhoneNumber phone = 4;\x0a}\x0a\x0amessage AddressBook {\x0a  repeated Person person = 1;\x0a}\x0a\x09" error instantiating
FAIL ../.dub/packages/dproto-1.2.1/.dub/build/library-unittest-linux.posix-x86_64-dmd_2067-DFDA0ACA4CA4BE7C5B817A00EE258E31/ dproto staticLibrary
Error executing command test: dmd failed with exit code 1.

RPC

RPC support is not currently functional, but the code is started.

Default value error on optional field

If you try and set a default value on an optional field, the compiler spits out an error:

mixin ProtocolBufferFromString!"
    message TestMessage {
        optional string foo = 1 [default = \"\"];
    }
";
../../../.dub/packages/dproto-1.1.0/import/dproto/dproto.d-mixin-41(44): Error: RealType.init is used as a type
../../../.dub/packages/dproto-1.1.0/import/dproto/dproto.d-mixin-41(44):        while looking for match for OptionalBuffer!(1, "string", string, false)

Setting fields on nested messages does not work in 1.3.2

I found a bug with 1.3.2 that is fixed in master. It might be worth adding the test though.

unittest
{
    mixin ProtocolBufferFromString!"
        message Stats {
            optional int32 agility = 1;
            optional int32 stamina = 2;
        }
        message Character {
            optional string name = 1;
            optional Stats stats = 2;
        }
        message Account {
            optional string owner = 1;
            optional Character main = 2;
        }
    ";
    const int agility = 200;
    auto acct = Account();
    auto main = Character();
    main.name = "Hogan";
    main.stats.agility = agility;
    acct.main = main;
    auto ser = acct.serialize();
    Account acct_rx;
    acct_rx.deserialize(ser);
    assert(acct_rx.main.stats.agility == agility, format("Expected %d, got %d", agility, acct_rx.main.stats.agility));
}

Runtime conversion

There should be a runtime version of ProtocolBufferString made available.

Forward referencing fails

I have received reports that attempting to forward-reference message types fails, though most other implementations support it.

would be nice to show examples derived from real use cases

would be nice to show examples derived from real use cases to see where it works and where it doesn't yet work;

eg with no imports:
https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto

more complex eg with imports:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/graph.proto

eg: loading from prototxt text format:
https://github.com/BVLC/caffe/blob/master/models/bvlc_reference_caffenet/solver.prototxt

eg: loading from proto wire format:
...

opGet

opGet should be replaced/enhanced with opCast support

errors 1.2.0 #2

import std.stdio;
import dproto.dproto;

mixin ProtocolBufferFromString!"
    message Person {
      required string name = 1;
      required int32 id = 2;
      optional string email = 3;

      enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
      }

      message PhoneNumber {
        required string number = 1;
        optional PhoneType type = 2 [default = HOME];
      }

      repeated PhoneNumber phone = 4;
    }
";

int main()
{
    Person person;
    person.name = "John Doe";
    person.id = 1234;
    person.email = "[email protected]";

    Person.PhoneNumber pn1;
    pn1.number = "0123456789";
    pn1.type = PhoneType.WORK;

    Person.PhoneNumber pn2;
    pn2.number = "0123456789";

    t.phone = [pn1, pn2];

    ubyte[] serializedObject = person.serialize();

    Person person2 = Person(serializedObject);
    writeln("Name: ", person2.name);
    writeln("E-mail: ", person2.email);
    return 0;
}

dub

Target dproto 1.2.1 is up to date. Use --force to rebuild.
Building dproto_simple ~master configuration "application", build type debug.
Compiling using dmd...
source/app.d(43): Error: undefined identifier PhoneType
source/app.d(48): Error: undefined identifier t
FAIL .dub/build/application-debug-linux.posix-x86_64-dmd_2066-D58341A42DC7432308F148A8247E9EF7/ dproto_simple executable
Error executing command run:
dmd failed with exit code 1.

Public import

protoc 2.5.0 introduced "public import". This is how dproto currently imports, which means that the import code should be changed to introduce module scope.

Parser

Create a parser that uses CTFE-able string operations to create string class definitions from a .proto file.

Range support

Deserializers should handle any arbitrary input range of ubyte, not just ubyte[].

By the same reasoning, serializers should handle output ranges.

Using [packed=true] option with repeated fields causes range violations

I've encountered an unpleasant issue with the [packed=true] option for repeated protobuf fields, as described here:
https://developers.google.com/protocol-buffers/docs/proto#specifying-field-rules

The following minimal example should illustrate the issue:

import dproto.dproto;

mixin ProtocolBufferFromString!`
    message Foo {
        repeated uint32 arr = 1 [packed=true];
    }
`;

void main()
{
    Foo foo;
    foo.arr = [1u, 2, 3, 4, 5];

    auto serialized_foo = foo.serialize();

    auto foo2 = Foo(serialized_foo);  // FAILS
}

When I run this, I get an error:

core.exception.RangeError@/usr/include/d/std/bitmanip.d(2964): Range violation

Removing the [packed=true] option from the arr field allows the above program to run without error.

In a more complex protobuf setup, I have encountered a different error that seems to result from the same issue:

core.exception.AssertError@/usr/include/d/std/range/primitives.d(2200): Attempting to fetch the front of an empty array of ubyte

but I have been unable for now to reduce this to a friendly minimal example (I will keep trying to do so, and report back here if I succeed). It seems to be triggered from here:

ret ~= src.front;

In either case, the deserialization problem vanishes if [packed=true] is not applied to repeated fields.

For reference, I'm using dproto 2.0.1 on Ubuntu 16.04, compiling with ldc 0.17.1.

Release 1.2.5

There are a good amount of updates in the current master, let's figure out how to get it ready for a 1.2.5 release.

Compiler errors on invalid proto syntax

Full code:

import dproto;

mixin ProtocolBufferFromString!q{
    message One {
        required string a;
        required int32 b;
    }
};

void main() {
    One one  = {
        a: "foo",
        b: 123,
    };
    auto buf = one.serialize();

    auto oneCopy = One(buf);
    assert(oneCopy == one);
}

Output:

> dub build
Target dproto 1.3.1 is up to date. Use --force to rebuild.
Building cer-test ~master configuration "application", build type debug.
Compiling using dmd...
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(283): Error: value of 'this' is not known at compile time
/usr/include/dmd/phobos/std/exception.d(524):        called from here: ex()
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(283):        called from here: enforce(cast(int)this.readChar() == 61, delegate Throwable() => this.unexpected("expected '='"))
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(144):        called from here: this.readField(label, type)
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(188):        called from here: this.readDeclaration(ret)
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(105):        called from here: this.readMessage()
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(56):        called from here: this.readDeclaration(ret)
../.dub/packages/dproto-1.3.1/import/dproto/parse.d(556):        called from here: ProtoSchemaParser(null, null, 0, 0, 0).this(name_, data_).readProtoPackage()
../.dub/packages/dproto-1.3.1/import/dproto/dproto.d(38):        called from here: ParseProtoSchema("<none>", "\x0a\x09message One {\x0a\x09\x09required string a;\x0a\x09\x09required int32 b;\x0a\x09}\x0a")
../.dub/packages/dproto-1.3.1/import/dproto/dproto.d(38): Error: argument to mixin must be a string, not (ParseProtoSchema("<none>", "\x0a\x09message One {\x0a\x09\x09required string a;\x0a\x09\x09required int32 b;\x0a\x09}\x0a").toD()) of type string
source/app.d(4): Error: mixin app.ProtocolBufferFromString!"\x0a\x09message One {\x0a\x09\x09required string a;\x0a\x09\x09required int32 b;\x0a\x09}\x0a" error instantiating
FAIL .dub/build/application-debug-linux.posix-x86_64-dmd_2067-060CA0ACE253FCF806DEFE03AEAAD9E7/ cer-test executable
Error executing command build:
dmd failed with exit code 1.

> dmd --version
DMD64 D Compiler v2.067.1
Copyright (c) 1999-2014 by Digital Mars written by Walter Bright

Error: no identifier for declarator BuffType!"int32"

../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(116,18): Error: no identifier for declarator BuffType!"int32"
../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(116,48): Error: semicolon expected
../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(116,49): Error: declaration expected, not '('
../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(137,20): Error: no identifier for declarator BuffType!"int32"[]
../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(137,27): Error: (condition) expected following version
../../.dub/packages/dproto-2.0.0/import/dproto/dproto.d-mixin-27(137,27): Error: declaration expected following attribute, not ';'
osmproto/package.d(6,1): Error: mixin osmproto.ProtocolBuffer!"osmformat.proto" error instantiating

proto file:

option optimize_for = LITE_RUNTIME;
option java_package = "crosby.binary";
package osmproto.osmformat;


//////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////


message HeaderBlock {
  optional HeaderBBox bbox = 1;
  repeated string required_features = 4;
  repeated string optional_features = 5;

  optional string writingprogram = 16; 
  optional string source = 17; // From the bbox field.


  // replication timestamp, expressed in seconds since the epoch, 
  // otherwise the same value as in the "timestamp=..." field
  // in the state.txt file used by Osmosis
  optional int64 osmosis_replication_timestamp = 32;

  // replication sequence number (sequenceNumber in state.txt)
  optional int64 osmosis_replication_sequence_number = 33;

  // replication base URL (from Osmosis' configuration.txt file)
  optional string osmosis_replication_base_url = 34;
}



message HeaderBBox {
   required sint64 left = 1;
   required sint64 right = 2;
   required sint64 top = 3;
   required sint64 bottom = 4;
}


///////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////


message PrimitiveBlock {
  required StringTable stringtable = 1;
  repeated PrimitiveGroup primitivegroup = 2;

  // Granularity, units of nanodegrees, used to store coordinates in this block
  optional int32 granularity = 17 [default=100]; 
  // Offset value between the output coordinates coordinates and the granularity grid in unites of nanodegrees.
  optional int64 lat_offset = 19 [default=0];
  optional int64 lon_offset = 20 [default=0]; 

// Granularity of dates, normally represented in units of milliseconds since the 1970 epoch.
  optional int32 date_granularity = 18 [default=1000]; 


  // Proposed extension:
  //optional BBox bbox = XX;
}

// Group of OSMPrimitives. All primitives in a group must be the same type.
message PrimitiveGroup {
  repeated Node     nodes = 1;
  optional DenseNodes dense = 2;
  repeated Way      ways = 3;
  repeated Relation relations = 4;
  repeated ChangeSet changesets = 5;
}


message StringTable {
   repeated bytes s = 1;
}

message Info {
   optional int32 version = 1 [default = -1];
   optional int64 timestamp = 2;
   optional int64 changeset = 3;
   optional int32 uid = 4;
   optional uint32 user_sid = 5; // String IDs

   // The visible flag is used to store history information. It indicates that
   // the current object version has been created by a delete operation on the
   // OSM API.
   // When a writer sets this flag, it MUST add a required_features tag with
   // value "HistoricalInformation" to the HeaderBlock.
   // If this flag is not available for some object it MUST be assumed to be
   // true if the file has the required_features tag "HistoricalInformation"
   // set.
   optional bool visible = 6;
}

message DenseInfo {
   repeated int32 version = 1 [packed = true]; 
   repeated sint64 timestamp = 2 [packed = true]; // DELTA coded
   repeated sint64 changeset = 3 [packed = true]; // DELTA coded
   repeated sint32 uid = 4 [packed = true]; // DELTA coded
   repeated sint32 user_sid = 5 [packed = true]; // String IDs for usernames. DELTA coded

   // The visible flag is used to store history information. It indicates that
   // the current object version has been created by a delete operation on the
   // OSM API.
   // When a writer sets this flag, it MUST add a required_features tag with
   // value "HistoricalInformation" to the HeaderBlock.
   // If this flag is not available for some object it MUST be assumed to be
   // true if the file has the required_features tag "HistoricalInformation"
   // set.
   repeated bool visible = 6 [packed = true];
}


// THIS IS STUB DESIGN FOR CHANGESETS. NOT USED RIGHT NOW.
// TODO:    REMOVE THIS?
message ChangeSet {
   required int64 id = 1;
//   
//   // Parallel arrays.
//   repeated uint32 keys = 2 [packed = true]; // String IDs.
//   repeated uint32 vals = 3 [packed = true]; // String IDs.
//
//   optional Info info = 4;

//   optional int64 created_at = 8;
//   optional int64 closetime_delta = 9;
//   optional bool open = 10;
//   optional HeaderBBox bbox = 11;
}


message Node {
   required sint64 id = 1;
   // Parallel arrays.
   repeated uint32 keys = 2 [packed = true]; // String IDs.
   repeated uint32 vals = 3 [packed = true]; // String IDs.

   optional Info info = 4; // May be omitted in omitmeta

   required sint64 lat = 8;
   required sint64 lon = 9;
}


message DenseNodes {
   repeated sint64 id = 1 [packed = true]; // DELTA coded

   //repeated Info info = 4;
   optional DenseInfo denseinfo = 5;

   repeated sint64 lat = 8 [packed = true]; // DELTA coded
   repeated sint64 lon = 9 [packed = true]; // DELTA coded

   // Special packing of keys and vals into one array. May be empty if all nodes in this block are tagless.
   repeated int32 keys_vals = 10 [packed = true]; 
}


message Way {
   required int64 id = 1;
   // Parallel arrays.
   repeated uint32 keys = 2 [packed = true];
   repeated uint32 vals = 3 [packed = true];

   optional Info info = 4;

   repeated sint64 refs = 8 [packed = true];  // DELTA coded
}

message Relation {
  enum MemberType {
    NODE = 0;
    WAY = 1;
    RELATION = 2;
  } 
   required int64 id = 1;

   // Parallel arrays.
   repeated uint32 keys = 2 [packed = true];
   repeated uint32 vals = 3 [packed = true];

   optional Info info = 4;

   // Parallel arrays
   repeated int32 roles_sid = 8 [packed = true];
   repeated sint64 memids = 9 [packed = true]; // DELTA encoded
   repeated MemberType types = 10 [packed = true];
}

Unit tests

Unit tests should be created for most functions

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.