swi-prolog/contrib-protobufs

An interface to Google Protocol Buffers (protobuf)

Google's Protocol Buffers

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once. This takes the form of a template that describes the data structure. You use this template to encode and decode your data structure into wire-streams that may be sent-to or read-from your peers. The underlying wire stream is platform independent, lossless, and may be used to interwork with a variety of languages and systems regardless of word size or endianness.

This document was produced using PlDoc, with sources found in protobufs.pl and protobufs_overview.md. There is a simple example (assuming you've installed swi-prolog-doc) in /usr/share/swi-prolog/doc/packages/examples/protobufs/interop/addressbook.pl:

protoc -I/usr/share/doc/protobuf-compiler/examples -I/usr/include --swipl_out=/tmp/ \
    --plugin=protoc-gen-swipl=/usr/lib/swi-prolog/library/protobufs/protoc-gen-swipl \
    /usr/share/swi-prolog/doc/packages/examples/protobufs/interop/addressbook.pl

This generates /tmp/addressbook_pb.pl … you should change the --swipl-out= specification to be the directory you want (e.g., interop).

Note that protoc requires an -I item that encompasses the .proto file, so this also works:

protoc -I -I/usr/include --swipl_out=/tmp/ \
    -I/usr/share/swi-prolog/doc/packages/examples/protobufs/interop \
    --plugin=protoc-gen-swipl=/usr/lib/swi-prolog/library/protobufs/protoc-gen-swipl \
    addressbook.proto

and, if /usr/lib/swi-prolog/library/protobufs is in your PATH, you can leave out the --plugin option:

EXPORT PATH=/usr/lib/swi-prolog/library/protobufs:$PATH
protoc -I/usr/share/swi-prolog/doc/packages/examples/protobufs/interop -I/usr/include --swipl_out=/tmp/ addressbook.proto

It’s possible that your installation puts files in a different place; you can find the addressbook example with:

dpkg -L protobuf-compiler | grep addressbook

To test this, run the following:

swipl -g test_write -g test_write -g test_read -t halt addressbook.pl  # outputs to addressbook.wire
protoc --decode=tutorial.AddressBook addressbook.proto <addressbook.wire

@see https://developers.google.com/protocol-buffers @author Jeffrey Rosenwald ([email protected]), Peter Ludemann ([email protected]) @license BSD-2

contrib-protobufs's Issues

int64_zigzag/2, uint32_codes, etc fail/error in a confusing way

This applies to all the predicates defined in protobufs.c

If an argument is instantiated and not of the appropriate type (integer, list(integer), etc., sometimes the predicate quietly fails, sometimes gives an incorrect "instantiation error", sometimes gives an appropriate error.

Implementation notes:
See PL_term_type() and return values PL_VARIABLE, PL_INTEGER.

Need to verify all instantiation errors and also add checks before some unification calls.

Folder names

Dear Peter,

(Not urgent at all. Not important either.)

As I wrote you in an email, I am currently trying to build an embedded swipl as an R package. The R build system complains that the folder names are too long. This is the output of R CMD build:

$ R CMD build rolog
    checking for file ‘rolog/DESCRIPTION’ … OK
    preparing ‘rolog’:
    checking DESCRIPTION meta-information … OK
    cleaning src
    checking for LF line-endings in source and make files and shell scripts
    checking for empty or unneeded directories
    building ‘rolog_1.0.tar.gz’

Warnung in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,
storing paths of more than 100 bytes is not portable:

‘rolog/src/swipl-devel/packages/protobufs/bootstrap/protoc_gen_prolog_pb/google/protobuf/compiler/plugin_pb.pl’

Warnung in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,
storing paths of more than 100 bytes is not portable:

‘rolog/src/swipl-devel/packages/protobufs/bootstrap/protoc_gen_prolog_pb/google/protobuf/descriptor_pb.pl’

Thank you for considering it, I know this is a strange request in 2021, and I am seriously wondering which systems they have in mind for portability.

Best wishes,

Matthias

Dependencies on GMP

Need to verify that none of the Prolog code depends on GMP.

Conditional running of some tests was done by commit 7607406 but that might have obscured some usages of GMP within the code.

repeated/packed negative enum fails

It's uncommon, but a negative enum is allowed by the spec (because an enum is represented as a varint without zigzag encoding, it occupies the maximum length of 10 bytes).

test.proto has NegEnum = -1 but its use in tests is commented out.

Are you interested in Proto parser written in Prolog?

This isn't a bug report. Just as an exercise, I wrote Protobuf definition parser in Prolog: https://github.com/wvxvw/protobuf-prolog-parser . It's not perfect (I haven't tested it extensively), and could certainly have more features, s.a. better error reporting. But, if you are interested, I can work on it to add missing features / fix bugs.

Oneof is not properly supported (also map?)

Currently, the oneof option is skipped, resulting in incorrect processing for the overlaid fields.

Need test cases (the "golden" message isn't sufficient).

This probably also applies to map.

integer_zigzag can't handle large values

integer_zigzag(A, 18446744073709551600).

ERROR: Arguments are not sufficiently instantiated

(The value is 0xfffffffffffffff0.)

x86 test failure

Seems there are some type issues wrt. 32 bit machines. See https://gitlab.alpinelinux.org/brebs/aports/-/jobs/521645/raw, copying the relevant section:

ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:556:
	test packed1: received error: protobufs:uint32_int32/2: Domain error: `not_less_than_zero' expected, found `-1643999741'
..
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:601:
	test not_packed_repeated2_gmp: received error: protobufs:uint64_int64/2: Type error: `integer' expected, found `12078126446000735008' (an integer)
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:611:
	test packed_repeated: received error: protobufs:uint32_int32/2: Domain error: `not_less_than_zero' expected, found `-1643999741'
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:631:
	test packed_repeated2: received error: protobufs:uint32_int32/2: Domain error: `not_less_than_zero' expected, found `-1643999741'
.
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:667:
	test packed_and_unpacked_repeated_gmp: received error: protobufs:uint32_int32/2: Domain error: `not_less_than_zero' expected, found `-1643999741'
. done
% PL-Unit: protobuf_segment_convert ......... done
% PL-Unit: codes 
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:835:
	test uint32_codes: assertion failed
	Assertion: -1==4294967295
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:835:
	test uint32_codes: assertion failed
	Assertion: -2147483648==2147483648
A....
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:856:
	test uint64_codes: received error: protobufs:uint64_codes/2: Type error: `integer' expected, found `18446744073709551615' (an integer)
....
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:892:
	test int64_float64: received error: protobufs:uint64_int64/2: Type error: `integer' expected, found `13830554455654793216' (an integer)
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:913:
	test int32_float32: received error: protobufs:uint32_int32/2: Domain error: `not_less_than_zero' expected, found `-1082130432'
 done
% PL-Unit: zigzag 
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:943:
	test zigzag: received error: protobufs:int64_zigzag/2: Type error: `integer' expected, found `18446744073709551614' (an integer)
..
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:964:
	test zigzag_e3: wrong error
ERROR:     Expected: error(representation_error(uint64_t),context(protobufs:int64_zigzag/2,A))
ERROR:     Got:      error(type_error(integer,18446744073709551616),context(protobufs:int64_zigzag/2,B))
.. done
% PL-Unit: coerce 
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:987:
	test coerce: received error: protobufs:uint64_int64/2: Type error: `integer' expected, found `18446744073709551615' (an integer)
..
ERROR: /builds/brebs/aports/testing/swi-prolog/src/swipl-8.5.0/packages/protobufs/test_protobufs.pl:1006:
	test coerce_e3: wrong error
ERROR:     Expected: error(representation_error(uint64_t),context(protobufs:uint64_int64/2,A))
ERROR:     Got:      error(type_error(integer,18446744073709551616),context(protobufs:uint64_int64/2,B))
..... done

protoc-gen-swipl should be in same directory as swipl executable

Currently, to run protoc for Prolog, the directory /usr/lib/swi-prolog/library/protobufs needs to be in the $PATH. It would be better if protoc-gen-swipl were in the same directory as the swipl executable (e.g., /usr/lib/swipl).

This could be done with a symlink from /usr/lib/protoc-gen-swipl to /usr/lib/swi-prolog/library/protobufs/protoc-gen-swipl -- how would this be specified in protobufs/CMakeLists.txt?

protoc plugin handling of imports

Current protoc behavior requires importing all the *_pb files that are used both directly and indirectly. This is inconvenient and should be fixed.

Protoc doesn't process all the imported .proto files, although it does make them available in the data passed to the plugin (via plugin.proto). This means that if you have x.proto importing y.proto, you need to explicitly run protoc on both x.proto and y.proto.

The documentation is less clear about whether code that uses x_pb.pl needs to also mention y_pb.pl. For user convenience, x_pb.pl should reference y_pb.pl. This means we could have duplicate meta-proto facts from y_pb.pl (however, the module loader might prevent this).

The protoc documentation is a bit vague (and I haven't explored fully what the Python (and C++, Java) plugins do. For Python, it appears that for x.proto that imports y.proto, the generated code can handle messages defined in y.proto without the program needing an 'import y_pb2"

"Map" has no special support

"Map" has no special support. You can do "map" fields, but it's a bit klunky; instead of something like this:

'.MapMessage'{number_ints: _{one:1, two:2}}

you need to do

'.MapMessage'{number_ints:[
    '.MapMessage.NumberIntsEntry'{key:"two",value:2},
    '.MapMessage.NumberIntsEntry'{key:"one",value:1}]}

There are a few reasons for not supporting "map" right now. The biggest one is: what representation should be done for the map?

• dict
• library(pairs) - key-value list
• library(assoc)
• library(rbtrees)
• others?

"Dict" only allows atoms and integers for the keys (and the current library(protobufs) code doesn't convert between atoms and string). The other choices have different APIs and performance characteristics. Also, library(pairs) raises the question of ordering and whether it's significant.

For the short term, the best solution is probably a "helper" predicate that translates between the various representations and the protobuf "map". This shouldn't be difficult to write, but I don't feel like doing it until it's actually needed.