bivas/protobuf-java-format

Provide serialization and de-serialization of different formats based on Google’s protobuf Message. Enables overriding the default (byte array) output to text based formats such as XML, JSON and HTML.

Fork from http://code.google.com/p/protobuf-java-format/

Description

Provide serialization and de-serialization of different formats based on Google’s protobuf Message. Enables overriding the default (byte array) output to text based formats such as XML, JSON and HTML.

##Example For XML output, use XmlFormat

Message someProto = SomeProto.getDefaultInstance();
XmlFormat xmlFormat = new XmlFormat();
String asXml = xmlFormat.printToString(someProto);

For XML input, use XmlFormat

Message.Builder builder = SomeProto.newBuilder();
String asXml = _load xml document from a source_;
XmlFormat xmlFormat = new XmlFormat();
xmlFormat.merge(asXml, builder);

For Json output, use JsonFormat

Message someProto = SomeProto.getDefaultInstance();
JsonFormat jsonFormat = new JsonFormat();
String asJson = jsonFormat.printToString(someProto);

For Json input, use JsonFormat

Message.Builder builder = SomeProto.newBuilder();
String asJson = _load json document from a source_;
JsonFormat jsonFormat = new JsonFormat();
jsonFormat.merge(asJson, builder);

For HTML output, use HtmlFormat

Message someProto = SomeProto.getDefaultInstance();
HtmlFormat htmlFormat = new HtmlFormat();
String asHtml = htmlFormat.printToString(someProto);

protobuf-java-format's Issues

JsonFormat.printToString() does not respect JsonFormat.setDefaultCharset()

Implementation of JsonFormat.printToString() (which is in ProtobufFormatter.java) is to print to a byte array using the Charset specified by JsonFormat.setDefaultCharset(), but converts the byte array back to a string using ByteArrayOutputStream.toString(), which uses Charset.defaultCharset().

Conversion from the ByteArrayOutputStream back to a string, https://github.com/bivas/protobuf-java-format/blob/master/src/main/java/com/googlecode/protobuf/format/ProtobufFormatter.java#L91, should also use the specified Charset instead of JVM default.

It is worth pointing out that conversion to bytes and back to string is not actually required if all we want to do is serialize JSON to a java.lang.String.

Publish to maven

Start publishing the project to maven central repo.

Can't Find this method printToString()

Sorry .I can't Find this method printToString()
in class com.googlecode.protobuf.format in tag 1.4
which I see In README.md
as

For Json output, use JsonFormat

Message someProto = SomeProto.getDefaultInstance();
JsonFormat jsonFormat = new JsonFormat();
String asJson = jsonFormat.printToString(someProto);

Error when parsing a valid XML file

Hi,
I am trying to parse a sample document into a Protobuf Message, using the AddressBook schema from Google examples:

Here is the document:

<AddressBook>
    <people>
        <name>John Doe</name>
        <id>42</id>
        <email>[email protected]</email>
    </people>
    <people>
        <name>Jane Doe</name>
        <id>41</id>
    </people>
</AddressBook>

Here is the code:

// All this initialization stuff is tested
InputStream inputData = XMLMapperTest.class.getResourceAsStream("/data/AddressBook_several.xml");
DynamicSchema schema = SchemaParser.parseSchema(XMLMapperTest.class.getResource("/schemas/AddressBook.desc").getPath(), false);
Descriptors.Descriptor descriptor = schema.getMessageDescriptor("AddressBook");

DynamicMessage.Builder builder = DynamicMessage.newBuilder(descriptor);

XmlFormat xmlFormat = new XmlFormat();
// Here is the instruction that raises the exception
xmlFormat.merge(inputData, StandardCharsets.UTF_8, builder);

Though, I get the following error:

com.googlecode.protobuf.format.ProtobufFormatter$ParseException: 5:21: Expected ">".

	at com.googlecode.protobuf.format.XmlFormat$Tokenizer.parseException(XmlFormat.java:619)
	at com.googlecode.protobuf.format.XmlFormat$Tokenizer.consume(XmlFormat.java:418)
	at com.googlecode.protobuf.format.XmlFormat.consumeClosingElement(XmlFormat.java:680)
	at com.googlecode.protobuf.format.XmlFormat.mergeField(XmlFormat.java:764)
	at com.googlecode.protobuf.format.XmlFormat.handleObject(XmlFormat.java:882)
	at com.googlecode.protobuf.format.XmlFormat.handleValue(XmlFormat.java:775)
	at com.googlecode.protobuf.format.XmlFormat.mergeField(XmlFormat.java:755)
	at com.googlecode.protobuf.format.XmlFormat.merge(XmlFormat.java:663)
	at com.googlecode.protobuf.format.AbstractCharBasedFormatter.merge(AbstractCharBasedFormatter.java:75)
	at com.googlecode.protobuf.format.AbstractCharBasedFormatter.merge(AbstractCharBasedFormatter.java:53)
	at com.googlecode.protobuf.format.ProtobufFormatter.merge(ProtobufFormatter.java:141)
[...]

I tried with UTF-8 and ISO-8859-1 encoding but I still get the error. Then I tried to remove the dots in the email address in my XML doc and I now parse successfully.

This is the working XML:

<AddressBook>
    <people>
        <name>John Doe</name>
        <id>42</id>
        <email>johndoe@examplecom</email>
    </people>
    <people>
        <name>Jane Doe</name>
        <id>41</id>
    </people>
</AddressBook>

If you want, I can also join the Protobuf schema if you want to try by yourself.

JsonFormat#printUnknownFIelds produce invalid JSON

This method may produce invalid json with hex numbers.
The convention on json:

http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
Numbers

A number is a sequence of decimal digits with no superfluous leading zero. It may have a preceding minus sign (U+002D). It may have a fractional part prefixed by a decimal point (U+002E).It may have an exponent, prefixed by e(U+0065) or E(U+0045) and optionally +(U+002B) or –(U+002D). The digits are the code points U+0030 through U+0039

The json produced by this method not parsed by another Json libraries. For example jackson.

HTML Format: Should Escape String

On line : https://github.com/bivas/protobuf-java-format/blob/master/src/main/java/com/googlecode/protobuf/format/HtmlFormat.java#L197

The HTML format when printing the String should do something like Commons Text StringEscapeUtils.escapeHtml4(text); to escape ampersands and such inside the text.

Support for havig attributes in XML formatter?

Hello,
Thank you for the great work. I would like to know if there is (or will be) any support for having attributes in XML elements, instead of having separate XML entities for each message field.

---

Example:
I have the following .proto file:

syntax = "proto2";

package tutorial;

option java_multiple_files = true;
option java_package = "com.example.tutorial.protos";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  required string email = 3;

  enum PhoneType {
    PHONE_TYPE_UNSPECIFIED = 0;
    PHONE_TYPE_MOBILE = 1;
    PHONE_TYPE_HOME = 2;
    PHONE_TYPE_WORK = 3;
  }

  message PhoneNumber {
    optional string number = 1;
    optional PhoneType type = 2 [default = PHONE_TYPE_HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}

When I use new XmlFormat().printToString(addressBook), the output string looks like this:

<AddressBook>
  <people>
    <name>John Doe</name>
    <id>1234</id>
    <email>[email protected]</email>
    <phones>
      <number>555-4321-9999</number>
      <type>PHONE_TYPE_HOME</type>
    </phones>
  </people>
</AddressBook>

However, I would like to have the possibility to have some of the fields as attributes:

<AddressBook>
  <people name="John Doe" id="1234" email="[email protected]">
    <phones>
      <number>555-4321-9999</number>
      <type>PHONE_TYPE_HOME</type>
    </phones>
  </people>
</AddressBook>

The second one is arguably more readable and more concise. So, is there (or will there be) any support for it in this library? Thanks!

JsonFormat.merge throw a exception when a 'message' field in json is null

message A {
//xxx
}
message B {
optional A a=0;
}

B.Builder builder = B.newBuild();
new JsonFormat().merge("{a: null}", ExtensionRegistry.getEmptyRegistry(), builder);

Will throw exception like
Exception in thread "main" com.googlecode.protobuf.format.JsonFormat$ParseException: 1:13: Expected "{"

version 1.4

Stop generating protobuf in src/test folder

sample protobuf should go in generated-test-sources; not src/test.

Escape / Unescape bytes are not perfect inverses

Escaping a ByteString (JavaFormat.escapeBytes(...)) produces a String that does not always get unescaped back to an equivalent of the original ByteString via JavaFormat.unescapeBytes(...). See pull request #27 for a test that demonstrates this.

Add gh-pages branch with documentation and refer to it from README

Next step on #23 requires some documentation on the format for JSON. This issue is to set up documentation by creating gh-pages branch and referring to it from README.md

/cc @PeteyPabPro

1.5 Maven version needed

Hi, could you please release version 1.5 to Maven, we need the 'bits' package. Thanks.

String encoding of ByteString(Bytes)

Google is using base64 encoding to convert raw bytes to JSON string https://developers.google.com/protocol-buffers/docs/proto3#json with Proto3. It would be nice if protobuf-java-format does the same

Add Travis integration.

http://docs.travis-ci.com/user/getting-started/#To-get-started-with-Travis-CI%3A

com.googlecode.protobuf.format.XmlFormat merge fails for content with URL

When the XML content contains special characters like URL the formatter fails to merge.
Is there a work around for this ?

XmlFormat xmlBinder = new XmlFormat();
xmlBinder.setDefaultCharset(StandardCharsets.UTF_8);
InputStream data = new ByteArrayInputStream(xmlView.getBytes("UTF-8"));
xmlBinder.merge(data, extensionRegistry, builder);

Input XML:


ltl.upsfreight.com/ProductsAndServices/terminals/index.aspx?qcountry=US&SvcCtrId=FON
1.0

1
Success

Upgrade to jackson 2.0

Missing 1.4 release tag

Hi,

The release tag for the version 1.4 is missing, could you add it please? The 1.4 commit is 1720434.

Thank you

Any types can't be deserialized

Tryin to deserialize a previous serialized message that contains Any types, returns an error.

XmlFormater : com.googlecode.protobuf.format.ProtobufFormatter$ParseException: 1:83: Expected ">".

JsonFormater: Protocol message contained an invalid tag (zero).

Examples refer to non-static methods

JsonFormat.printToString(...) is not static so the examples don't work.

Add some documentation to contribute to the project

It would be find if we had more documentation about how to contribute to the project, cause I cannot manage to build the source code to solve issue #44.

I installed protoc and ran mvn test but I get errors about packages not found like that:

[ERROR] /home/wsh/Documents/protobuf-java-format/target/generated-test-sources/protobuf_unittest/UnittestProto.java:[13217,44] error: package com.google.protobuf.GeneratedMessageV3 does not exist

I suppose there is some more steps to generate these files but I don't know which...

Exception on Special Characteres "//"

I'm getting errors when informing special characteres like '//' (char generated from encoded base64 file) at a proto bytes field.

protos

message InformaContestacaoCliente {
	required Contestacao contestacao = 1;
}

message Contestacao {
	repeated Anexo anexos = 1;
}

message Anexo {
    optional bytes anexo = 2;
}

formatFactory.java

FormatFactory formatFactory = new FormatFactory();
ProtobufFormatter formatter = formatFactory.createFormatter(FormatFactory.Formatter.XML_JAVAX);
InputStream in = TextUtils.toInputStream(paramString);
formatter.merge(in, this.builder);

I got a java.lang.RuntimeException: Can't get here. message exception at XmlJavaxFormat.java:566.

Change the formater, from XML_JAVAX to XML, I got this exception:
com.googlecode.protobuf.format.ProtobufFormatter$ParseException: 4:22: Expected ">".

Request sending:

<ANEXOS>
    <anexos>
        <tipoAnexo>3</tipoAnexo>
        <descricao>foto frontal</descricao>
        <anexo><![CDATA[//]]></anexo>
    </anexos>
</ANEXOS>

Add at least one unit test

Does not build

Get error on line no. 235 saying symbol not found. I've been using 1.7
I had to remove the usage of toUnsignedLong and unsignedLong again on 240 to make it build.

proto3 JSON mapping support

I'm using this with proto3 syntax, and it's working well (thanks for all the work you've put into it). With proto3, they've introduced a canonical encoding into JSON. Do you have plans to support this when using proto3?

JsonJacksonFormat: unsigned int and long aren't output/input as unsigned.

@kunzchr reported recently that UINT32, FIXED32, UINT64 and FIXED64 types write out and read in as signed ints/longs, whereas they need to use the next larger types (long / BigInteger) to load and then cast (after any appropriate range checks).

XML Format: Should Escape String

On Line: https://github.com/bivas/protobuf-java-format/blob/master/src/main/java/com/googlecode/protobuf/format/XmlFormat.java#L181

Should perform a Commons Text StringEscapeUtils.escapeXml() so it escapes Ampersands and such inside the text.

JsonFormat throws an exception when it faces an empty json object

I have this json = {"changed_aspect": "media", "object": "tag", "object_id": "??", "time": 1450979289, "subscription_id": 0, "data": {}}

JsonFormatter throws an error when it faces data field:

1:127: Expected identifier. -}
com.googlecode.protobuf.format.JsonFormat$ParseException: 1:127: Expected identifier. -}
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:734)
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.consumeIdentifier(JsonFormat.java:537)
    at com.googlecode.protobuf.format.JsonFormat.handleMissingField(JsonFormat.java:887)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:853)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:875)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:875)

Missing release tags

Hi,

The GitHub repository is missing some release tags for the old releases, could you please consider adding them? Here are the commits matching the release dates on https://code.google.com/archive/p/protobuf-java-format/downloads

git tag 1.2   8d4e755fca74820578885909f93ce50c14fc3889
git tag 1.1.1 dabc9ef2f74e67e30a0596874a24f51248580ad0
git tag 1.1   660fcd81a67c48b2ea160e0aee3e5fa853c248ae
git tag 1.0   daeb86721223a1f12be6b6026aa1c7444f4d0603
git push --tags

Thank you

Unknown fields not being repopulated upon de-serialization from json

If I have a protobuf message with unknown fields:

leaf1: "Hello"
leaf2: 23
leaf3: 41
4: "world"

where leaf1, leaf2, and leaf3 are my known fields, and 4 is an unknown field. I then convert to json like so:

return new JsonFormat().printToString(protobuf);

I get the following:

{"leaf1": "Hello","leaf2": 23,"leaf3": [41], "4": ["world"]}`

However, if I then deserialize the json back like so:

public static <T extends Message> T convertToProtobuf(String json, T defaultInstance)
        throws ParseException {
        Builder builder = defaultInstance.newBuilderForType();
        JsonFormat format = new JsonFormat();
        format.merge(json, ExtensionRegistry.getEmptyRegistry(), builder);
        return (T)builder.build();
    }

then my result is the following:

leaf1: "Hello"
leaf2: 23
leaf3: 41

The problem is field 4 is not present in the reconstituted proto as an unknown field. Am I doing something wrong, or is this not supported? Thanks!

Exception when consuming unknown field with value being empty list

parsing something like
{
"unknown": []
}

fails with
com.googlecode.protobuf.format.JsonFormat$ParseException: 1:32: Expected string.
at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:734)
at com.googlecode.protobuf.format.JsonFormat$Tokenizer.consumeString(JsonFormat.java:683)
at com.googlecode.protobuf.format.JsonFormat.handleMissingField(JsonFormat.java:909)
at com.googlecode.protobuf.format.JsonFormat.handleMissingField(JsonFormat.java:895)
at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:853)
at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:875)
at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:790)

Upgrade protobuf dependency

To 2.5.0