infinispan/protostream

ProtoStream is a serialization library based on Protocol Buffers

ProtoStream

ProtoStream is a serialization library based on Protobuf data format. It is open source software released under the Apache License, v2.0. ProtoStream is part of the Infinispan data grid platform. For more information about Infinispan visit the project's website on https://infinispan.org

Usage (via Maven)

Add this dependency to your pom.xml file:

<dependency>
   <groupId>org.infinispan.protostream</groupId>
   <artifactId>protostream</artifactId>
   <version>5.0.1.Final</version>
</dependency>

Annotation processor

The Java compiler discovers annotation processors differently, depending on the version and the options in use. With Java up to version 21, it's enough to put the protostream-processor dependency on the classpath and it will be automatically discovered by the service loader mechanism. Another way, which is mandatory since Java 22, is to use the --processor-path option.

javac --processor-path /path/to/protostream-processor-5.0.1.Final.jar:... ...

Using Maven:

<plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>3.12.1</version>
  <configuration>
    <!-- Annotation processors -->
    <annotationProcessorPaths>
      <annotationProcessorPath>
        <groupId>org.infinispan.protostream</groupId>
        <artifactId>protostream-processor</artifactId>
        <version>5.0.1.Final</version>
      </annotationProcessorPath>
    </annotationProcessorPaths>
  </configuration>
</plugin>

The annotation processor supports some configuration options:

• protostream.debug prints out debug information during the processing phase.
• protostream.fullyqualifiedannotations when generating the .proto files, the annotation names are stripped of the package name. This flag keeps the fully-qualified annotation names in the generated files.

You can pass these arguments to javac using the -Aoption=value argument. The following pom.xml snippet shows how to do it with Maven:

<plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>3.12.1</version>
  <configuration>
    <compilerArgs>
      <arg>-Aprotostream-debug=true</arg>
    </compilerArgs>
    <!-- Annotation processors -->
    <annotationProcessorPaths>
      <annotationProcessorPath>
        <groupId>org.infinispan.protostream</groupId>
        <artifactId>protostream-processor</artifactId>
        <version>5.0.1.Final</version>
      </annotationProcessorPath>
    </annotationProcessorPaths>
  </configuration>
</plugin>

Requirements

Target runtime platform is Java 17 or newer.

Requires Java 17 or newer to build.

Maven 3.6.0 or newer.

Bugs

Bug reports go here

protostream's Issues

Standalone usage

Is it possible to use protostream standalone without an Infinispan cache? Just for serialization and deserialization from/to a byte array or ByteBuffer?

Caused by: org.infinispan.protostream.AnnotationParserException: Error: 1,43: illegal escape character: u

Not able to use @protofield with kotlin.

Got this:

        at org.infinispan.protostream.impl.parser.AnnotationLexer.scanLiteralChar(AnnotationLexer.java:167)
        at org.infinispan.protostream.impl.parser.AnnotationLexer.nextToken(AnnotationLexer.java:388)
        at org.infinispan.protostream.impl.parser.AnnotationParser.expect(AnnotationParser.java:55)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseArray(AnnotationParser.java:247)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseValue(AnnotationParser.java:184)

Gradle kotlin dsl is

kapt("org.infinispan.protostream:protostream-processor:4.6.2.Final")
	annotationProcessor("org.infinispan.protostream:protostream-processor:4.6.2.Final")
	implementation("org.infinispan.protostream:protostream-processor:4.6.2.Final")

Context is

@AutoProtoSchemaBuilder(

    includeClasses = [ UserData2::class],
)
interface LibraryInit : SerializationContextInitializer

UserData2 class is

open class UserData2 {

    @get:ProtoField(number = 1)
    @Max(501)
    var description: String? = null
}

if I move UserData2 to pure java module and include into first module - everything works fine.

full stack trace

api/build/tmp/kapt3/stubs/main/com/proto/LibraryInit.java:5: error: org.infinispan.protostream.DescriptorParserException: org.infinispan.protostream.AnnotationParserException: Error: 1,43: illegal escape character: u
public abstract interface LibraryInit extends org.infinispan.protostream.SerializationContextInitializer {
                ^
        at org.infinispan.protostream.descriptors.FileDescriptor.resolveDependencies(FileDescriptor.java:266)
        at org.infinispan.protostream.descriptors.FileDescriptor.resolveDependencies(FileDescriptor.java:210)
        at org.infinispan.protostream.descriptors.ResolutionContext.resolve(ResolutionContext.java:57)
        at org.infinispan.protostream.impl.SerializationContextImpl.registerProtoFiles(SerializationContextImpl.java:118)
        at org.infinispan.protostream.annotations.impl.BaseProtoSchemaGenerator.generateAndRegister(BaseProtoSchemaGenerator.java:171)
        at org.infinispan.protostream.annotations.impl.processor.AutoProtoSchemaBuilderAnnotationProcessor.processClass(AutoProtoSchemaBuilderAnnotationProcessor.java:336)
        at org.infinispan.protostream.annotations.impl.processor.AutoProtoSchemaBuilderAnnotationProcessor.processElement(AutoProtoSchemaBuilderAnnotationProcessor.java:266)
        at org.infinispan.protostream.annotations.impl.processor.AutoProtoSchemaBuilderAnnotationProcessor.process(AutoProtoSchemaBuilderAnnotationProcessor.java:189)
        at org.jetbrains.kotlin.kapt3.base.incremental.IncrementalProcessor.process(incrementalProcessors.kt:90)
        at org.jetbrains.kotlin.kapt3.base.ProcessorWrapper.process(annotationProcessing.kt:197)
        at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:1018)
        at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:934)
        at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1262)
        at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1377)
        at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1248)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.jetbrains.kotlin.kapt3.base.AnnotationProcessingKt.doAnnotationProcessing(annotationProcessing.kt:90)
        at org.jetbrains.kotlin.kapt3.base.AnnotationProcessingKt.doAnnotationProcessing$default(annotationProcessing.kt:31)
        at org.jetbrains.kotlin.kapt3.base.Kapt.kapt(Kapt.kt:47)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.jetbrains.kotlin.gradle.internal.KaptExecution.run(KaptWithoutKotlincTask.kt:311)
        at org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask$KaptExecutionWorkAction.execute(KaptWithoutKotlincTask.kt:257)
        at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:63)
        at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:66)
        at org.gradle.workers.internal.NoIsolationWorkerFactory$1$1.create(NoIsolationWorkerFactory.java:62)
        at org.gradle.internal.classloader.ClassLoaderUtils.executeInClassloader(ClassLoaderUtils.java:100)
        at org.gradle.workers.internal.NoIsolationWorkerFactory$1.lambda$execute$0(NoIsolationWorkerFactory.java:62)
        at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:44)
        at org.gradle.workers.internal.AbstractWorker$1.call(AbstractWorker.java:41)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
        at org.gradle.workers.internal.AbstractWorker.executeWrappedInBuildOperation(AbstractWorker.java:41)
        at org.gradle.workers.internal.NoIsolationWorkerFactory$1.execute(NoIsolationWorkerFactory.java:59)
        at org.gradle.workers.internal.DefaultWorkerExecutor.lambda$submitWork$2(DefaultWorkerExecutor.java:212)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runExecution(DefaultConditionalExecutionQueue.java:187)
        at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.access$700(DefaultConditionalExecutionQueue.java:120)
        at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner$1.run(DefaultConditionalExecutionQueue.java:162)
        at org.gradle.internal.Factories$1.create(Factories.java:31)
        at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
        at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
        at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:114)
        at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.runBatch(DefaultConditionalExecutionQueue.java:157)
        at org.gradle.internal.work.DefaultConditionalExecutionQueue$ExecutionRunner.run(DefaultConditionalExecutionQueue.java:126)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1623)
  Caused by: org.infinispan.protostream.AnnotationParserException: Error: 1,43: illegal escape character: u
        at org.infinispan.protostream.impl.parser.AnnotationLexer.scanLiteralChar(AnnotationLexer.java:167)
        at org.infinispan.protostream.impl.parser.AnnotationLexer.nextToken(AnnotationLexer.java:388)
        at org.infinispan.protostream.impl.parser.AnnotationParser.expect(AnnotationParser.java:55)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseArray(AnnotationParser.java:247)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseValue(AnnotationParser.java:184)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseAttribute(AnnotationParser.java:173)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseAttributes(AnnotationParser.java:152)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parseAnnotation(AnnotationParser.java:89)
        at org.infinispan.protostream.impl.parser.AnnotationParser.parse(AnnotationParser.java:43)
        at org.infinispan.protostream.impl.AnnotatedDescriptorImpl.processAnnotations(AnnotatedDescriptorImpl.java:83)
        at org.infinispan.protostream.impl.AnnotatedDescriptorImpl.getProcessedAnnotation(AnnotatedDescriptorImpl.java:309)
        at org.infinispan.protostream.descriptors.Descriptor.setFileDescriptor(Descriptor.java:124)
        at org.infinispan.protostream.descriptors.FileDescriptor.collectDescriptors(FileDescriptor.java:311)
        at org.infinispan.protostream.descriptors.FileDescriptor.resolveDependencies(FileDescriptor.java:245)
        ... 58 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':api:kaptKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask$KaptExecutionWorkAction
   > java.lang.reflect.InvocationTargetException (no error message)

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12s

Generated java class is:

package com.sashka.proto;

@kotlin.Metadata(mv = {1, 9, 0}, k = 1, xi = 48, d1 = {"\u0000\n\n\u0002\u0018\u0002\n\u0002\u0018\u0002\n\u0000\bg\u0018\u00002\u00020\u0001\u00a8\u0006\u0002"}, d2 = {"Lcom/sashka/proto/LibraryInit;", "Lorg/infinispan/protostream/SerializationContextInitializer;", "api"})
@org.infinispan.protostream.annotations.AutoProtoSchemaBuilder(includeClasses = {com.sashka.model.UserData2.class})
public abstract interface LibraryInit extends org.infinispan.protostream.SerializationContextInitializer {
}

ProtoField for abstract class

I am trying to create a @ProtoAdapter for a third party class and I couldn't find examples for usage with inheritance.
Lets say there is a class called CachedAnimal that contains an Animal and CacheDuration.

@ProtoAdapter(CachedAnimal.class)
public class CachedAnimalAdapter {
  @ProtoFactory 
  CachedAnimal create(Animal animal, String cacheDuration) {
    return new CachedAnimal(animal, cacheDuration);
  }

  @ProtoField
  Animal getAnimal(CachedAnimal cachedAnimal) {
     cachedAnimal.getAnimal();
  }
  
  @ProtoField
  String getCacheDuration(CachedAnimal cachedAnimal) {
     cachedAnimal.getDuration();
  }
}

However, this fails to compile with @AutoProtoSchemaBuilder saying the @ProtoField type "should not be abstract".

Please provide details on how to deal with abstract types and add some examples to the documentation.

Support for generic marshallers

I would like a way to use generic marshallers without relying on deprecated methods, right now org.infinispan.protostream.impl.SerializationContextImpl looks up marshallers using a hashmap with the class as a key which doesn't find a compatible marshaller. I am currently working around this by creating a MarshallerProvider.. would you consider keeping MarshallerProvider in order to maintain this functionality?

 serializationContext.registerMarshallerProvider(new MarshallerProvider() {
            @Override
            public BaseMarshaller<?> getMarshaller(String typeName) {
                return null;
            }

            @Override
            public BaseMarshaller<?> getMarshaller(Class<?> javaClass) {
                if (AbstractActor.class.isAssignableFrom(javaClass)) {
                    return getContext().getMarshaller(AbstractActor.class);
                }
                if (ActorObserver.class.isAssignableFrom(javaClass)) {
                    return getContext().getMarshaller(ActorObserver.class);
                }
                if (RemoteReference.class.isAssignableFrom(javaClass)) {
                    return getContext().getMarshaller(RemoteReference.class);
                }
                return null;
            }
        });

Protobuf encoding marshling is not working when in a Java collection is used in the value part of the map

Hi

With the following cache definition e.g: Cache<String, ArrayList<Sensor>> , with Protobuf, the values are not being marshalled.

To recreated this issue, you can create a new project:

mvn archetype:generate -DarchetypeGroupId=org.infinispan.archetypes  -DarchetypeArtifactId=embedded -DarchetypeVersion=14.0.1.Final

added the latest lib in the pom.xml:

  ...
        <dependency>
            <groupId>org.infinispan.protostream</groupId>
            <artifactId>protostream-processor</artifactId>
            <version>14.0.0.CR2</version>
        </dependency>
  ...

Edited the upd xml file:

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:xi="http://www.w3.org/2001/XInclude"
        xsi:schemaLocation="urn:infinispan:config:14.0 http://infinispan.org/schemas/infinispan-config-14.0.xsd"
        xmlns="urn:infinispan:config:14.0">

    <cache-container default-cache="default" statistics="true">

        <serialization marshaller="org.infinispan.commons.marshall.ProtoStreamMarshaller">
            <context-initializer class="test.com.LibraryInitializerImpl"/>
            <allow-list>
                <class>test.com.Sensor</class>
            </allow-list>
        </serialization>

        <transport stack="udp" cluster="mycluster" />

        <distributed-cache name="stockTickers" mode="SYNC">
            <encoding>
                <key media-type="application/x-protostream"/>
                <value media-type="application/x-protostream"/>
            </encoding>
            <locking isolation="REPEATABLE_READ" striping="false"/>
        </distributed-cache>

        <replicated-cache name="default"/>

    </cache-container>
</infinispan>

A very simple POJO

public class Sensor {

	@ProtoField(value = 1, defaultValue = "0")
	public int anInt;

	public Sensor() {
	}

	public Sensor(int anInt) {
		this.anInt = anInt;
	}
}

This code:

   public void basicUse() {
      System.out.println("\n\n1.  Demonstrating basic usage of Infinispan.  This cache stores arbitrary Strings.");
      Cache<String, ArrayList<Sensor>> cache = cacheManager.getCache("stockTickers", true);

      cache.put("test", new ArrayList<>(List.of(new Sensor(1), new Sensor(2), new Sensor(3), new Sensor(4))));

      // this fails
      List<Sensor> test = cache.get("test");

   }

yields in: Exception in thread "main" org.infinispan.commons.dataconversion.EncodingException: ISPN000495: ProtostreamTranscoder encountered error transcoding content because in this code, since in the example the array contains 4elements (4tags and 4 bytes arrays), the field count gives 8, throwing the error. If you only add one into the list the code works:

   //this works
   cache.put("test", new ArrayList<>(List.of(new Sensor(1))));

I've tried simple types, like a list of integers Cache<String, ArrayList<Integer>> and it works because of the break in the switch

Use ByteBuddy instead of Javassist

https://issues.redhat.com/projects/IPROTO/issues/IPROTO-259

ObjectArrayAdapter generating marshaller in java.lang

I'm trying to create an adaptor for a dto which contains an object field, that's either an object of a registered marshaller, or an object array of objects of various registered marshallers [1].. I tried adding the ObjectArrayAdapter to my project but protostream-processor is generating an Object marshaller in java.lang inside the generated sources folder, am I using it incorrectly / bug?

[1]

public class Message
{
    private int messageType;
    private int messageId;

    private NodeAddress fromNode;
    private NodeAddress toNode;

    private Map<String, Object> headers;

    // target reference fields
    private int interfaceId;
    private Object objectId;
    private NodeAddress referenceAddress;

    // method fields
    private int methodId;
    private Object payload; // problematic field

    private long localRequestTime;

    private long remoteInvocationCreationTime;
    private long remoteInvocationCompletionTime;

Compilation error "@ProtoAdapter can only be applied to classes" on enum

Hi, I wrote a ProtoAdapter for an Enum and get a compilation error:

[ERROR] /home/user/.../caching/src/main/java/com/example/ColorEnumAdapter.java:[7,8] @ProtoAdapter can only be applied to classes.

But documentation says

@ProtoAdapter is a marshalling adapter for a class or enum that you cannot annotate directly.

I also found an Unit Test in source code doing this. My example files are adopted from there:

ColorEnum.java

package com.example;

public enum ColorEnum {
    RED, GREEN, BLUE
}

ColorEnumAdapter.java

package com.example;

import org.infinispan.protostream.annotations.ProtoAdapter;
import org.infinispan.protostream.annotations.ProtoEnumValue;

@ProtoAdapter(ColorEnum.class)
public enum ColorEnumAdapter {

    @ProtoEnumValue(number = 0, name = "red")
    RED,

    @ProtoEnumValue(number = 1, name = "green")
    GREEN,

    @ProtoEnumValue(number = 2, name = "blue")
    BLUE
}

Can you help me to find out what I'm doing wrong?

ProtoReserved statements always commented in proto file

Annotating a class with @ProtoReserved(numbers = 3) results in the following:

message Foo {
   //reserved 3;
}

However the proto format for reserved statements is as follows:

message Foo {
  reserved 3;
}

java.lang.NoClassDefFoundError: ProtoSchema - v5.0.1.Final

Using the new 5.0.1.Final dependency creates the following error:
Execution failed for task ':compileJava'.

java.lang.NoClassDefFoundError: org/infinispan/protostream/annotations/ProtoSchema

This happens as soon as the annotation processor is added to the project. This also happens with all the other v5 versions. Any feedback on this ? Thank you