pegasystems/dataset-integrations

Kafka custom serializer/deserializer implementations

dataset-integrations's Issues

Unable to turn off Warning logging in ClipboardPageToGenericRecordConverter.java

Line93: LOGGER.warn(FORCE_LOGGING_MARKER, "Property couldn't be matched with the potential schema.", e);
The warning generates lots of logs, and we are not able to turn this off via Pega Log category
com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter
Pls advice how to disable this warning.

Question about key

Hi I had q question regarding publishing of key.
I see in the code that
delegateValueSerializer.configure(additionalConfiguration, false);
delegateValueDeserializer.configure(additionalConfiguration, false);
only the value serde is there no key serde.
I have a requirement where i have to publish the key and value both in avro format.
For now when i publish messages only value is sent.
How can this be done?

Missing prpublic-dsmint

When try to build example - face an issue:

C:\Users\name\Downloads\dataset-integrations-master\dataset-integrations-master\kafka-dataset-samples>gradlew.bat build -x test
> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find :prpublic-dsmint:.
     Searched in the following locations:
       - file:/C:/Users/name/Downloads/dataset-integrations-master/dataset-integrations-master/kafka-dataset-samples/libs/prpublic-dsmint.jar
     Required by:
         project :

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
1 actionable task: 1 executed

Where I suppose to get prpublic-dsmint? It doesn't exist in the installation distro. Also nothing in the readme about that dependency.

Issue while consuming messages

Hi,

I tried doing a small POC on this project and when I try to consume a message it gives me the error
I have set it up according to the instructions. Also when I had done the import of libs it did not take so I had to store it in the lib folder of the prweb and it worked after that. But then got this error.

Error line in java code --
delegateValueSerializer.configure(additionalConfiguration, false);

Error message

Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.confluent.kafka.schemaregistry.client.rest.RestService
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.(CachedSchemaRegistryClient.java:104) ~[kafka-schema-registry-client-5.2.1.jar:?]
at io.confluent.kafka.serializers.AbstractKafkaAvroSerDe.configureClientProperties(AbstractKafkaAvroSerDe.java:62) ~[kafka-avro-serializer-5.3.1.jar:?]
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.configure(AbstractKafkaAvroSerializer.java:42) ~[kafka-avro-serializer-5.3.1.jar:?]
at io.confluent.kafka.serializers.KafkaAvroSerializer.configure(KafkaAvroSerializer.java:48) ~[kafka-avro-serializer-5.3.1.jar:?]
at com.pega.integration.kafka.AvroSchemaRegistrySerde.configure(AvroSchemaRegistrySerde.java:90) ~[PegaAvroSerde.jar:?]
at com.pega.dsm.dnode.api.dataset.kafka.KafkaDataSet$KafkaDataSetBuilder.build(KafkaDataSet.java:305) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.dataset.kafka.KafkaDataSetFactory.createFromPage(KafkaDataSetFactory.java:94) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.dataset.DataSetFactory.createFromKey(DataSetFactory.java:39) ~[d-node.jar:?]

Single top-level gradle project

Hello,
This project has four gradle projects in subdirectories. Can we please have a top level build.gradle which will allow us to build all the subprojects at once? This will help with developing CI and scanning pipelines.

I'll create a branch with a proposed structure for your review.

mTLS support

I am referring the code and wanted to understand how can we pass SSLConfig details for mTLS supported schema registry url?