icemachined/kafka-kotlin-native

Kafka client for Kotlin Native

Kafka-Kotlin-Native

Fully Native Apache Kafka client for Kotlin Native. Uses native c-interop with highly performant and reliable librdkafka C client library. This library contains no JVM dependencies, no jvm runtime required. It uses Kotlin Native memory model and Multithreaded coroutines for non-blocking interaction with native callbacks and asynchronous workers.

Contribution

We will be glad if you will test kafka-kotlin-native or contribute to this project. In case you don't have much time for this - at least spend 5 seconds to give us a star to attract other contributors!

Thanks! 🙏 🥳

Acknowledgement

Special thanks to those awesome developers who give us great suggestions, help us to maintain and improve this project: @olonho, @akuleshov7.

Supported platforms

The code has both common and native parts. It can be built for each platform for which librdkafka has support. Currently, this is Linux, Windows and Mac OSX.

For more information about platforms and how to install librdkafka see librdkafka installation

Dependency

The library is hosted on the Maven Central. To import kafka-kotlin-native library you need to add following dependency to your code:

Maven

<dependency>
  <groupId>com.icemachined</groupId>
  <artifactId>kafka-client</artifactId>
  <version>0.2.0</version>
</dependency>

Gradle Groovy

implementation 'com.icemachined:kafka-client:0.2.0'

Gradle Kotlin

implementation("com.icemachined:kafka-client:0.2.0")

Features

• Synchronous and asynchronous send
• Leverages kotlin-native coroutines and memory model
• Polling Kafka consumer
• Parallel polling Kafka consumer
• Headers enrichment
• Error handling extension points
• Possibility to leverage kotlinx.serialization features

How to use

See example of usage in example project This example shows how to start/stop producer and consumer and how to configure it.

Configuration

• librdkafka configuration properties, which you can pass to producer and consumer constructor.
• Serializer/Deserializer for key and value.

kafka-kotlin-native's Issues

Collaboration / Integration

Hey @icemachined, and all other contributors,

Thanks for this amazing effort 🙏 It's really awesome seeing support for something like this in Kotlin 🙌
Over the summer I started a Kafka streaming project, similar to reactor-kafka and Alpakka Kafka but instead of respectively Project Reactor and Akka build on top of KotlinX Coroutines Flow. The project lives here.

Since the JVM implementation is almost finished, I was hoping to also support Kotlin JS and Kotlin Native in the future but before pulling in your library and trying to integrate it using expect/actual I wanted to reach out and ask what your ambitions were with this project. I don't want to create competing libraries, or conflict in the Kotlin MPP eco-system. Additionally, this project clearly took a lot of amazing work. Big kudos to everyone involved!

So my question, would there be interest in create a Kafka 4 Kotlin OSS org on Github where we could group these efforts as a neutral space for all different Kafka efforts happening atm in Kotlin? Do you mind if I depend on this library to provide a Kotlin Native implementation for the kafka-kotlin streaming project?

Ultimate goal would be to have some Kotlin MPP facade for Kafka, and ideally expose integrations for Ktor, or other Kotlin MPP frameworks.

What do you think? Again, thanks for brining such awesome library to the Kotlin eco-system 🎉

MacOS error: `ld: library not found for -lrdkafka`

When I run ./gradlew assemble on my Mac, I get an error:

plugins {
  kotlin("multiplatform") version "1.7.21"
}

group = "demo"
version = "0.0.1"

kotlin {

  macosX64("native") {
    binaries {
      executable {
        entryPoint = "main"
      }
    }
  }

  sourceSets {
    val nativeMain by getting {

      dependencies {
        implementation(platform(kotlin("bom")))

        implementation("com.icemachined:kafka-client:0.2.0")
      }
    }

    val nativeTest by getting {
      dependencies {
      }
    }
  }
}

When I checkout kafka-kotlin-native and run ./gradlew assemble, it fails, and the output includes this stacktrace:

> Task :kafka-client:cinteropLibrdkafkaMacosX64
Exception in thread "main" java.lang.Error: /var/folders/wl/2bzbzknd077dyh2ww_6xjty80000gp/T/3344506493200322852.c:1:10: fatal error: 'librdkafka/rdkafka.h' file not found
	at org.jetbrains.kotlin.native.interop.indexer.UtilsKt.ensureNoCompileErrors(Utils.kt:274)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.indexDeclarations(Indexer.kt:1196)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.buildNativeIndexImpl(Indexer.kt:1185)
	at org.jetbrains.kotlin.native.interop.indexer.IndexerKt.buildNativeIndexImpl(Indexer.kt:1181)
	at org.jetbrains.kotlin.native.interop.gen.jvm.DefaultPlugin.buildNativeIndex(Plugins.kt:33)
	at org.jetbrains.kotlin.native.interop.gen.jvm.MainKt.processCLib(main.kt:289)
	at org.jetbrains.kotlin.native.interop.gen.jvm.MainKt.processCLibSafe(main.kt:214)
	at org.jetbrains.kotlin.native.interop.gen.jvm.MainKt.interop(main.kt:80)
	at org.jetbrains.kotlin.cli.utilities.InteropCompilerKt.invokeInterop(InteropCompiler.kt:45)
	at org.jetbrains.kotlin.cli.utilities.MainKt.mainImpl(main.kt:40)
	at org.jetbrains.kotlin.cli.utilities.MainKt.main(main.kt:62)

deserialization error handling: can't serialize Throwable

kotlin native requires @serializable for all fields and throw exception as far as DeserializationExceptionData.cause is of type Throwable.