Cannot use data stream with time_series mode

[jpthomasset]

Cannot use data stream with time_series mode

[jpthomasset]

I ended up forking the repo and adding an extra setting allowing to not set ID when creating the index request. jpthomasset@7b44693
Very basic implementation but it solved my problem for an experiment.

And then building the image is done with

ARG PLUGINS_FOLDER=plugins

FROM maven:3.9.5-eclipse-temurin-11 as maven-builder
COPY ./custom-elasticsearch-plugin /mvn-build
WORKDIR /mvn-build

RUN git clone https://github.com/jpthomasset/kafka-connect-elasticsearch.git --branch unset-key --single-branch

WORKDIR /mvn-build/kafka-connect-elasticsearch
RUN mvn -s /mvn-build/m2-settings.xml -P standalone package

FROM docker-registry-proxy.internal.stuart.com/confluentinc/cp-server-connect-base:7.5.0

ARG PLUGINS_FOLDER
RUN mkdir -p /usr/share/java/kafka-connect-storage-common
# Install S3 sink, transform plugins
RUN     confluent-hub install --no-prompt --component-dir /usr/share/java/kafka-connect-storage-common confluentinc/kafka-connect-s3:latest \
    &&  confluent-hub install --no-prompt --component-dir /usr/share/java/kafka-connect-storage-common confluentinc/connect-transforms:latest

# Disable standard elastic search plugin
#    &&  confluent-hub install --no-prompt --component-dir /usr/share/java/kafka-connect-storage-common confluentinc/kafka-connect-elasticsearch:latest

# Add custom elasticsearch plugin with feature to not push id
COPY --from=maven-builder /mvn-build/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-*-SNAPSHOT-standalone.jar /usr/share/java/kafka-connect-storage-common/

ENV CONNECT_REST_ADVERTISED_PORT 8084

And you can then add a key.unset setting to your kafka connector

{
    "name": "some-connector-name",
    "config": {
       [...]
        "key.unset": "true",
       [...]
    }
}

[morrone]

Agreed! We needed to add an elasticsearch ingest pipeline to strip the id, but for other reasons. Not ideal. Giving us a a connector configuration option to opt out of the id would be great. I don't know yet if our pipeline to strip the id will work with index.mode set to time_series.