sanjuthomas/kafka-connect-gcp-bigtable

Kafka Sink Connect to GCP Bigtable - https://www.confluent.io/hub/sanjuthomas/kafka-connect-gcp-bigtable

kafka-connect-gcp-bigtable's Issues

Work with latest Kafka (ie without Zookeeper)

Hi
Great contribution for the community, thanks!
Looking at the prerequisites, it says zookeeper is one of them. So wondering, if the connector can still function without zookeeper.
Kind regards
Karan

Handling column not present in data

Hi @sanjuthomas

Does the transform handle a column that is not present in the map?
Say for example, my config looks like

    families:
    - data
    familyQualifiers:
    - data:
      - anonymousId
      - context

And a test data looks like

{
   "anonymousId" : "random123"
}

Will the transform you have baked into, will automatically set the context to null or throw an error. Guess, I will find out pretty soon but if it throws an error, is there a good way to handle it ?

As GCP Bigtable sink connect user, I wanted to use schema registry so that I can reject that bad data on the producer end itself.

As GCP Bigtable sink connect user, I wanted to use schema registry so that I can reject that bad data on the producer end itself.

Bigtable Kafka Sink - Implement error handling and retry

Implement error handling and retry

As GCP Bigtable sink connect user, I wanted to use Avro serializer and deserializer.

As GCP Bigtable sink connect user, I wanted to use Avro serializer and deserializer.

Tombstone support for delete

Any plans to add support for deleting records through tombstone?

Support for transformation of KSQL Structs.

I am trying to sink our KSQL streams into BigTable. We have a relatively complex JSON schema (nested arrays and objects). We can use the big query connector from Wepay to sink into big query.

Having never written any Kafka Connectors before, is it possible to replace the Transformer provided in your code, with an object from the Wepay connector, or, is the Transformer specific to this connector's sink target?