Comments (3)
Hi @hpgrahsl,
Thanks so much for the good work and your commitment to this project.
As of now, this issue has been solved managing it as a string. However, as you have already indicated for a more future definitive solution I would have to follow the option b. By then, It will be my pleasure to make a PR.
Again, really appreciated the support.
BR
Luis
from kafka-connect-mongodb.
thx for reaching out and clearly explaining your use case with sample messages & documents. the question for me is the following though: if the _id field in the kafka message contains always an objectid compliant 24 chars hex string, why is it not in extended json serialization using $oid see https://docs.mongodb.com/manual/reference/mongodb-extended-json-v1/#oid ? then it would work as is.
since it's only the raw hex-string, there is currently no way to support this out-of-the-box. you have two options though to make it work without much effort, yet it involves writing some code.
a) write a custom SMT (single message transform) and add this SMT configuration to your connector config
b) write a custom post-processor for the mongodb sink connector which is probably even easier than a) for your use case
I'm happy to receive a PR of course to integrate this "new" post-processor then into the project 😉 If you need any further information just let me know.
from kafka-connect-mongodb.
Great to hear you got it working for your needs! If you don't mind I'd be happy if you could close this issue then ... at least for now. Feel free to re-open anytime when you start with a definitive solutions for this.
from kafka-connect-mongodb.
Related Issues (20)
- Log is not getting appended in File HOT 3
- Are there any configuration setting to get fullDocument Json only? HOT 1
- support new compound shard key handling of Debezium Source Connector
- Version 1.4.0 on Confluent Hub HOT 3
- Avro schema "TopicNameStrategy" error HOT 1
- Removed records are put back again immediately. HOT 4
- where is the jar? HOT 1
- Use two WriteModel Strategies in one config? HOT 5
- Cannot convert Record to BSON HOT 2
- Issue with MongoDB sink connector, Records inserted to MongoDB are in the form of CDC structure instead of a mongo document structure HOT 1
- Is there "update"(NOT "replace") as operation type of the result from MongoSinkConnector? HOT 4
- Can I customize Kafka topic from MongoSourceConnector? HOT 1
- Error with shard mongoDB
- Error: `operationType` field is doc is missing
- Post Processor Chain not working HOT 3
- MongoDB Sink Connector Fails with NullPointerException HOT 1
- Question: has this been removed from Confluent Hub? HOT 2
- Generating unwanted key for Delete HOT 1
- Write model strategy - optional upsert HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-mongodb.