Comments (10)
So ES 5.x is not yet supported by this library?
from kafka-connect-elasticsearch.
For ES 6.0 and above, the string type has been completely removed. In order to get the existing elasticsearch-connector to work with ES, we can set schema.ignore
to true
in the connector configuration. This will skip creating the mappings which causes the tasks to fail.
Importing data into ES will work with the above workaround. When the mappings are not specified, ES server will detect the type using the field mapping rules specified here. A string in your JSON document will most likely be converted into a text type and a keyword type (if it's within the length limits). This will allow full text searches on the analyzed text.
from kafka-connect-elasticsearch.
@shikhar Would probably be good to get the system tests to run against ES 5.0 too, will probably reveal many issues.
from kafka-connect-elasticsearch.
Same question here...
from kafka-connect-elasticsearch.
I believe that one of the issues with using the string type in ES 5 is combining it with null_value. Omitting the null_value setting in Mapping.java
allows ES 5 to create the mapping.
from kafka-connect-elasticsearch.
It's not fully supported yet (though note that you can still use it -- just the incompatibility noted above). This should get closed once ES 5 support is resolved.
from kafka-connect-elasticsearch.
Is this issue is still open?
I would like to work on it 😄
from kafka-connect-elasticsearch.
elasticsearch 5.0 will silently convert string to the new text field data type, but in 6.0 this fails miserably.
This line:
from kafka-connect-elasticsearch.
See #154 (comment)
from kafka-connect-elasticsearch.
This is fixed with #169
from kafka-connect-elasticsearch.
Related Issues (20)
- How to Convert JSON String field to ES Object?
- Capture Kafka key without using it as ID HOT 2
- Suggestion for INSERT operation "Ignoring EXTERNAL version conflict for operation INDEX on document"
- Used Elastic Java REST client is deprecated in 7.15.0 HOT 1
- Error with `"behavior.on.null.values": "delete"`
- Consumer paused indefinitely when using `AsyncOffsetTracker` with lot of null values
- Cannot use data stream with time_series mode HOT 2
- Error: Cannot infer mapping without schema HOT 1
- Connector fails with payloads >20 MB HOT 1
- Can't create a connector even if its loaded in Strimzi
- Support requests per second configuration options
- Log when there are too many requests errors
- [BUG] `TOO_MANY_REQUESTS` error craches the tasks with a unrecoverable exceptions without retries
- Ignore 'document_parsing_exception' HOT 1
- Inconsistent Logging for Tombstone Messages in Elastic Sink Connector
- abnormal data loss question
- Data Stream naming is far too restrictive HOT 1
- Creating index based on Timestamp doesn't work
- Limit retry backoff (and unlimited retries) HOT 4
- add support for index templates other than logs and metrics as types when using data streams
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-elasticsearch.