Comments (6)
It is not the goal of a Committer to "configure" its target, but that could be handy nonetheless, as long as it does not create new problems.
What would happen if we submit the mappings again every time the collector runs? Or if you make changes to the mappings between two runs? Or worse, what if someone changed the schema manually on ES and this new option overwrite such changes?
If none of the above is a concern, we can consider adding this option.
from committer-elasticsearch.
What would happen if we submit the mappings again every time the collector runs?
If the index already exists, ES will complain. I think the collector should ignore and log this specific issue and continue. At least, this is what I have seen other tools do.
Or if you make changes to the mappings between two runs?
Since the index won't be changed because it already exists, collector will continue to run as normal again ignoring/logging the ES complaint. It would up to the user to delete the index to allow the committer apply the new configuration.
Or worse, what if someone changed the schema manually on ES and this new option overwrite such changes?
Once the mapping is indexed, it cannot be changed unless the index is deleted. You can however add new mappings (as it hasn't been indexed) but this request is to just apply the initial mapping before it is indexed.
IMHO, none of the issues are of concern. However, the current method of applying an initial index mapping manually is working fine. It's just an extra step that could be added as a feature if you think it is worth it. Either way, I am very impressed with this product and really appreciate your support. I will continue to let you know if I run into any other issues. Thanks again.
from committer-elasticsearch.
Many thanks for your great feedback. I am sold. :-)
from committer-elasticsearch.
I think there is a better way of doing this... I have been using templates to specify my mappings which provides the functionality described above without having to send any initial mappings. I am not sure if you have begun work on this but may want to reconsider. Thanks
from committer-elasticsearch.
Thanks for the update. If there is a better way then we'll forget this. Can you elaborate on your template solution?
from committer-elasticsearch.
I think it is safe to forget about this request because in ES you can specify your mappings and settings in a template. I am able to accomplish 1-4 above with the following template:
PUT _template/wmsearch
{
"template": "wmsearch",
"settings": {
"index.mapping.total_fields.limit": 1000,
"analysis": {
"analyzer": {
"custom": {
"type": "custom",
"tokenizer": "custom_token",
"filter": [
"lowercase"
]
},
"custom2": {
"type": "custom",
"tokenizer": "keyword",
"filter": [
"lowercase"
]
}
},
"tokenizer": {
"custom_token": {
"type": "ngram",
"min_gram": 1,
"max_gram": 30
}
}
}
},
"mappings": {
"Documents": {
"properties": {
"content": {
"type": "text",
"term_vector" : "with_positions_offsets",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"title": {
"type": "text",
"term_vector" : "with_positions_offsets",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"document": {
"properties": {
"reference": {
"type": "text",
"index_options": "offsets",
"analyzer": "custom",
"term_vector" : "with_positions_offsets",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"filename": {
"type": "text",
"index_options": "offsets",
"analyzer": "custom",
"term_vector" : "with_positions_offsets",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"generatedTitle": {
"type": "text",
"index_options": "offsets",
"analyzer": "custom",
"term_vector" : "with_positions_offsets",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"collector": {
"properties": {
"filesize": {
"type": "long",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"lastmodified": {
"type": "long",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"scope" : {
"type" : "nested",
"properties" : {
"level" : {
"type" : "integer"
},
"ancestors" : {
"type" : "keyword",
"index" : "true"
},
"value" : {
"type" : "keyword",
"index" : "true"
},
"order" : {
"type" : "integer"
}
}
}
}
}
}
}
With the above template saved, whenever I delete and re-index the above settings and mappings are automatically applied for "wmsearch". Therefore, I don't need to send an initial call to ES to set the above when re-indexing (deleting and creating).
I am not sure if anyone else has a use case for sending an initial settings/mappings or other call to ES prior to indexing, but mine are handled with the template feature of ES.
Let me know if you need more detail. Thanks.
from committer-elasticsearch.
Related Issues (20)
- id is too long, must be no longer than 512 bytes but was: 520 HOT 14
- No error for connecting to elasticsearch even though elasticsearch service is stopped? HOT 3
- Committer for AWS Elastic Search HOT 4
- Feature request - provide refresh and wait for active shards as configurables
- pipeline parameter for Elasticsearch 5+ HOT 2
- Message: listener timeout after waiting for [30000] ms HOT 1
- Narconix Collecttor is not committed to Elastic search.
- Naroconix have any one used for siteminder url what additional setting is required? HOT 1
- Reversed logic in the JSON field name matcher HOT 2
- Caused by: java.lang.IllegalArgumentException: Negative buffer size HOT 1
- Not able to commit the processed items to Elastic search using norconex file system collector HOT 1
- <typeName> should be set to not requried HOT 2
- Remove warning “Specifying types in bulk requests is deprecated.” HOT 2
- Deletion not working when a non "document.reference" value is used HOT 2
- Removal of mapping types HOT 1
- Committer closed without sending any documents HOT 1
- ElasticSearch Committer Error HOT 14
- WARN - unknown role HOT 2
- Feature request - support secure elasticsearch apikeys index
- utf-8 unicode HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from committer-elasticsearch.