Comments (5)
Hi all. As an update, I realized the reason the index was stopping short was due to a detached Heroku dyno timing out after 24 hours. I increased the pagination size and was able to complete the job within 24 hours.
I am really interested in a way to limit the index job size. I have someone adding around 8,000 records per week, so if I could just set a job to run each night and index the last 10,000 records that would be perfect. Anybody think of a way to easily implement that?
from django-elasticsearch-dsl.
Yes it will iterate though every records, this could be a good thing to add a way to filter/order via the command for this kind of problem. There is a PR (#58) but i didn't really look into it yet.
from django-elasticsearch-dsl.
Ok thanks. I am having trouble indexing all of my 12 million records. It seems to stop around 9.5 million. I wonder if using queryset has anything to do with it? Should I write my own bulk update function or is it already doing that in the background?
from django-elasticsearch-dsl.
Is that same as #118?
from django-elasticsearch-dsl.
Yes, close in favor of #118
from django-elasticsearch-dsl.
Related Issues (20)
- How to connect Django Backend and Elastic Search under ver 8.8.1? HOT 1
- Multi model single index possiblity HOT 2
- Multi Index Elastic Search HOT 1
- Add knn documentation HOT 2
- range fields support
- Error: Unexpected queryset model
- Showing Progress During Populate or Rebuild Commands HOT 1
- Save Only in Elasticsearch HOT 1
- ModuleNotFoundError: No module named 'MyIndex'
- elasticsearch.helpers.BulkIndexError: 1 document(s) failed to index
- How to add a mapping on the Document class HOT 1
- Cleaning up failed indexes when using `--use-alias`
- Question : Save without indexing, index a list of objects HOT 1
- Shards/Replicas setting defaults to one when populate index command without creating the index first
- failed to index
- I get connection timeout, however ES logs seems okay
- Deprecation Warning for ES v7
- Invalid metadata HOT 2
- Compatibility with django-countries HOT 3
- Data filtering function
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from django-elasticsearch-dsl.