Populating index

[django-es]

Populating index

[caseydm]

Hi all. As an update, I realized the reason the index was stopping short was due to a detached Heroku dyno timing out after 24 hours. I increased the pagination size and was able to complete the job within 24 hours.

I am really interested in a way to limit the index job size. I have someone adding around 8,000 records per week, so if I could just set a job to run each night and index the last 10,000 records that would be perfect. Anybody think of a way to easily implement that?

[sabricot]

Yes it will iterate though every records, this could be a good thing to add a way to filter/order via the command for this kind of problem. There is a PR (#58) but i didn't really look into it yet.

[caseydm]

Ok thanks. I am having trouble indexing all of my 12 million records. It seems to stop around 9.5 million. I wonder if using queryset has anything to do with it? Should I write my own bulk update function or is it already doing that in the background?

[cocoakekeyu]

Is that same as #118?

[sabricot]

Yes, close in favor of #118