Delay consumption of new files

[jonaswinkler]

Delay consumption of new files

[jonaswinkler]

If you scan the file somewhere else and move it into the consumption folder afterwards, does that work? That would mean the consumer is trying to read the file while its still being written, resulting in the error about no pages in the PDF.

[bavarialogy]

It's not consuming any PDF anymore (not even new ones) now. I also restarted the containers just to be sure. The following exception occurs:

`tuple index out of range : Traceback (most recent call last):
File "/usr/src/paperless/src/documents/consumer.py", line 173, in try_consume_file
classifier=classifier
File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 179, in send
for receiver in self._live_receivers(sender)
File "/usr/local/lib/python3.7/site-packages/django/dispatch/dispatcher.py", line 179, in
for receiver in self._live_receivers(sender)
File "/usr/src/paperless/src/documents/signals/handlers.py", line 127, in set_tags
matched_tags = matching.match_tags(document.content, classifier)
File "/usr/src/paperless/src/documents/matching.py", line 36, in match_tags
predicted_tag_ids = classifier.predict_tags(document_content)
File "/usr/src/paperless/src/documents/classifier.py", line 224, in predict_tags
tags_ids = self.tags_binarizer.inverse_transform(y)[0]
File "/usr/local/lib/python3.7/site-packages/sklearn/preprocessing/label.py", line 1017, in inverse_transform
if yt.shape[1] != len(self.classes):
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
res = f(*task["args"], **task["kwargs"])
File "/usr/src/paperless/src/documents/tasks.py", line 68, in consume_file
override_tag_ids=override_tag_ids)
File "/usr/src/paperless/src/documents/consumer.py", line 187, in try_consume_file
raise ConsumerError(e)
documents.consumer.ConsumerError: tuple index out of range`

[jayme-github]

If you scan the file somewhere else and move it into the consumption folder afterwards, does that work? That would mean the consumer is trying to read the file while its still being written, resulting in the error about no pages in the PDF.

This is actually a pretty common issue when using watchdog as it does not (and will not) expose close_write events (gorakhargosh/watchdog#184). Especially with larger files (in my case from a Brother document scanner) one will run into this with almost every document.

It's not consuming any PDF anymore (not even new ones) now. I also restarted the containers just to be sure. The following exception occurs:

That's probably unrelated, I just created #47 for that.

[totti4ever]

This is actually a pretty common issue when using watchdog as it does not (and will not) expose close_write events (gorakhargosh/watchdog#184). Especially with larger files (in my case from a Brother document scanner) one will run into this with almost every document.

@jayme-github, how did you solve this? I am running a while loop, checking if file_size still increases. It is still prone to errors as sometimes (large documents or network issues) my scanner apparently caches more, most of the time less. I guess there are smarter ways? :-)

[jayme-github]

@jayme-github, how did you solve this? I am running a while loop, checking if file_size still increases. It is still prone to errors as sometimes (large documents or network issues) my scanner apparently caches more, most of the time less. I guess there are smarter ways? :-)

I switched to inotify ;-)

[totti4ever]

Hmmm and then just use close_write? Damn, I should use that instead of create/move, too