Not getting past validity check

[malloydelacroix]

Not getting past validity check

No, the subfolders weren't even created.

…

On 18 March 2018 at 04:01, MalloyDelacroix ***@***.***> wrote: Was content being downloaded during this time? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG6OqTmjizqLJG6JfrwwEOInfgYHFcQqks5tfU92gaJpZM4Sujdm> .

[MalloyDelacroix]

You did have a connection error in this log, about a min. before you stopped the downloader. This is the problem that you were having before, but was fixed yesterday. I believe it was handled properly and the extraction continued.

I think you may be confusing the regular operation for an error. When the downloader is run, an object is verified, then passed on the extractor. The extractor extracts each post available for this object according to the settings that you set. Content is not downloaded until each post has been extracted for the object. The progress bar shows the progress of extraction, but only at the object level. So the bar only moves for every object that is completely extracted. This is why it seems to be stuck at 50%.

According to your log, the subreddit being extracted when the connection error occurred contained 4076 content items that were to be downloaded. This indicates that the extractor was most likely working properly, it had just not reached the download phase yet.

It is taking a long time to extract content to download, but this is most likely because you have set the post limit at 1000. It should not be a problem to extract that many posts, but it will most likely take a long time to run, depending on your internet connection. To test that this is in fact the problem, please set the post limit to 10 items and see if the problem persists.

Yes, that seems to work. I'll just let it run longer from now on.

Is there any way to download each post in turn rather than loading a bulk lot before processing?

I just need to be able to create extractors for all the domain errors I'm getting.

Also, in your settings dialog, it should say "Enforce" not "Inforce".

Keep up the great work.

Edit: When I set the value back to 1000, it now only seems to download new items since the first 10 (one), not all the historical ones beyond the first 10.

[MalloyDelacroix]

Thanks for pointing that out, I had never noticed that typo.

There's not a way to download the posts immediately after extraction at this time, but while I was looking through the code to solve this issue I had an idea of how to implement that. I think that would provide a better experience and I am going to try to get that feature into the next minor update.

[MalloyDelacroix]

In regards to your edit:

When you downloaded the 10 items, it set your post date limit to the last date extracted.

Since you only have two subreddits in the list I would recommend opening the subreddit settings dialog for each of them and unchecking "Restrict by Date". This will allow it to download as many posts as possible regardless of date. It will be automatically re-enabled with the last extraction date again after a download.