Feature request: Stop crawl at time

[samnissen]

Feature request: Stop crawl at time

[svenaas]

We would benefit from :max_pages or :max_time options, especially in development and test environments.

[stewartmckee]

If the exception is raised then would you want the whole crawl to stop at that point?

I think you get the same from max_pages by submitting crawl_limit, there is also a crawl_limit_by_page boolean which i think is false by default. crawl_limit is the max number of urls, and if crawl_limit_by_page is set to true then the crawl_limit only applies to text/html content.

Like the idea of max_time though, hadn't thought of that before, thinking that would set a datetime and include that date into the within_crawl_limits to check if it has passed, so could also consume a stop_at datetime. max_time would just do the arithmetic for you.

[samnissen]

Yes, I think raising the error, breaking, or returning should stop the crawl as the default.

Wasn't aware of the crawl_limit – will check that out thank you.

As for max_time, I'm thinking that would probably be an integer, whereas something like stop_at could be a datetime.