Use Timer Tasks to start crawling jobs concurrently

[my8100]

Use Timer Tasks to start crawling jobs concurrently

[my8100]

Please provide the screenshots of the detais of your tasks in the Timer Tasks page, as well as necessary logs.

[CoderZHV]

图二看出 task1 与 task2 交替执行 五秒执行一次 同一个task开始时间间隔十秒 但是 我创建task时设置的是同一个task 5秒执行一次

[my8100]

As you can see from the Job column, both task#1 and task#2 are being executed every 5 seconds, whereas the Start column indicates the actual start time of the crawling jobs.

Apparently, the Scrapyd server is too busy to start the job as soon as the task is executed, as there are 376 pending jobs in the final screenshot.

Also, see https://scrapyd.readthedocs.io/en/latest/config.html#max-proc to find out the maximum number of concurrent Scrapy process that will be started. And such info could be found when you launch the Scrapyd service:

2019-04-19T11:28:32+0800 [Launcher] Scrapyd 1.2.0 started: max_proc=16, runner='scrapyd.runner'

[my8100]

You also need to adjust poll_interval

The interval used to poll queues, in seconds. Defaults to 5.0. Can be a float, such as 0.2