ReactorNotRestartable

[zanpen2000]

ReactorNotRestartable

[zanpen2000]

注释掉了 d.addBoth()方法，可以了

def start_crawl(rule, task_id):
    rule = dict2obj(rule)
    runner = CrawlerRunner(settings=settings)

    for r in rule.rules:
        r = dict2obj(r)
        runner.crawl(get_spider(rule.name)['class'], user_rule=r, task_id=task_id)
        # d.addBoth(lambda _: reactor.stop())

    reactor.run()

更正注释掉了 d.addBoth()方法不可行

因为会导致task只received而不accepted的情况发生，如，beat发送了4个任务，而你的CPU是4Core且没有做任何concurrency的设置（所以默认为4），那么，前面4个task都会成功执行，之后的task则会只received而不accepted

后来，实在没办法，试着升级了一下

pip install scrapy --upgrade

神奇的事情发生了，不再出现ReactorNotRestartable的提示了.

现在的代码是这个样子的

def start_crawl(rule, task_id):
    rule = dict2obj(rule)
    runner = CrawlerRunner(settings=settings)

    def crawl():
        for r in rule.rules:
            r = dict2obj(r)
            runner.crawl(get_spider(rule.name)['class'], user_rule=r, task_id=task_id)
        d = runner.join()
        d.addBoth(lambda _: reactor.stop())

    crawl()
    reactor.run()

[clarksun]

最新的1.0版本还是有这个问题, 把run.py最后一行process.start()移出for循环就可以了.
详见Pull request.