trandoshan-io/scheduler

Go orchestrator process used to schedule website parsing

scheduler's Issues

Fix duplicate persist because of latency

Because it take times for the persister to save content It is possible that the same version of a website is schedule for crawling even if it has been already crawled. This is because the API does not know that the website has been crawled before the persister has saved it.

This should be fix to remote the consistency check done in the persister

Complete readme

Refactor to use elasticsearch

Allow duplicate website crawling

Currently, the scheduler does not allow the crawling of an already crawled website.

This behavior has to change to support the upcoming website snapshot feature.

The scheduler should use the crawling date and compare to a threshold to determinate if crawling has to be done

Remove forbidden extensions check

Currently the scheduler prevent the crawler from parsing unwanted resources by doing a suffix comparison between the URL and a list of forbidden extensions (taken from database)

This check is partially useless and it would be great to remove it when trandoshan-io/crawler#5 is merged.

(Don't forget to to appropriate change in trandoshan-parent / API)

Add travis ci support

Remove now useless .onion check

Now done in crawler since: trandoshan-io/crawler#7

Update go version