How to setup in a cluster environment?

[dbuarque]

How to setup in a cluster environment?

[taganaka]

Hi!
A minimal cluster setup is actually really easy.

1. A Redis endpoint where all of the instances running polipus have access to
2. A page storage endpoint (MongoDB or S3) where all of the instances running polipus have access to

You can even have Redis and Mongo running on the same instance, this really depend on the size of your crawling session

Once storage and Redis is configured is just a matter of running multiple process of polipus on multiple instances, just deploying the same software. The underlying use of Redis to dispatch urls to crawl does the rest.

Take a look at this example:
https://github.com/taganaka/polipus/blob/master/examples/basic.rb

Just set the right connection endpoint for Redis and Mongo end you will be fine

[dbuarque]

Awesome! Thanks a lot! Polipus is really awesome!