URL patching

[nengine]

URL patching

[taganaka]

you can use focus_crawl and use your logic to extract/patch all of the links that will be visited

crawler.focus_crawl do |page|
  page.links.map{ |link| URI.encode(URI.decode(link.to_s.gsub("%E2%80%93","%96"))) }.uniq
end

[nengine]

Ok. Thank you very much.

[nengine]

Is there an option for crawl delay?

[taganaka]

you can enable the sleeper plugin: https://github.com/taganaka/polipus/blob/master/lib/polipus/plugins/sleeper.rb

or

polipus.on_page_dowloaded {|page| sleep 1}

[nengine]

Hi thanks!