brutuscat/medusa-crawler

The Official Medusa Crawler gem

medusa-crawler's Issues

skip_links_like does not match patterns in query strings

Steps:
Attempt to skip patterns in a query string during a crawl:

Medusa.crawl(url, depth_limit: 1, discard_page_bodies: true) do |medusa|
  medusa.skip_links_like(/sort=desc/)
  medusa.on_every_page do |page|
    puts page.url if /sort=desc/.match(page.url.to_s)
  end
end

Expected Results:

• no URLs are crawled that include the pattern in the query string

Actual Results:
URLs are crawled that include the pattern in the query string:

https://www.va.gov/vapubs/Search_action.cfm?formno=&tkey=&dType=1&SortBy=pub_num&sort=desc&oid=0
https://www.va.gov/vapubs/Search_action.cfm?formno=&tkey=&dType=1&SortBy=issue&sort=desc&oid=0
https://www.va.gov/vapubs/Search_action.cfm?formno=&tkey=&dType=1&SortBy=title&sort=desc&oid=0
https://www.va.gov/vapubs/Search_action.cfm?formno=&tkey=&dType=1&SortBy=rpch&sort=desc&oid=0

This seems like an oversight, rather than the desired behavior, and I can put in a PR to fix this. If this is the desired behavior, there should be another mechanism to skip links based on query string patterns/keys/values, etc.

Add example of crawling a private site with login form

support phantomjs

Look into queue mem issue

See chriskite/anemone#70

Implement storage module in medusa using moneta

Migrating this issue from Anemone, can we add a data accessor for this database? Thanks.