Incremental Crawling

[taganaka]

Incremental Crawling

[taganaka]

Not implemented yet.
Working on a set of new features where you can specify an expire time for each page.
Whether the page is expired, it will downloaded again, by overwriting the old stored doc

As for now a working alternative could be to clean redis-bloomfilter data and override the storage.exists? in your storage adapter to return false even though the page already exists:

https://github.com/taganaka/polipus/blob/master/lib/polipus/storage/mongo_store.rb#L32-L37

module Polipus
  module Storage
    class MongoStore < Base
      def exists? page
        false
      end
    end
  end
end

[taganaka]

@neuralnw
You can try this branch:

https://github.com/taganaka/polipus/tree/incremental_crawling

Major changes:

• An url part of the initial seeder is always re-downloaded, no matter what
• A new option is now available:

# Page TTL: mark a page as expired after ttl_page seconds
# Default: nill
:ttl_page => 60

• Mongo store only:
  page.fetched_at is populated by BSON::ObjectId(_id).generation_time only if page.fetched_at is nil (for legacy data)

Going to merge in the next few days

[nengine]

Hi Thanks, I tested and it is working great. Regards.