innoq/feedworker

feedworker

Library for processing RSS/Atom feeds.

Setup

Add this to you project.clj:

 [feedworker "0.1.0"]

What's this?

The purpose of feedworker is to make processing feeds easy. It manages which entries have been processed already and schedules periodic processing.

It's configured with a Clojure data structure like this:

{:workers
 {:dilbert {:url "http://feed.dilbert.com/dilbert/most_popular?format=xml"
            :handler example-handler
            :processing-strategy :at-most-once
            :interval 10000}}
 :processed-entries-dir "processedentries"
 :cleanup {:keep 10 :max 200}
 :metrics {:http {:port 8080
                  :path "/feedworker/metrics"}}}

All durations are given in milliseconds. The actual processing of each feed entry is done by the given handler (here: example-handler). It's a function of three arguments:

• A single feed entry as parsed by feedparser-clj.
• The ID of the worker (e.g. :dilbert).
• The entire configuration.

Paginated feeds are not properly supported, yet. Also, feedworker needs to be able to track which entries it already processed. By default, this is currently done by hashing each entry to generate a unique id which is then stored on disk. This means, that feedworker cannot handle feeds that contain entries which are exact duplicates (not only regarding the content but considering all metadata, e.g. timestamps). Optionally, the configuration of a worker may contain an :entry-identifier function which is then applied to each entry to determine the id. Use ':uri' to access the unique id of an atom feed entry.

License

Copyright 2014 innoQ Deutschland GmbH. Published under the Apache 2.0 license.

feedworker's Issues

support conditional get with etag when loading feed

When polling a feed that has an ETag, the last seen ETag should be stored and used to perform a conditional GET.

Worker specific way to identify entries

In order to keep track of which entries have been processed already, feedworker needs a unique ID for each feed entry. This is currently obtained by hashing the entire entry. This causes entries to be reprocessed for every change (e.g. additional metadata, different timestamp format etc.).

It should be possible to configure a worker specific function that is used to identify a feed entry (e.g. :uri for atom feeds with unique entry IDs).

e-mail addresses are erroneously interpreted as mentions

a statuses message like ...

hello [email protected] world

... leads to a notification to [email protected] (i.e. it's interpreted as @example

Support paginated atom feeds

innoq/statuses#147

Provide a config for innoq/statuses

As http://github.com/innoq/statuses is an open-source project, we should think about linking the two projects publicly as we do use it this way internally.