ravern/gollum

Robots.txt parser and fetcher for Elixir

Gollum

Robots.txt parser with caching. Modelled after Kryten. Docs can be found here.

Usage

Call Gollum.crawlable?/3 to obtain whether a certain URL is permitted for the specified user agent.

iex> Gollum.crawlable?("hello", "https://google.com/")
:crawlable
iex> Gollum.crawlable?("hello", "https://google.com/m/")
:uncrawlable

Gollum is an OTP app (For the cache) so just remember to specify it in the extra_applications key in your mix.exs to ensure it is started.

Gollum allows for some configuration in your config.exs file. The following shows their default values. They are all optional.

config :gollum,
  name: Gollum.Cache, # Name of the Cache GenServer
  refresh_secs: 86_400, # Amount of time before the robots.txt will be refetched
  lazy_refresh: false, # Whether to setup a timer that auto-refetches, or to only refetch when requested
  user_agent: "Gollum" # User agent to use when sending the GET request for the robots.txt

Author

Ravern Koh - <[email protected]>

gollum's Issues

Compile-time warning in Elixir 1.9

Compiling Gollum with Elixir 1.9 shows the following warning:

warning: function init/1 required by behaviour GenServer is not implemented (in module Gollum.Cache).

We will inject a default implementation for now:

    def init(init_arg) do
      {:ok, init_arg}
    end

You can copy the implementation above or define your own that converts the arguments given to GenServer.start_link/3 to the server state.

  lib/gollum/cache.ex:1: Gollum.Cache (module)

Generated gollum app

Update HTTPoison

Gollum requires an old (0.x) version of HTTPoison. Since the latest version is somewhere around 1.5, this can cause dependency conflicts.