aig787/feeds

Feeds

Fetch indicators from feeds in various formats to push towards the storage of your choice.

Requirements

Feeds requires JDK 11 or greater

Running

Docker

CLI

docker run -v $CONF:/opt/sightingdb/conf/application.conf aig787/feeds:$VERSION

Compose

version: "3"
services:
  feeds:
    build: .
    volumes:
      - $CONF:/opt/sightingdb/conf/application.conf

Gradle

Run ./gradlew run from the project root directory

Outputs

Included Outputs

Class	Description
com.devo.feeds.output.LoggingOutputFactory	Log attributes to stdout
com.devo.feeds.output.SyslogOutputFactory	Send attributes to syslog Configuration: host (required): Syslog server port (required): Syslog port tags (required: e.g. ['misp.feed']): List of syslog tags, attributes will be sent with each chain (optional): Syslog TLS certificate keystore (optional): Client TLS keystore keystorePass (optional): Keystore password threads (default: 1): Number of output threads
com.devo.feeds.output.DevoOutputFactory	Send attributes to Devo Configuration: host (required): Devo Relay FQDN port (required): Devo Relay port chain (required): Devo Relay chain cert keystore (required): Devo keystore keystorePass (required): Devo keystore password threads (default: 1): Number of output threads
com.devo.feeds.output.KafkaOutputFactory	Send attributes and events to Kafka Configuration eventTopic (required): Topic where events will be written attributeTopic (required): Topic where attributes will be written properties (required): Map of Kafka producer configs. Must include bootstrap.servers

Custom Outputs

Arbitrary outputs can be configured if they are present on the classpath. Custom outputs require a fromConfig method that takes a typesafe config as input and returns a subclass of Output. See the Kafka output for an example.

Caches

Included Caches

Class	Description
com.devo.feeds.storage.InMemoryAttributeCacheFactory	Cache attributes in memory, cache is lost on restarts
com.devo.feeds.storage.FilesystemAttributeCacheFactory	Cache attributes on disk Configuration: path (required): Directory to store local cache

Custom Caches

Arbitrary caches can be configured if they are present on the classpath. Custom caches require a fromConfig method that takes a typesafe config as input and returns a subclass of AttributeCache.

Configuration

Reference configuration

Example configuration

feeds {
  misp {
    url = "https://localhost:4433"
  }
  # Number of http client threads
  http.client.threads = 5
  # How often to update feeds
  feedUpdateInterval = "1 hour"
  # How often to check config in MISP
  mispUpdateInterval = "1 hour"
  cache {
    class = com.devo.feeds.storage.FilesystemAttributeCacheFactory
    path = /tmp/feeds
  }
  outputs: [
    {
      class = com.devo.feeds.output.LoggingOutputFactory
    },
    {
      class = com.devo.feeds.output.DevoAttributeOutputFactory
      host = usa.elb.relay.logtrust.net
      port = 443
      chain = chain.crt
      keystore = devo-keystore.pkcs12
      keystorePass = changeme
    }
  ]
}