adityaxdiwakar/broker-dl

Downloading system using Kafka as a Message Broker

broker-dl

broker-dl is a Kafka-based download thru notification system using LFTP.

Use Case

The typical use case of broker-dl is where one has a server receiving files (usually used as a high-speed cache in a datacenter) that must be synced (upon some criteria) to a local server (possibly through a configurable proxy). Files should not be considered transferred in the case of local server being offline, proxy being offline, or local server partially completing a download and failing (intermittent network failure, power failure, etc). Hence, Kafka is utilized as a message broker to send/receive notifications.

Design

The remote server monitors some directory or a service that generates/receives files (possibly using inotify), and produces a notification using cmd/brokerdl-producer. This produces a message sent to the configurable Kafka server. The client runs cmd/brokerdl-consumer as a running process to receive messages from the broker. Messages received are committed if and only if the transfer and subsequent move (from an incomplete to complete directory) is successful.