stairlin/kargo

[BETA] The plugin-driven CLI for backing up & restoring data.

🚢 Kargo - Backup & Restore

Kargo is a CLI written in Go for transporting everything everywhere, which makes it the ideal companion to backup & restore data.

The design is unapologetically taken from Telegraf ❤️ with its plugin system. That allows developers to create adapters for any kind of databases, storages, or notifiers.

For example, you might want to backup a Couchbase database, compress and encrypt the backup, and store it on Amazon S3. You also may want to send a Pagerduty alert when a backup failed or timed out. Kargo can be seen as a database-agnostic backup tool.

Kargo is plugin-driven and has the concept of 4 distinct plugins:

1. Source Plugins backup and restore a source, such as a database, a folder, or anything else.
2. Processor Plugins encode/decode backup data (compress, cipher, ...).
3. Store Plugins persist backups and can be queried.
4. Notification Plugins send notifications on success/failure.

New plugins are designed to be easy to contribute, we'll eagerly accept pull requests and will manage the set of plugins that Kargo supports.

Installation

You can download the binaries from the release section.

From Homebrew (macOS):

1. Run brew tap stairlin/homebrew-tap
2. Run brew install stairlin/tap/kargo

From Source:

Kargo is currently tested on go 1.9.

1. Install Go
2. Setup your GOPATH
3. Run go get -d github.com/stairlin/kargo
4. Run cd $GOPATH/src/github.com/stairlin/kargo
5. Run go get -d ./...
6. Run go run main.go

Usage

Backup & restore data:

kargo backup
kargo restore my_backup_key

List backups:

kargo list
kargo list --from 2018-02-14
kargo list --prefix foo --to 2017-12-31
kargo list --pattern ^[a-z]{3}.*
kargo list --limit 50

Backup with a custom key name

kargo backup --key dat_snapshot

Pull a backup to local disk & restore data:

kargo pull my_backup_key
kargo restore --local my_backup_key

Help:

kargo help

Configuration

Environment variables

Environment variables can be used anywhere in the config file, simply prepend them with $. The variable must be within quotes (ie, "$STR_VAR"). Numbers, booleans, and other data types are not supported.

Configuration file locations

The location of the configuration file can be set via the --config command line flag.

Order on which Kargo looks for a configuration file:

1. --config flag
1. kargo.toml in the working directory
2. $KARGO_CONFIG environment variable
4. default location `/etc/kargo/kargo.toml`

Show the current configuration:

kargo config

Example of a configuration file.

[agent]
  debug = true

[source.dir]
  path = "/oh/my/dir/"

[processors.gzip]

[processors.cipher]
  default = 0
  keys = ["my_base64_key", "another_key"]

[storage.s3]
  id = "foo"
  secret = "bar"
  token = ""
  bucket = "my_bucket"
  region = "eu-central-1"

[[notifiers.pagerduty]]
  key = "api_key"

[[notifiers.slack]]
  username = "My Project"
  url = "https://hooks.slack.com/services/foo"

Plugins

Sources

1. Consul
2. Couchbase
3. Directory
4. FoundationDB
5. InfluxDB
6. PostgreSQL

Storages

1. Filesystem
2. Amazon S3

Processors

1. Cipher
2. GZip

Notifiers

1. Pagerduty
2. Slack

Dependencies

Most dependencies are packed into the binary, such as AWS S3, gzip, Pagerduty, etc. However, Source plugins mainly rely on shell commands to work, so these dependencies must be installed separately and set to the $PATH.

Internal design

Kargo piggyback on the powerful Go I/O library to keep the memory and disk footprint minimal. In most cases, data is being streamed from the source to the storage with no or minimal internal buffering. There are plugins, such as cipher that must work data by chunks for obvious reasons, so it will use a small buffer.

It is worth noting that most Source plugins cannot stream data right away. Indeed, they have to create a temporary file that contains the backup before. postgres is currently the only plugin that can stream data directly, thanks to pg_dump.

Contributing

This project is still unstable and not production ready. However, new plugins and bug fixes are welcome.

There is currently no guidelines, no red tape to contribute. Enjoy while it last. :)

License