Add support for S3-compatible services

[benbjohnson]

Add support for S3-compatible services

[bobek]

Hello, first of all, thank you for a great project. I wanted to try litestream with MinIO. Unfortunately it is not that simple as one need to specify endpoint (otherwise go-aws will try to create it from bucket and region) and also path style has to be forced. Otherwise bucket will be passed as prefix of the hostname, which will not work with MinIO.

Changes have been based on https://docs.min.io/docs/how-to-use-aws-sdk-for-go-with-minio-server.html

I am attaching patch and configuration file, which I have ended with, as an inspiration (happy to open PR, if you want):

• 0001-Hack-in-MinIO-support.patch.txt
• litestream.yml.txt
• pushed to fork for a simpler diffing -- https://github.com/bobek/litestream/tree/minio

[benbjohnson]

I added support and wrote up some brief docs in the PR for running Litestream with MinIO, B2, & GCS: #66

The code was a bit more complicated as I wanted to support easily running these using the replica URL instead of requiring folks to set up a configuration file. Feedback is welcome if you get a chance to try it. I'll post guides for each backend once the v0.3.3 is cut.

[bwghughes]

+1 - Thanks Ben !! Same on Backblaze, no way of explicitly calling out region, as such:

litestream replicate db.sqlite3 s3://s3.us-west-000.backblazeb2.com/octo-litestream

Errors with:

litestream v0.3.2
initialized db: /Users/ben/db.sqlite3
replicating to: name="s3" type="s3" bucket="s3.us-west-000.backblazeb2.com" path="octo-litestream" region="us-west1"
/Users/ben/db.sqlite3(s3): monitor error: RequestError: send request failed
caused by: Get "https://s3.us-west1.amazonaws.com/s3.us-west-000.backblazeb2.com?delimiter=%2F&prefix=octo-litestream%2Fgenerations%2F": dial tcp: lookup s3.us-west1.amazonaws.com: no such host

[oderwat]

I can confirm that the minio patch works for me too with an external https minio server. Very nice and just what I needed.

[benbjohnson]

Thanks for all the testing everyone. I'm trying to wrap up Windows support (#7) first and then I'll get this functionality working and tested and merged in.

[oderwat]

I added my observations in #66