allegro/akubra

Simple solution to keep a independent S3 storages in sync

akubra's Issues

Suppress number of storage bucket index requests

In order to determine semi-privacy for the bucket, akubra is requesting bucket metadata on every request. But unless there is no X-Untrusted header in request, we can assume that request is trusted so semi-privacy rules will not apply. So please limit SBI calls to only those requests, which have set X-Untrusted header

Need of /dev/log, even if syslog is disabled

akubra try to open /dev/log, even if syslog is disabled:

root@37545add92a4:/home# cat /etc/akubra/akubra.config |grep syslog
  #  syslog: LOG_LOCAL1  # default: LOG_LOCAL1
  #  syslog: LOG_LOCAL2  # default: LOG_LOCAL2
  #  syslog: LOG_LOCAL3  # default: LOG_LOCAL3

root@37545add92a4:/home# /usr/local/bin/akubra -c /etc/akubra/akubra.config                    
2018/10/11 13:48:24 Akubra (0.1.1d-dev version)
2018/10/11 13:48:24 Improperly configured Unix syslog delivery error

end of strace:

[pid   276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] <... connect resumed> )     = -1 ENOENT (No such file or directory)
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] close(5)                    = 0
[pid   276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
[pid   276] close(5)                    = 0
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... socket resumed> )      = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ENOENT (No such file or directory)
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] close(5)                    = 0
[pid   276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... socket resumed> )      = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] <... connect resumed> )     = -1 ENOENT (No such file or directory)
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] close(5)                    = 0
[pid   276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] <... socket resumed> )      = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... setsockopt resumed> )  = 0
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
[pid   276] close(5)                    = 0
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... socket resumed> )      = 5
[pid   276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] <... setsockopt resumed> )  = 0
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ENOENT (No such file or directory)
[pid   276] close(5)                    = 0
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] close(3 <unfinished ...>
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... close resumed> )       = 0
[pid   276] openat(AT_FDCWD, "/etc//localtime", O_RDONLY) = 3
[pid   276] read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 127
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   276] read(3,  <unfinished ...>
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid   276] <... read resumed> "", 4096) = 0
[pid   276] close(3)                    = 0
[pid   276] write(2, "2018/10/11 13:48:41 Akubra (0.1."..., 482018/10/11 13:48:41 Akubra (0.1.1d-dev version)
) = 48
[pid   276] write(2, "2018/10/11 13:48:41 Improperly c"..., 69 <unfinished ...>
[pid   277] <... nanosleep resumed> NULL) = 0
[pid   277] nanosleep({tv_sec=0, tv_nsec=20000}, 2018/10/11 13:48:41 Improperly configured Unix syslog delivery error

It prevent running akubra in default docker environment.

Set more than one backend to maintenance mode

Supports multiple backends compatible with API S3 such as Ceph, AWS S3, Minio, ...

Akubra panics when one of backends has hickups

1. Lot's of TIME_WAITS on machines
2. fatal error: concurrent map read and map write

goroutine 81068 [running]:
runtime.throw(0x7d4709, 0x21)
/home/local/ALLEGROGROUP/michal.jarco/go/src/runtime/panic.go:566 +0x95 fp=0xc4206c5bf0 sp=0xc4206c5bd0
runtime.mapaccess1_faststr(0x78c760, 0xc4201db080, 0x7cc461, 0xe, 0xc4201302d0)
/home/local/ALLEGROGROUP/michal.jarco/go/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc4206c5c50 sp=0xc4206c5bf0
net/textproto.MIMEHeader.Get(0xc4201db080, 0x7cc461, 0xe, 0x748f20, 0xc420153440)
/home/local/ALLEGROGROUP/michal.jarco/go/src/net/textproto/header.go:33 +0x75 fp=0xc4206c5c88 sp=0xc4206c5c50
net/http.Header.Get(0xc4201db080, 0x7cc461, 0xe, 0x1, 0xc420153401)
/home/local/ALLEGROGROUP/michal.jarco/go/src/net/http/header.go:39 +0x3f fp=0xc4206c5cc0 sp=0xc4206c5c88
github.com/allegro/akubra/httphandler.(*responseMerger).synclog(0xc420013840, 0xc420455a40, 0xc4201db0b0)
/home/local/ALLEGROGROUP/michal.jarco/osroot/src/github.com/allegro/akubra/httphandler/response_merger.go:41 +0x177 fp=0xc4206c5dc8 sp=0xc4206c5cc0

Add syslog/stdout/file logging switch

Akubra with sharding outputs three kinds of logs, each of them should be configurable.

• synclog - is where all inconsistencies between cluster backends are stored
• accesslog - keeps information as known http servers access.log does (request date, uri, user-agent, response code etc.)
• mainlog - where we gather events from akubra runtime (like errors/panics etc)

Best place to have configuration for that is config yaml file in Logging section eg.:

Logging:
    Synclog:
           file: "/var/log/akubra-sync.log"
           stderr: true
           stdout: false
           syslog: FACILITY //see `man syslog` facility section

Defaults would be:

Accesslog:
    syslog: LOG_LOCAL0
Synclog:
    syslog: LOG_LOCAL1
Mainlog:
    syslog: LOG_LOCAL2
ClusterSynclog:
    syslog: LOG_LOCAL3

Add support for different clusters configuration

We should identify user request with Authorization header and pick backends/clusters configuration.

Application version printed to the logs on startup

Akubra should know it's version, and print it to the logs on startup. It will improve the usability.

Runtime metrics and monitoring

We would like to have some metrics of akubra runtime. Eg.:
Dla usprawnienia monitoringu akubry powinniśmy agregować i eksponować następujące metryki:

1. Number of requests per second

• split by method
• split by reponse status code
• for each akubra's backend

2. Response times

• split by method
• split by reponse status code
• for each akubra's backend

3. Number of non http errors
4. Amount of allocated memory
5. Number of running goroutines

Error during listing objects in bucket

Akubra panics during attempt of listing objects in a bucket.
Example:

s3cmd -c test.cfg ls s3://bucket                                     
WARNING: Retrying failed request: /?delimiter=%2F (Remote end closed connection without response)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?delimiter=%2F (Remote end closed connection without response)
WARNING: Waiting 6 sec...

Akubra panics because backend responds with:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>bucket</Name>
  <Prefix/>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>true</IsTruncated>
  <CommonPrefixes>
    <Prefix>prefix_1/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>prefix_2/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>prefix_3/</Prefix>
  </CommonPrefixes>
  ...
  <Marker/>
  <NextMarker>...</NextMarker>
</ListBucketResult>

Response doesn't contain Contents field which is required by Akubra.

Pluggable structured logging

Fix syncing of gzipped files

All gzipped files sync operations ends up with content-length related error. Please investigate and fix

Don't compile with current version of golint

--disable=gotype \
--disable=dupl \
--deadline=600s \
--enable=goimports \
--vendor \
        --fast
config/config.go:141::warning: Potential file inclusion via variable,MEDIUM,HIGH (gosec)
config/config.go:174::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:175::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:212::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:220::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:229::warning: Errors unhandled.,LOW,HIGH (gosec)
log/sql/connection.go:86::warning: Errors unhandled.,LOW,HIGH (gosec)
Makefile:8: recipe for target 'lint' failed
make: *** [lint] Error 1

It help to add --disable=gosec to gometalinter parameters.

Example dockerfile to try to build:

FROM golang:1.11
ENV GOROOT /usr/local/go/
RUN mkdir -p /go/src/github.com/allegro/
RUN cd /go/src/github.com/allegro/ && git clone https://github.com/allegro/akubra.git
WORKDIR /go/src/github.com/allegro/akubra
RUN make build