allegro / akubra Goto Github PK
View Code? Open in Web Editor NEWSimple solution to keep a independent S3 storages in sync
License: Apache License 2.0
Simple solution to keep a independent S3 storages in sync
License: Apache License 2.0
In order to determine semi-privacy for the bucket, akubra is requesting bucket metadata on every request. But unless there is no X-Untrusted
header in request, we can assume that request is trusted so semi-privacy rules will not apply. So please limit SBI calls to only those requests, which have set X-Untrusted
header
akubra try to open /dev/log, even if syslog is disabled:
root@37545add92a4:/home# cat /etc/akubra/akubra.config |grep syslog
# syslog: LOG_LOCAL1 # default: LOG_LOCAL1
# syslog: LOG_LOCAL2 # default: LOG_LOCAL2
# syslog: LOG_LOCAL3 # default: LOG_LOCAL3
root@37545add92a4:/home# /usr/local/bin/akubra -c /etc/akubra/akubra.config
2018/10/11 13:48:24 Akubra (0.1.1d-dev version)
2018/10/11 13:48:24 Improperly configured Unix syslog delivery error
end of strace:
[pid 276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] <... connect resumed> ) = -1 ENOENT (No such file or directory)
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] close(5) = 0
[pid 276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
[pid 276] close(5) = 0
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... socket resumed> ) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ENOENT (No such file or directory)
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] close(5) = 0
[pid 276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... socket resumed> ) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/dev/log"}, 11 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] <... connect resumed> ) = -1 ENOENT (No such file or directory)
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] close(5) = 0
[pid 276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] <... socket resumed> ) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... setsockopt resumed> ) = 0
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/syslog"}, 18) = -1 ENOENT (No such file or directory)
[pid 276] close(5) = 0
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... socket resumed> ) = 5
[pid 276] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] <... setsockopt resumed> ) = 0
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] connect(5, {sa_family=AF_UNIX, sun_path="/var/run/log"}, 15) = -1 ENOENT (No such file or directory)
[pid 276] close(5) = 0
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] close(3 <unfinished ...>
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... close resumed> ) = 0
[pid 276] openat(AT_FDCWD, "/etc//localtime", O_RDONLY) = 3
[pid 276] read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 127
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 276] read(3, <unfinished ...>
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 276] <... read resumed> "", 4096) = 0
[pid 276] close(3) = 0
[pid 276] write(2, "2018/10/11 13:48:41 Akubra (0.1."..., 482018/10/11 13:48:41 Akubra (0.1.1d-dev version)
) = 48
[pid 276] write(2, "2018/10/11 13:48:41 Improperly c"..., 69 <unfinished ...>
[pid 277] <... nanosleep resumed> NULL) = 0
[pid 277] nanosleep({tv_sec=0, tv_nsec=20000}, 2018/10/11 13:48:41 Improperly configured Unix syslog delivery error
It prevent running akubra in default docker environment.
goroutine 81068 [running]:
runtime.throw(0x7d4709, 0x21)
/home/local/ALLEGROGROUP/michal.jarco/go/src/runtime/panic.go:566 +0x95 fp=0xc4206c5bf0 sp=0xc4206c5bd0
runtime.mapaccess1_faststr(0x78c760, 0xc4201db080, 0x7cc461, 0xe, 0xc4201302d0)
/home/local/ALLEGROGROUP/michal.jarco/go/src/runtime/hashmap_fast.go:201 +0x4f3 fp=0xc4206c5c50 sp=0xc4206c5bf0
net/textproto.MIMEHeader.Get(0xc4201db080, 0x7cc461, 0xe, 0x748f20, 0xc420153440)
/home/local/ALLEGROGROUP/michal.jarco/go/src/net/textproto/header.go:33 +0x75 fp=0xc4206c5c88 sp=0xc4206c5c50
net/http.Header.Get(0xc4201db080, 0x7cc461, 0xe, 0x1, 0xc420153401)
/home/local/ALLEGROGROUP/michal.jarco/go/src/net/http/header.go:39 +0x3f fp=0xc4206c5cc0 sp=0xc4206c5c88
github.com/allegro/akubra/httphandler.(*responseMerger).synclog(0xc420013840, 0xc420455a40, 0xc4201db0b0)
/home/local/ALLEGROGROUP/michal.jarco/osroot/src/github.com/allegro/akubra/httphandler/response_merger.go:41 +0x177 fp=0xc4206c5dc8 sp=0xc4206c5cc0
Akubra with sharding outputs three kinds of logs, each of them should be configurable.
Best place to have configuration for that is config yaml file in Logging section eg.:
Logging:
Synclog:
file: "/var/log/akubra-sync.log"
stderr: true
stdout: false
syslog: FACILITY //see `man syslog` facility section
Defaults would be:
Accesslog:
syslog: LOG_LOCAL0
Synclog:
syslog: LOG_LOCAL1
Mainlog:
syslog: LOG_LOCAL2
ClusterSynclog:
syslog: LOG_LOCAL3
We should identify user request with Authorization header and pick backends/clusters configuration.
Akubra should know it's version, and print it to the logs on startup. It will improve the usability.
We would like to have some metrics of akubra runtime. Eg.:
Dla usprawnienia monitoringu akubry powinniśmy agregować i eksponować następujące metryki:
Akubra panics during attempt of listing objects in a bucket.
Example:
s3cmd -c test.cfg ls s3://bucket
WARNING: Retrying failed request: /?delimiter=%2F (Remote end closed connection without response)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?delimiter=%2F (Remote end closed connection without response)
WARNING: Waiting 6 sec...
Akubra panics because backend responds with:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>bucket</Name>
<Prefix/>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>true</IsTruncated>
<CommonPrefixes>
<Prefix>prefix_1/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>prefix_2/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>prefix_3/</Prefix>
</CommonPrefixes>
...
<Marker/>
<NextMarker>...</NextMarker>
</ListBucketResult>
Response doesn't contain Contents
field which is required by Akubra.
All gzipped files sync operations ends up with content-length
related error. Please investigate and fix
--disable=gotype \
--disable=dupl \
--deadline=600s \
--enable=goimports \
--vendor \
--fast
config/config.go:141::warning: Potential file inclusion via variable,MEDIUM,HIGH (gosec)
config/config.go:174::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:175::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:212::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:220::warning: Errors unhandled.,LOW,HIGH (gosec)
config/config.go:229::warning: Errors unhandled.,LOW,HIGH (gosec)
log/sql/connection.go:86::warning: Errors unhandled.,LOW,HIGH (gosec)
Makefile:8: recipe for target 'lint' failed
make: *** [lint] Error 1
It help to add --disable=gosec to gometalinter parameters.
Example dockerfile to try to build:
FROM golang:1.11
ENV GOROOT /usr/local/go/
RUN mkdir -p /go/src/github.com/allegro/
RUN cd /go/src/github.com/allegro/ && git clone https://github.com/allegro/akubra.git
WORKDIR /go/src/github.com/allegro/akubra
RUN make build
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.