iby / docker-riak-cs Goto Github PK
View Code? Open in Web Editor NEWAWS S3 inside a Docker container
License: MIT License
AWS S3 inside a Docker container
License: MIT License
Any chance you would be willing to add a license to the code in this repository?
==========docker-compose.yml=============
riak-cs:
image: ianbytchek/riak-cs
container_name: riak-cs-amazon-s3
healthcheck:
timeout: 60s
retries: 10
environment:
- RIAK_CS_BUCKETS=riak
- RIAK_CS_KEY_ACCESS=CC2QOJQ4VJFR8USBS5Y3
- RIAK_CS_KEY_SECRET=dKoX0b8dKbSFS0lJ_euHAVdgHMPDuB9GDZgJ8Q==
ports:
- "8080:8080"
========docker logs===========
riak-cs-amazon-s3 | Waiting for riak kv service to startup… OK!
riak-cs-amazon-s3 | Starting Stanchion… OK!
riak-cs-amazon-s3 | Starting Riak CS… OK!
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | ############################################################
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Riak admin credentials, make note of them, otherwise you
riak-cs-amazon-s3 | will not be able to access your files and data. Riak
riak-cs-amazon-s3 | services will be restarted to take effect.
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Access key: CC2QOJQ4VJFR8USBS5Y3
riak-cs-amazon-s3 | Secret key: dKoX0b8dKbSFS0lJ_euHAVdgHMPDuB9GDZgJ8Q==
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | ############################################################
riak-cs-amazon-s3 |
riak-cs-amazon-s3 | Restarting Riak… OK!
riak-cs-amazon-s3 | Waiting for riak kv service to startup… OK!
riak-cs-amazon-s3 | Restarting Stanchion…Restarting Riak CS…Node '[email protected]' not responding to pings.
riak-cs-amazon-s3 | Node '[email protected]' not responding to pings.
riak-cs-amazon-s3 | OK!
riak-cs-amazon-s3 | Creating Riak CS buckets.
riak-cs-amazon-s3 | riak… Failed!
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 CRIT Supervisor running as root (no user in config file)
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/stanchion.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/riak-cs.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,371 WARN Included extra file "/entrypoint/configuration/supervisord/riak.conf" during parsing
riak-cs-amazon-s3 | 2018-02-19 09:35:48,374 INFO supervisord started with pid 4906
riak-cs-amazon-s3 | 2018-02-19 09:35:49,376 INFO spawned: 'riak' with pid 4909
riak-cs-amazon-s3 | 2018-02-19 09:35:49,378 INFO spawned: 'stanchion' with pid 4910
riak-cs-amazon-s3 | 2018-02-19 09:35:49,380 INFO spawned: 'riak-cs' with pid 4911
riak-cs-amazon-s3 | 2018-02-19 09:35:49,839 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:49,859 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,001 INFO success: riak entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,007 INFO spawned: 'stanchion' with pid 5248
riak-cs-amazon-s3 | 2018-02-19 09:35:51,013 INFO spawned: 'riak-cs' with pid 5249
riak-cs-amazon-s3 | 2018-02-19 09:35:51,440 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:51,492 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:53,494 INFO spawned: 'stanchion' with pid 5454
riak-cs-amazon-s3 | 2018-02-19 09:35:53,496 INFO spawned: 'riak-cs' with pid 5455
riak-cs-amazon-s3 | 2018-02-19 09:35:53,860 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:53,874 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:56,878 INFO spawned: 'stanchion' with pid 5660
riak-cs-amazon-s3 | 2018-02-19 09:35:56,879 INFO spawned: 'riak-cs' with pid 5661
riak-cs-amazon-s3 | 2018-02-19 09:35:57,254 INFO exited: stanchion (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:57,256 INFO gave up: stanchion entered FATAL state, too many start retries too quickly
riak-cs-amazon-s3 | 2018-02-19 09:35:57,256 INFO exited: riak-cs (exit status 1; not expected)
riak-cs-amazon-s3 | 2018-02-19 09:35:58,257 INFO gave up: riak-cs entered FATAL state, too many start retries too quickly
I'm following the directions to set up/configure and am not able to connect.
I've downloaded a fresh Docker image from DockerHub. Run it. Seems healthy. If I climb inside with docker exec, I can run 'riak ping', 'riak-cs ping' and 'stanchion ping' and all return 'pong'.
I downloaded a fresh install of s3cmd (2.0.2) and configured with this .s3cfg file: (the 2 security keys came from the log output of the running container)
[default]
access_key = OPSBRJYKYIDJQMCDJU6Y
host_base = s3.amazonaws.dev
host_bucket = %(bucket)s.s3.amazonaws.dev
proxy_host = localhost
proxy_port = 8080
secret_key = eUARz8q1KDfap_Dp2t3solbr_8soo5Ubw8UaJA==
signature_v2 = True
The sample docker launch command given in the instructions here create an initial bucket, testbucket, so I thought I should be able to see it.
Running "s3cmd ls" I get this response:
WARNING: Retrying failed request: / (No status line received - the server has closed the connection)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: / (No status line received - the server has closed the connection)
WARNING: Waiting 6 sec...
^CSee ya!
Am I doing something wrong, or are the instructions not complete?
Thanks,
Greg
Hi,
I am unable to get the image to work. It fails to create the initial buckets and rejects any commands. Here is the logs, maybe something that can be fixed:
016-03-25 01:48:45.157 [error] <0.285.0> Supervisor {<0.285.0>,poolboy_sup} had child riak_cs_riakc_pool_worker started with riak_cs_riakc_pool_worker:start_link([{address,"127.0.0.1"},{port,8087}]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:45.160 [error] <0.278.0> Supervisor {<0.278.0>,poolboy_sup} had child riak_cs_riak_client started with riak_cs_riak_client:start_link([]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:45.161 [error] <0.148.0> Supervisor {<0.148.0>,poolboy_sup} had child riak_cs_riak_client started with riak_cs_riak_client:start_link([]) at undefined exit with reason shutdown in context shutdown_error
2016-03-25 01:48:49.342 [error] <0.431.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.374 [error] <0.432.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.385 [error] <0.433.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.396 [error] <0.434.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.406 [error] <0.435.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 01:48:49.418 [error] <0.436.0>@riak_cs_wm_common:maybe_create_user:210 Retrieval of user record for s3 failed. Reason: <<"{pr_val_unsatisfied,1,0}">>
2016-03-25 02:06:08.100 [error] <0.3911.0> CRASH REPORT Process <0.3911.0> with 0 neighbours crashed with reason: no function clause matching riak_cs_s3_rewrite:separate_bucket_from_path("s3.amazonaws.dev:443") line 173
Admin must be created only if it doesn't exist, same applies to buckets.
I did the following:
docker pull 'ianbytchek/riak-cs'
docker run -e 'RIAK_CS_BUCKETS=foo,bar,baz' --name 'riak-cs' ianbytchek/riak-cs
The output I am getting:
Starting Riak… OK!
Starting Stanchion… OK!
Starting Riak CS… OK!
############################################################
Riak admin credentials, make note of them, otherwise you
will not be able to access your files and data. Riak
services will be restarted to take effect.
Key:
Secret:
############################################################
Restarting Riak… OK!
Restarting Stanchion… OK!
Restarting Riak CS… OK!
Creating Riak CS buckets.
… Failed!
… Failed!
… Failed!
2015-09-20 09:18:10,970 CRIT Supervisor running as root (no user in config file)
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/riak.conf" during parsing
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/riak-cs.conf" during parsing
2015-09-20 09:18:10,970 WARN Included extra file "/docker/configuration/supervisord/stanchion.conf" during parsing
2015-09-20 09:18:10,973 INFO supervisord started with pid 3511
2015-09-20 09:18:11,975 INFO spawned: 'riak' with pid 3514
2015-09-20 09:18:11,976 INFO spawned: 'stanchion' with pid 3515
2015-09-20 09:18:11,978 INFO spawned: 'riak-cs' with pid 3516
2015-09-20 09:18:12,981 INFO success: riak entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-20 09:18:12,981 INFO success: stanchion entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-09-20 09:18:12,981 INFO success: riak-cs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
It doesn't output credentials, I am unable to connect with it.
Should be on default port 8098, no luck to find correct config parameter to listen on ip not loopback (versions?) http://docs.basho.com/riak/kv/2.2.0/using/admin/riak-control/
We would like to use your container to provide our developers environment (based on a set of docker images) with a local s3, we're using a Vagrantfile to combine the different containers and to do the linking/mounting/provisionning
The problem we're having with your containers is that we can not create a user with read/write access on a bucket (as you state in the readme, if we set the RIAK_CS_KEY_ACCESS
and RIAK_CS_KEY_SECRET
it fails )
it would be nice to either
/var/whatever/credentials
, so that we can mount /var/whatever
from other containers
I'm looking to simulate S3 locally. On debian stretch amd64 (a Thinkpad X1 Gen 6 Carbon), it takes about 98s from typing
docker run -p 127.0.0.1:1234:8080 ianbytchek/riak-cs
till the final output
2018-09-10 11:37:15,595 INFO success: riak-cs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Until that final output, riak is not usable, and the CPU load is 100%.
In contrast, competing projects take about 0.1s to start.
If there is any compilation needed, it should be done as part of the build process, not during spin-up of the container.
Invoked as: /usr/local/bin/s3cmd ls s3://foo
Problem: BadStatusLine: ''
S3cmd: 1.6.1
python: 2.7.11 (default, Jan 22 2016, 08:29:18)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
environment LANG=None
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2996, in
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2915, in main
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 137, in cmd_ls
File "/usr/local/lib/python2.7/site-packages/s3cmd-1.6.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 170, in subcmd_bucket_list
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 327, in bucket_list
for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 358, in bucket_list_streaming
response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 389, in bucket_list_noparse
response = self.send_request(request)
File "build/bdist.macosx-10.11-x86_64/egg/S3/S3.py", line 1235, in send_request
conn = ConnMan.get(self.get_hostname(resource['bucket']))
File "build/bdist.macosx-10.11-x86_64/egg/S3/ConnMan.py", line 249, in get
conn.c.connect()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1270, in connect
HTTPConnection.connect(self)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 839, in connect
self._tunnel()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 811, in _tunnel
(version, code, message) = response._read_status()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 417, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
Firstly, thank you for perfect docker image.
I would like to store bigger data and make them persistent (docker service restart / server restart).
I guess that I can set custom --volume
param, but unfortunately I do not know which directory I have to mount.
UPDATE
I can change storage by -v /my/storage:/var/lib/riak
but unfortunately (and obviously) it doesn't work - this corrupt/clear the riak DB internal storage. Is it possible to fix it somehow?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.