Issues with docker-compose

[ing-bank]

Issues with docker-compose

[arempter]

Hey,
Ranger is applying quite few sql updates, first time it starts. Depending on your HW, it may need some more time to come up.
Check container logs, if it is still applying them, just leave it running. It should eventually come up.
Worst case, restart Ranger container once more. Just do not remove container with postgres.

[DurgaChill]

Hi,

Thanks for the info. Now I can see ranger up and running, but for some reason I can see the ceph image exits in 2 minutes in docker ps status, whereas other services are up and running continuously.

Find the attached result below:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6db06055fd07 wbaa/rokku-dev-apache-atlas:0.0.8 "sh -c /entrypoint.sh" 39 hours ago Up 8 hours 0.0.0.0:21000->21000/tcp rokku_atlas_1
aabd5b5545f4 confluentinc/cp-kafka:5.4.0 "/etc/confluent/dock…" 39 hours ago Up 8 hours 0.0.0.0:9092->9092/tcp kafka
b46a05f41825 wbaa/rokku-dev-apache-ranger:0.0.21 "sh -c /tmp/entrypoi…" 39 hours ago Up 8 hours 0.0.0.0:6080->6080/tcp rokku_ranger-admin_1
a021ee7941c7 wbaa/rokku-sts:0.3.4 "/opt/docker/bin/rok…" 39 hours ago Up 8 hours 0.0.0.0:12345->12345/tcp rokku_rokku-sts_1
efd14918eb0d hazelcast/management-center "bash /mc-start.sh" 39 hours ago Up 8 hours 8081/tcp, 8443/tcp, 0.0.0.0:8081->8080/tcp rokku_cache-management_1
a69a5fb22bde wbaa/rokku-dev-mariadb:0.0.8 "docker-entrypoint.s…" 39 hours ago Up 8 hours 0.0.0.0:3307->3306/tcp rokku_mariadb_1
5711021516ce confluentinc/cp-zookeeper:5.4.0 "/etc/confluent/dock…" 39 hours ago Up 8 hours 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zookeeper
1a7494d8b3c9 wbaa/rokku-dev-keycloak:0.0.8 "/opt/jboss/tools/do…" 39 hours ago Up 8 hours 0.0.0.0:8080->8080/tcp, 8443/tcp rokku_keycloak_1
bc1fd486abcc ceph/daemon:v3.0.5-stable-3.0-luminous-centos-7 "/entrypoint.sh demo" 39 hours ago Exited (1) 2 minutes ago rokku_ceph_1
067fe2973f64 wbaa/rokku-dev-apache-ranger-postgres:0.0.21 "docker-entrypoint.s…" 39 hours ago Up 8 hours 5432/tcp rokku_postgres-server_1
cd1eabb67d64 cassandra:3 "docker-entrypoint.s…" 39 hours ago Up 8 hours 7000-7001/tcp, 0.0.0.0:9042->9042/tcp, 7199/tcp, 0.0.0.0:9160->9160/tcp cassandra

Note: I have started the proxy also and I can see -->
.......
[info] {"timestamp":"2020-04-22T21:54:56.485Z","level":"INFO","thread":"rokku-akka.actor.default-dispatcher-3","logger":"com.ing.wbaa.rokku.proxy.Server$$anon$1","message":"Proxy service started listening: /127.0.0.1:8987","context":"default","application_name":"rokku-proxy"}

Any help would be much appreciated. Thanks in advance.

[arempter]

if you do docker ps -a do you see any previous ceph instances? If some please remove them all, before starting new ceph ...
If this not helps check logs for ceph containter which exited.

[DurgaChill]

Hi, Thanks I had a one old ceph instance. When I re ran docker compose after removing the old ceph instance, it worked for me :)

And when proceeding further, I can see the below errors:

(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

I tried to create a bucket , assuming demo bucket wasn't there. Then I got this unauthorized error.

(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api create-bucket --bucket my-bucket --region us-east-1

An error occurred (Unauthorized) when calling the CreateBucket operation: Unauthorized

Then, I used list-buckets to confirm.

(rokku) [DURGACHILLAKURU@localhost rokku]$
(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-buckets --query "Buckets[].Name"
[
"demobucket"
]
(rokku) [DURGACHILLAKURU@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Can you please let me know what has to be done. Thanks in advance.

[arempter]

ok, that could be related to permission. Just make sure you do run setupS3Env.sh. If not ok, restart rokku afterwards...

[arempter]

Also one more tip. Add following to hosts file:
127.0.0.1 localhost s3.localhost

it simulates AWS virtual bucket access for local env

[DurgaChill]

Hi, I have added the above entry in localhost.

I am not able to run setupS3Env.sh because of permission issues. Any hints on how could I resolve those ? Thanks in advance.

[DurgaChill]

And also, I have added all config params too -

Got Keycloak token and then have set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN. Then I am trying to run the setupS3Env.sh, I am getting permission denied errors.

[arempter]

Above ENV vars are not needed to run setupS3Env...
Script itself only connects to ceph docker and sets ACL and bucket polices ... there should be no high privileges required to run it.

What error are you getting when running script?

[arempter]

btw. make sure that you add --endpoint-url (for aws s3api) and --endpoint (aws s3) with local rokku proxy eg.

aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

[DurgaChill]

Hi, I am not able to set ACL. I am suspecting the validity of the token.
Steps:
Once I generate the token(keycloak) and get the session details, I would replace them in the credentails file under /.aws directory -->
[rokku]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key= YOUR_SECRET_ACCESS_KEY
aws_session_token =YOUR_SESSION_TOKEN.

In general what is the default expiry for these credentials?

Then I ran the setup script (previously it wasn't an executable file, that was the problem)

Image :

Manually I created the buckets that are mentioned in the script setupS3Env.sh.

[durgachillakuru@localhost rokku]$ workon rokku
(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls
2017-12-31 19:00:00 rokku_hc_bucket
2017-12-31 19:00:00 home
2017-12-31 19:00:00 demobucket
2017-12-31 19:00:00 shared

But when I am trying to run the below command,

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$ aws --profile rokku s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Can you please let me know what I am missing over here ? I did try to export aws credentials in terminal and also, I tried to change the params in setup script for aws credentials in line 13 and 14 in the script. When I do so, I can see..

---

upload: '/etc/issue' -> 's3://demobucket/subdir/issue' [1 of 1]
23 of 23 100% in 0s 592.57 B/s done
Bucket 's3://home/' created
Bucket 's3://rokku_hc_bucket/' created
upload: '/etc/issue' -> 's3://home/testuser/issue' [1 of 1]
23 of 23 100% in 0s 658.37 B/s done
upload: '/etc/issue' -> 's3://home/testuser1/issue' [1 of 1]
23 of 23 100% in 0s 721.64 B/s done
upload: '/etc/issue' -> 's3://home/userone/issue' [1 of 1]
23 of 23 100% in 0s 754.25 B/s done
Bucket 's3://shared/' created

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketAcl operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Unknown

Kindly let me know what can be done here. Thanks in advance.

[arempter]

ok, so lets start simple first. Make sure you run setup script in new terminal window, and AWS credentials are not set via /.aws.

you do not need them, script directly talks to ceph container. So run this step first.

And once you have no errors after running it, lets proceed with rest.

BTW.
In general what is the default expiry for these credentials?

• it can be configured in STS (STS_DEFAULT_TOKEN_SESSION_HOURS)

once you have run setupEnv, to get credentials use:

$ source scripts/dev_sts_get_credentials.sh - this will setup credentials correctly

[DurgaChill]

Hi,
(Have removed my previous credentials in the /.aws)
1.Have ran the setup script in a new terminal window. - No errors

(rokku) [durgachillakuru@localhost rokku]$ ./setupS3Env.sh
{
"user_id": "ceph-admin",
"display_name": "Ceph demo user",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "ceph-admin",
"access_key": "accesskey",
"secret_key": "secretkey"
}
],
"swift_keys": [],
"caps": [
{
"type": "buckets",
"perm": ""
},
{
"type": "metadata",
"perm": ""
},
{
"type": "usage",
"perm": ""
},
{
"type": "users",
"perm": ""
}
],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}

upload: '/etc/issue' -> 's3://demobucket/subdir/issue' [1 of 1]
23 of 23 100% in 0s 624.00 B/s done
Bucket 's3://home/' created
Bucket 's3://rokku_hc_bucket/' created
upload: '/etc/issue' -> 's3://home/testuser/issue' [1 of 1]
23 of 23 100% in 0s 400.01 B/s done
upload: '/etc/issue' -> 's3://home/testuser1/issue' [1 of 1]
23 of 23 100% in 0s 473.20 B/s done
upload: '/etc/issue' -> 's3://home/userone/issue' [1 of 1]
23 of 23 100% in 0s 769.93 B/s done
Bucket 's3://shared/' created
(rokku) [durgachillakuru@localhost rokku]$ cd scripts/

2.Then ran source scripts/dev_sts_get_credentials.sh

Read keycloak token: xxx
Aws Json recieved: {
"Credentials": {
"AccessKeyId": "xxxx",
"SecretAccessKey": "xxxx",
"SessionToken": "xxxx",
"Expiration": "2020-04-27T21:48:59.885Z"
}
}

After this, have even checked echo $AWS_ACCESS_KEY_ID --> correct value has been printed, the one I got when I ran the dev_sts_get_credentials.sh script.

But still the issue stays the same,

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls
2017-12-31 19:00:00 rokku_hc_bucket
2017-12-31 19:00:00 home
2017-12-31 19:00:00 demobucket
2017-12-31 19:00:00 shared
(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation:

Do you want me to restart the proxy or can you please let me know what else I am missing over here.? Thanks in advance.

[DurgaChill]

I even tried by restarting the proxy and repeated the steps, but no luck.

If I give aws s3 ls -> I can see all buckets. But when I list the contents, no such bucket exception is been thrown.

---

(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://demobucket

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://home

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://rokku_hc_bucket

An error occurred (Unauthorized) when calling the ListObjectsV2 operation: Unauthorized
(rokku) [durgachillakuru@localhost rokku]$ aws s3 ls s3://demobucket

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: Unknown

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Do you think any ranger policy is being applied ? Do you want me to investigate from that perspective ?

Can you please let me know what I could be missing over here ? Thanks in advance.

[DurgaChill]

I suspected Ranger policies to be in action, so I performed the step that you have mentioned.

Well, your policy in Ranger only allows you to read objects from the demobucket. So we'll need to allow a write as well.

Go to Ranger on http://localhost:6080 and login with admin:admin.
Go to the testservice under the S3 header.
Edit the one existing policy. You'll have to allow the testuser to write, but also don't forget to remove the deny condition!
Save the policy at the bottom of the page.

Still no luck.

---

(rokku) [durgachillakuru@localhost rokku]$ aws s3api list-objects --bucket demobucket --endpoint-url http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown
(rokku) [durgachillakuru@localhost rokku]$

A quick question, how do I check if my credentials are still active ?

[arempter]

ok, since you mentioned ranger, one thing to confirm. Do you see any errors when starting proxy?
for instance this one:
#74

Ranger plugin needs config file to be able to talk to ranger admin server

[arempter]

Also please have a look at logs in rokku-sts container for possible errors with authentication. If there is problem with token it will be reported in sts console

[DurgaChill]

ok, since you mentioned ranger, one thing to confirm. Do you see any errors when starting proxy?
for instance this one:
#74

Ranger plugin needs config file to be able to talk to ranger admin server

No I didn't see any errors like this. And I have placed the config file too. I can login to ranger and can even edit the policy.

[DurgaChill]

Also please have a look at logs in rokku-sts container for possible errors with authentication. If there is problem with token it will be reported in sts console

I did check in sts logs before, I can even see the authorization is success -

---

[info] {"timestamp":"2020-04-28T03:10:00.329Z","level":"INFO","thread":"rokku-akka.actor.default-dispatcher-228","mdc":{"request.id":"a0c11b48-d8c3-4ced-9811-043f4294a525"},"logger":"com.ing.wbaa.rokku.proxy.handler.LoggerHandlerWithId","message":"User (UserName(testuser)) successfully authorized for request: S3Request(AwsRequestCredential(AwsAccessKey(NgPS4OlnAblRKKUpI12jGs3kaNRid42w),Some(AwsSessionToken([jmYuH...]))),Some(/demobucket),None,Read(GET),127.0.0.1:58010,HeaderIPs(None,None,None),none/none)","context":"default","application_name":"rokku-proxy"}

I will try to stop all this once and maybe give a try.

Steps:
1.run docker-compose up (after removing all previous instances)
2. Will place the ranger-s3-security.xml file
3.Create a virtualenv -and then will create aws folder, and then run sbt run in virtualenv.
4. in virtualenv- Run setupS3Env.sh and dev_sts_get_credentials.sh
5. Run this command - docker-compose exec ceph radosgw-admin user modify --uid ceph-admin --system
6. Then check for aws s3api list-objects --bucket demobucket
aws s3api put-object --bucket demobucket --key SOME_FILE commands.

Kindly Let me know if I am missing anything ?

[arempter]

ok, please do, point 5, is not really needed.

Please capture proxy log and add it as attachment, so we will see what proxy is saying during list operation.

[DurgaChill]

Hey, I am getting the below error during sbt run,

[info] {"timestamp":"2020-04-28T06:18:13.606Z","level":"ERROR","thread":"main","logger":"org.apache.ranger.authorization.hadoop.config.RangerConfiguration","message":"addResourceIfReadable(ranger-s3-audit.xml): couldn't find resource file location","context":"default","application_name":"rokku-proxy"}

What could be the reason for this ? I wasn't sure before if it had popped up. Apologies.

Thanks in advance.

[arempter]

Hey,
no this one can be ignored. thats for audit....

[DurgaChill]

Hi, No luck still the same .

Attaching the complete logs for your reference. Kindly let me know what else I can provide with.

rokku_script logs.txt
rokku_proxyLogs.txt
rokku_proxyCurrentLogsLatest.txt

docker_compose_rangerLogs.txt
docker_compose_atlasLogs.txt
ceph_docker_logs.txt

Images in Ranger:

View policy for testpolicy:

Any help is much appreciated.

[DurgaChill]

I have noticed one thing, when I ran sbt run for the very first time. I can see it took a lot of time to start, now it's just hardly a minute. Just a hunch, might strike something. Should I do anything apart from sbt run everytime ?

Thanks in advance.

[arempter]

All the setup seems fine from what you have sent.

Could you please try this one also (not s3api):
aws s3 --endpoint=http://s3.localhost:8987 ls s3://home/testuser/
aws s3 --endpoint=http://localhost:8987 ls s3://home/testuser/

[arempter]

yep. confirmed. I was able to reproduce. It seems to be a problem with s3api. s3 works (above commands).

I also get
aws s3api list-objects --bucket home --endpoint-url=http://localhost:8987

An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown

Could you please now use s3 to list objects?

[arempter]

you could also please try:
aws s3api --endpoint-url=http://s3.localhost:8987 list-objects --bucket home

that one works for me

[DurgaChill]

Hey thanks a lot, it worked like a charm.

[arempter]

Good to hear! Thanks ;)

[DurgaChill]

just a quick question, will this setup work for some other ceph instance running on docker ?

[arempter]

yes, it should...

[DurgaChill]

That's good to hear. Can you just please let me know to specify the endpoint for the same? And also, what would be the config changes regarding the same. Thanks in advance

[DurgaChill]

I am trying NOT to deploy a ceph instance. I just need to hit a ceph instance already running on different server. Can you please let me know if that's something that can be done ?

As I can see a method in getAmazonS3 in S3SdkHelpers.scala

---

def getAmazonS3(authority: Authority,
credentials: AWSCredentials = new BasicSessionCredentials("accesskey", "secretkey", "token")
): AmazonS3 = {
val cliConf = new ClientConfiguration()
cliConf.setMaxErrorRetry(1)

AmazonS3ClientBuilder
  .standard()
  .withClientConfiguration(cliConf)
  .withCredentials(new AWSStaticCredentialsProvider(credentials))
  .withPathStyleAccessEnabled(true)
  **.withEndpointConfiguration(new EndpointConfiguration(s"http://s3.localhost:${authority.port}", awsRegion))**
  .build()

}

---

where Endpoint configuration is mentioned as localhost. I might be referring to files that are irrelevant. Can you just throw some insights from where can I look at those config setup, so that I can connect to the ceph instance running on different server (NOT LOCALHOST) ? Thanks in advance.

[arempter]

please have a look at appliction.conf or reference.conf in the resources directory. All settings are there.

To change S3 settings there is the whole section:

storage.s3 {
# Settings for reaching backing storage.
host = ${?ROKKU_STORAGE_S3_HOST}
port = ${?ROKKU_STORAGE_S3_PORT}

[DurgaChill]

Hey, thanks for the above info. Just a quick question on ceph access.

With the current rokku setup !

1. Do we need to have admin access to the ceph instance ?
2. Are there any default health checks to see if the ceph instance is up and running - by creation/deletion of buckets.

Thanks in advance.

[arempter]

p1. yes you need admin user, which is setup by
admin {
accesskey = ${?ROKKU_STORAGE_S3_ADMIN_ACCESSKEY}
secretkey = ${?ROKKU_STORAGE_S3_ADMIN_SECRETKEY}
}

p2. rokku has /ping endpoint. which you can query to see if rokku/rgw is alive

[DurgaChill]

Thanks for confirming.

Now, I am trying to interact entire rokku setup with ceph nano(https://github.com/ceph/cn) instead of ceph(rokku's).

Steps:

1. Started Ceph cluster - one of the step in ceph nano setup --> which gives me the access key and secret key - enables me to create and delete buckets.
2. In rokku project setup -->

Have removed the ceph component in docker compose.yaml file.
Have set the aws credentials with the access key and secret key generated above in step1.
Have changed the ports also in resources.conf

---

storage.s3 {
    # Settings for reaching backing storage.
    host = "127.0.0.1"
    port = 8001

    admin {
        accesskey = "xxxx"
        secretkey = "xxxx" /// generated in ceph nano
    }
    region = "us-east-1"
    v2SignatureEnabled = false
    enabledCache = false
    eligibleCachePaths = "/home/,/test/"
    maxEligibleCacheObjectSizeInBytes = 5242880
    cacheDStructName = "S3Cache"

    healthCheck {
       # can be one of:
       #   rgwListBuckets - uses RGW admin to list all buckets in Ceph
       #   s3ListBucket  - uses AWS S3 client to list single bucket
       method = "s3ListBucket"
       interval = 5000
       bucketName = "bucket1"
    }

---

Note: I did not receive any such session id in ceph nano - while creating a cluster. All i got is
Endpoint: http://172.20.20.20:8001
Dashboard: http://172.20.20.20:5001
Access key: xxxx
Secret key: xxxx
Working directory: /~/cephnano/cluster

Anyways, I am not able to interact with this ceph nano cluster. Any idea or pointers that could help me at this point ? I haven't checked any ranger settings to be configured. Any help is appreciated.

Also, I know this is irrelevant in this thread. Do let me know, if I can open a new one for this scenario, since the original scenario has been solved.

Thanks in advance.

[arempter]

so I do not know how ceph nano is build / works, to be honest. We only test and run rokku for now with regular ceph.
Feel free to play with it but, I am not sure if this will work...