ing-bank/rokku

Rokku project. This project acts as a proxy on top of any S3 storage solution providing services like authentication, authorization, short-term tokens, and lineage.

rokku's Issues

Connecting rokku with other ceph

Hi I am trying to replace the ceph container of the project with a ceph of my own, i think i need to modify the environment variables like:
-ROKKU_STORAGE_S3_HOST
-ROKKU_STORAGE_S3_PORT
-ROKKU_STORAGE_S3_ADMIN_ACCESSKEY
-ROKKU_STORAGE_S3_ADMIN_SECRETKEY
with this I will be instructing my ceph to be used.
Where in the project are these variables set?
Would it be necessary to apply some more changes in the project to be able to use an external container?

On the other hand I have one problem running and testing the entire project, with the guide and the configuration set in the project github. The problem came when being all the services up I tried to execute some commands, for example "aws s3api --profile radosgw list-buckets", in which always showed the same error (NoSuchBucket), could you tell me what could be due to this error?

Thanks

Unable to start Rokku Proxy

I'm doing a POC to see how Rokku Proxy works. I've installed Docker, aws cli and sbt, and the containers are running before I try to start the proxy:

Name                        Command               State                         Ports
kafka                     start-kafka.sh                   Up      0.0.0.0:9092->9092/tcp
rokku_atlas_1             sh -c /entrypoint.sh             Up      0.0.0.0:21000->21000/tcp
rokku_ceph_1              /entrypoint.sh demo              Up      0.0.0.0:5000->5000/tcp, 0.0.0.0:8010->8010/tcp
rokku_keycloak_1          /opt/jboss/docker-entrypoi ...   Up      0.0.0.0:8080->8080/tcp
rokku_mariadb_1           docker-entrypoint.sh mysqld      Up      0.0.0.0:3307->3306/tcp
rokku_postgres-server_1   docker-entrypoint.sh postgres    Up      5432/tcp
rokku_ranger-admin_1      sh -c /tmp/entrypoint.sh         Up      0.0.0.0:6080->6080/tcp
rokku_rokku-sts_1         /opt/docker/bin/rokku-sts        Up      0.0.0.0:12345->12345/tcp
rokku_zookeeper_1         /bin/sh -c /usr/sbin/sshd  ...   Up      0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp

Environment

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Also, I have created the /etc/rokku/ranger-s3-security.xml file from the sample file provided without any modifications since I'm running the project with the provided containers.
Below is the error seen while running sbt run:

[info] {"timestamp":"2019-05-22T17:48:26.586Z","level":"INFO","thread":"main","logger":"com.ing.wbaa.rokku.proxy.Server$$anon$1","message":"kerberos credentials are not provided","context":"default","application_name":"rokku-proxy"}
[info] {"timestamp":"2019-05-22T17:48:26.733Z","level":"ERROR","thread":"main","logger":"org.apache.ranger.authorization.hadoop.config.RangerConfiguration","message":"addResourceIfReadable(ranger-s3-audit.xml): couldn't find resource file location","context":"default","application_name":"rokku-proxy"}
[info] {"timestamp":"2019-05-22T17:48:26.735Z","level":"ERROR","thread":"main","logger":"org.apache.ranger.authorization.hadoop.config.RangerConfiguration","message":"addResourceIfReadable(ranger-s3-security.xml): couldn't find resource file location","context":"default","application_name":"rokku-proxy"}
[error] Exception in thread "main" com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger$RangerException: Ranger serviceType or appId not found (serviceType=s3, appId=testservice)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.liftedTree1$1(AuthorizationProviderRanger.scala:37)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.com$ing$wbaa$rokku$proxy$provider$AuthorizationProviderRanger$$rangerPlugin(AuthorizationProviderRanger.scala:27)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.com$ing$wbaa$rokku$proxy$provider$AuthorizationProviderRanger$$rangerPlugin$(AuthorizationProviderRanger.scala:26)
[error] 	at com.ing.wbaa.rokku.proxy.Server$$anon$1.com$ing$wbaa$rokku$proxy$provider$AuthorizationProviderRanger$$rangerPlugin$lzycompute(Server.scala:24)
[error] 	at com.ing.wbaa.rokku.proxy.Server$$anon$1.com$ing$wbaa$rokku$proxy$provider$AuthorizationProviderRanger$$rangerPlugin(Server.scala:24)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.rangerPluginForceInit(AuthorizationProviderRanger.scala:48)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.rangerPluginForceInit$(AuthorizationProviderRanger.scala:48)
[error] 	at com.ing.wbaa.rokku.proxy.Server$$anon$1.rangerPluginForceInit(Server.scala:24)
[error] 	at com.ing.wbaa.rokku.proxy.Server$$anon$1.<init>(Server.scala:39)
[error] 	at com.ing.wbaa.rokku.proxy.Server$.delayedEndpoint$com$ing$wbaa$rokku$proxy$Server$1(Server.scala:24)
[error] 	at com.ing.wbaa.rokku.proxy.Server$delayedInit$body.apply(Server.scala:22)
[error] 	at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] 	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] 	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] 	at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] 	at scala.collection.immutable.List.foreach(List.scala:392)
[error] 	at scala.App.main(App.scala:80)
[error] 	at scala.App.main$(App.scala:78)
[error] 	at com.ing.wbaa.rokku.proxy.Server$.main(Server.scala:22)
[error] 	at com.ing.wbaa.rokku.proxy.Server.main(Server.scala)
[error] Caused by: java.lang.NullPointerException
[error] 	at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
[error] 	at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
[error] 	at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:216)
[error] 	at com.ing.wbaa.rokku.proxy.provider.AuthorizationProviderRanger.liftedTree1$1(AuthorizationProviderRanger.scala:29)
[error] 	... 19 more

Can someone point out what's missing here?

AWS java SDK is not work

when jupyterhub send s3 request like: ttp://{bucket_name}.s3.s3_region.amazonaws.com/object_name, so rokku forwarding the request will make an error.

Running rokku setup on Openshift

Hi,

Just a few questions regarding running the rokku setup in openshift.

1. Are rokku sts and rokku proxy are supposed to run together ?
2. I have rokku sts image to be run as image - I can make it run on openshift. How about rokku-proxy ? I can see in the Readme - "if you are running proxy as image" . But my question is how to run proxy as image ? Can you please share the image details ?
3. Also, If I am running rokku proxy and rokku sts in two different servers, will I be able to make those 2 interact by tweaking any config params ? If yes, can you please guide me in this direction?

Thanks in advance !!

Issues with docker-compose

Hi All,

After running docker-compose up, I am not able to see ranger up and running at port 6080, whereas I can see atlas up and running at 21000.

But I can see the ranger running at port 6080 as part of docker ps result.

Any thoughts on why could this happen ?? Please let me know if I need to provide anything.

Thanks in advance.

The introduction "What is Airlock" still refers to it as Gargoyle

See What is Airlock for the content.