rucio/k8s-tutorial

Rucio K8s tutorial

Rucio - Scientific Data Management

Rucio is a software framework that provides functionality to organize, manage, and access large volumes of scientific data using customisable policies. The data can be spread across globally distributed locations and across heterogeneous data centers, uniting different storage and network technologies as a single federated entity. Rucio offers advanced features such as distributed data recovery or adaptive replication, and is highly scalable, modular, and extensible. Rucio has been originally developed to meet the requirements of the high-energy physics experiment ATLAS, and is continuously extended to support LHC experiments and other diverse scientific communities.

Documentation

General information, API/REST description and guides can be found in our documentation or on our webpage.

Try it out

We provide a dockerized environment which serves both as a demo environment and a development environment. It includes all the necessary preconfigured components for multiple storage and transfers developments.

Developers

For information on how to contribute to Rucio, please refer and follow our CONTRIBUTING guidelines. We strongly recommend to use the dockerized environment for development.

Operators

To learn how to deploy and configure Rucio, consult the documentation available online.

Getting Support

If you are looking for support, please contact us via one of our official channels.

k8s-tutorial's Issues

Various issues & improvement suggestions for the tutorial

I recently run the tutorial and in addition to # 16 and #17, I found the following flaws and suboptimal explanations:

• init pod step: documentation says that log should finish with STATUS: Completed when for me it terminates with Schema set in config, trying to create schema: test. As it works at the end, I suspect that either the pod must be updated or the doc.
• The last step described for the first part of the tutorial (Run FTS storage authentication delegation) is not present in the associated script provided
• IMO, the first script is mainly about setting server components: it is suboptimal to setup the client in the middle as it doesn't really bring any value (not required by the following steps in first script). I'd suggest to move it at the very end of the first script or at the beginning of the second one.
• It is not clear what is expected from the last step (Add some more complications) in the second part of the tutorial, also absent from the associated script

Also, I'd suggest to break the first script in 2 scripts: the one to setup the Minikube environment and the Rucio part. I've been able to run the script successfully on a real Kubernetes cluster after removing the initial part, so it should be easy to do and it is very handy when you are in a qualitification/testing phase where you may want to rerun the tutorial several times.

In addition, the client was not working (ModuleNotFound Error exception) due to a regression rucio/rucio#6490. With version forced to release-33.6.1, I was able to work around the problem. Probably doesn't require a fix in the tutorial itself but rather the fix in the client latest version...

Ununtuitive tutorial steps for Windows

The tutorial steps that include alternative commands for running on Windows should be reworded/reformatted so that it is easier to understand which commands have to be run on which system.

rucio upload fails due to authentication issue with xrootd

The commands here seem to be missing the crucial step of creating an authentication proxy for xrootd.

Checking the test setup in the main rucio repository, it creates the proxy like this:
https://github.com/rucio/rucio/blob/788a0d43fa1caa141efcaaa2780765913a8a49be/tools/docker_activate_rses.sh#L35

Calling this line first make upload / download succeed.