nbisweden/localega-helm

Helm chart to deploy LocalEGA to any kubernetes cluster.

localega-helm's Issues

S3 secrets mismatch

Expected Behavior

S3 secrets should both contain the same contents.

Current Behavior

kubectl describe secret localega-s3-keys -n lega
Name:         localega-s3-keys
Namespace:    lega

s3_access_key:  16 bytes
s3_secret_key:  32 bytes

kubectl describe secret myminio -n lega
Name:         myminio
Namespace:    lega

accesskey:  18 bytes
secretkey:  34 bytes 

Interestingly, myminio secret is including the "" around the actual secret.

Possible Solution

The latest Minio Helm chart seems to be broken. We might need to downgrade or fix a version in our deployments.

Context (Environment)

Cluster deployed using rke-openstack and legainit

NAME   	REVISION	UPDATED                 	STATUS  	CHART         	APP VERSION                 	NAMESPACE
cega   	1       	Mon Apr 15 16:24:32 2019	DEPLOYED	cega-0.1.2    	1.0                         	cega     
lega   	1       	Mon Apr 15 16:24:34 2019	DEPLOYED	localega-0.2.3	1.0                         	lega     
myminio	1       	Mon Apr 15 16:24:25 2019	DEPLOYED	minio-2.4.12  	RELEASE.2019-04-09T01-22-30Z	lega 

End to End testing

Description

We would like add an end to end testing as part of the deployment that tests:

1. Encrypts a file
2. Encrypted file can be ingested
3. Mocks the stable_id generation
4. Mocks the assignation of files to datasets for permissions
5. Retrieves and compares the file via RES using the session key and IV from generation
6. Retrieves and compares the file from DataEdge using JWT token

Scripts for testing available here: https://gist.github.com/blankdots/bb004abdd6faaed2e59d5687f619eb1e

Images used for M4: EGA-archive/LocalEGA#7 (comment)

Needs a container added to the network for testing.

Definition of Done

Test run in CI on each PR.

How to test

Peer review and testing scenario is successful.

Add TLS option to the chart

Description

Definition of Done

TLS certificates are deployed an used by the containers.

How to test

Configurable PV access modes

Description

The Helm chart should allow the possibility of configurable PV access modes

Proposed solution

Templating access modes in the resource definitions

Definition of Done

The chart passed the tests and works as expected.

Add RBAC rules to chart

Description

Add RBAC and PSP rules to the helm chart

Definition of Done

Chart can be deployed in a namespace with minimal privileges

How to test

Roll deployment on config update

Description

We should roll the depoyment on configmap chages

Proposed solution

Keep ttrack of the checksum for the CM i question and recreate the pod if the checksum changes.

Definition of Done

Changes to a configmap will recreate all related pods.

Defaults in values.yml for fakecega

Description

I'm trying to get local versions of CEGA and LEGA chatting with one another and having a hard time wrangling the config. What variables in values.yml (for fakecega and lega) need to be changed?

As of now, my values.yml looks like so:

config:
  log: "debug"
  broker_connection_attempts: 30
  broker_enable_ssl: "false"
  broker_heartbeat: 0
  broker_host: "lega-localega-mq" 
  broker_port: 5672
  broker_retry_delay: 10
  broker_username: "guest" # local broker user
  broker_vhost: "/"
  cega_users_host: http://cega-users
  cega_users_endpoint: "/lega/v1/legas/users/%s?idType=username" # "/lega/v1/legas/users/%s?idType=username"
  cega_endpoint_json: "response.result"
  cega_mq_host: "cega-mq" # FQDN to cega mq host
  cega_vhost: "/"
  cega_port: 5672
  cega_username: "cega" # cega user
  keyserver_host: "lega-localega-keys" # defaults to localega-keys
  keyserver_endpoint: "/keys/retrieve/%s/private/bin?idFormat=hex"
  postgres_db_name: "lega"
  postgres_db_schema: "local_ega"
  postgres_host: "lega-localega-db" # defaults to localega-db
  postgres_try: 30
  postgres_sslmode: "prefer"
  postgres_user: "lega"
  res_host: "lega-localega-res" # defaults to localega-res
  filedatabase_host: "lega-localega-filedatabase" # defaults to localega-filedatabase
  dataedge_host: "lega-localega-dataedge" # defaults to localega-dataedge
  data_storage_type: "S3Storage" # S3Storage or FileStorege
  data_storage_url: "https://umccr-localega-dev.s3.amazonaws.com" # URl to S3 instance
  data_storage_s3_bucket: "umccr-localega-dev"
  data_storage_s3_region: "ap-southeast-2"
  data_storage_s3_chunk_size: 4 # Chunk size in MB
  data_storage_location: "/ega/data_archive" # path to data archive volume
  data_storage_mode: 2750

persistence: 
  enabled: true

secrets:
  cega_creds: "cega"
  cega_mq_pass: "lega"
  pgp_passphrase: "guest"
  shared_pgp_password: "guest"
  mq_password: "lega"
  postgres_password: "lega"
  s3_access_key: "xxxx"
  s3_secret_key: "xxxxx"

I have a helm installation of fakecega running as cega and am able to resolve these names.

A few other issues I've run into:

• There are hardcoded hashes in cega/conf/cega.json what values do these correspond to? Do these have anything to do with what's in values.yml?
• I'm also unsure what cega_creds cega_mq_pass mq_password refer to since they are undocumented in the readme.
• What does the dummy user password hash in dummy.yml correspond to?
• Should I be using values from trace.yml? Can we use these programmatically?

Proposed solution

• Document default/hardcoded hashes or generate them on the fly
• Document cega_creds cega_mq_pass mq_password
• A simple procedure to spin up CEGA and LEGA together such that the defaults work -- ie if cega and lega are spun up with particular names in helm it "just works".

Definition of Done

• Documentation and defaults updated

Great work on this by the way! I'm looking forward to getting these components singing! 🚀

Move fake CEGA

Move fake CEGA to either a subchart or a separate chart.

Remove HostPath volumes

Description

HostPath volumes should be removes as they are not secure enough to be used.

Proposed solution

Remove HostPath volumes and add an example to the README on how to use the local volume provisioner that is GA as of k8s 1.14.0.

Definition of Done

HostPath volumes are replaces by LocalVolume volumes.

Add option to use python based data out components

Description

See: https://github.com/EGA-archive/LocalEGA/tree/dev-bkp

Proposed solution

Definition of Done

FileStorage backend for Archive fails with permission denied

Expected Behavior

ingest and verify should be able to use a co-located archive on a hostPath

Current Behavior

When setting in values.yaml the following config parameters:

data_storage_type: "FileStorage"
data_storage_location: "/ega/archive" # path to data archive volume
data_storage_mode: 2750

ingest service fails with:

[lega.ingest][ INFO ] (L82) [FileStorage] Moving the rest of 2019-07-18_04-37-15.c4ga to /000/000/000/000/000/000/01
[lega.utils.db][ERROR ] (L246) Exception: <class 'PermissionError'> in /usr/local/lib/python3.6/site-packages/lega/utils/db.py on line: 277
[lega.utils.db][ERROR ] (L249) PermissionError(13, 'Permission denied') (from user: False)
[lega.utils.db][DEBUG ] (L131) Setting error for 1: [Errno 13] Permission denied: '/ega/archive/lega' | Cause: None

The ingest service fails to access the volume due to group permissions.

Possible Solution

1. Add volumeMounts to both ingest and verify based on .Values.config.data_storage_type equal to FileStorage.
2. Add volume to both ingest and verify based on .Values.config.data_storage_type equal to FileStorage also make it co-located aware.
3. The first two steps might not be enough as even though the file might be ingested it still needs to share directory with verify service
4. Setting node affinity for verify to be on the same node as ingest. Might need the same for res service.

Steps to Reproduce

1. Set in values.yaml the values as pointed in Current Behavior above
2. Run e2e tester

Context (Environment)

Run helm charts with FileStorage as archive.

Move message broker deployment outside of localega helm chart

Description

Move message broker deployment outside of localega helm chart

Definition of Done

Message broker deployment is not in localega helm chart

How to test

See DoD

Set up test matrix in github actions

Description

Definition of Done

Test all inbox/storage combinations
mina + filesystem archive
mina + s3 archive
s3inbox +s3 archive

How to test

Break out cega_address into is separate parts.

It is a literal pain to test things when connections are hard coded.
cega_mq_pass is already a secret, move the rest of the connections string to env. variables.

Make conf.ini into env variables.

All connection targets need to be set via helm variables.

cega-mq does not start

Expected Behavior

cega-mq should successfully mount the necessary volumes and start accordingly

Current Behavior

lega-se-testy-service-000  MountVolume.SetUp failed for volume "temp"
 : configmap references non-existent config key: cega.config

lega-se-testy-service-000  Unable to mount volumes for pod:timeout expired waiting for
 volumes to attach or mount for pod "cega"/"cega-mq-7845f789d6-l79qj". list of unmounted
 volumes=[temp]. list of unattached volumes=[cega-mq-entrypoint temp rabbitmq 
 default-token-lnzxx]

Context (Environment)

rke-openstack + legainit

Add functionality to work with Hashicorps Consul and Vault

Description

Definition of Done

OOB functionality that makes use of features found in Vault and/or Consul

How to test

Move database deployment outside of localega helm chart

Description

The Database deployment should not be in the main helm-chart.

Definition of Done

Database is setup in a different chart before the main one.

How to test

See DoD

Local Persistent Volumes Option for Deployment

Description

We will be using POSIX file system for kubernetes and the helm charts deployment would be nice if it supports it.
Useful resource: https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/

We (CSC) might be able to spin the environment for running the end to end tests.

Definition of Done

Helm charts provides option for persistence local system.

How to test

End to end test pass with File System as backend.

Segfault in ega_ssh_keys

Expected Behavior

Users are unable to login to inbox because the ega_ssh_keys binary is segfaulting during operation.

Current Behavior

[root@lega-localega-inbox-66f95f4f45-vt9ks ega]# /usr/local/bin/ega_ssh_keys lega
Segmentation fault (core dumped)

[root@lega-localega-inbox-66f95f4f45-vt9ks ega]# /usr/local/bin/ega_ssh_keys dummy
Segmentation fault (core dumped)

After compiling with debug symbols:

(gdb) bt
#0  0x00007f0e63090f19 in vfprintf () from /lib64/libc.so.6
#1  0x00007f0e630b52cb in vsprintf () from /lib64/libc.so.6
#2  0x00007f0e63097467 in sprintf () from /lib64/libc.so.6
#3  0x00000000004012d0 in main (argc=<optimized out>, argv=<optimized out>) at keys.c:38

Line 38 in keys.c:

  if(sprintf(endpoint, options->cega_endpoint_username, username) < 0){ D1("Endpoint formatting error"); return 2; }

auth.conf is as follows:

[root@lega-localega-inbox-66f95f4f45-vt9ks /]# cat /etc/ega/auth.conf
##################
# Central EGA
##################

cega_endpoint_username = cega-users/lega/v1/legas/users/%s?idType=username
cega_endpoint_uid = cega-users/lega/v1/legas/users/%u?idType=uid
cega_creds = lega:cega
cega_json_prefix = response.result

##################
# NSS & PAM
##################
#prompt = Knock Knock:
#ega_shell = /bin/bash
#ega_uid_shift = 10000

ega_gid = 997
chroot_sessions = yes
db_path = /run/ega.db
ega_dir = /ega/inbox
ega_dir_attrs = 2750 # rwxr-s---
#ega_dir_umask = 027 # world-denied

Possible Solution

My guess here is that options is uninitialised.

Steps to Reproduce

1. Shell into an inbox container kubectl exec -it *inbox* bash
2. Run /usr/local/bin/ega_ssh_keys lega or /usr/local/bin/ega_ssh_keys dummy

Context (Environment)

Environment spun up via the helm chart.

Healthchecks are failing

Expected Behavior

Healthchecks for the DB should work

Current Behavior

Checks fail due to wrong user

Possible Solution

Steps to Reproduce

Context (Environment)

Readiness probe fails in Inbox

Expected Behavior

Inbox Readiness should not fail in OpenSSH inbox.

Current Behavior

There is an error message:

Did not receive identification string from x.x.x.x port 56420
[MQ] Loading configuration /etc/ega/mq.conf
Did not receive identification string from x.x.x.x port 56458
[MQ] Loading configuration /etc/ega/mq.conf
Did not receive identification string from x.x.x.x port 56510
[MQ] Loading configuration /etc/ega/mq.conf
Did not receive identification string from x.x.x.x port 56552

Possible Solution

     readinessProbe:
          exec:
            command: 
            - sh
            - -ec
            - ps -ef | grep ega-sshd
          initialDelaySeconds: 30
          periodSeconds: 15

or maybe it should be different depending on the inbox Mina or OpenSSH

Steps to Reproduce

1. Start Helm charts
2. wait till inbox starts
3. kubectl logs -f -n lega lega-localega-inbox-xxxxx

Context (Environment)

Started the Helm charts with OpenSSH inbox

Add S3-proxy deployment to loclaega chart

Description

Definition of Done

How to test

New profile for DataEdge

Description

After the EGA-archive/ega-data-api#135 is merged we need to add LocalEGA profile to DataEdge, and this issue is so that we don't forget

Definition of Done

DataEdge has the new profile

How to test

e2e test still pass.