serlo/infrastructure-env-staging

infrastructure-env-staging's Issues

Unexpected changes at node_pool

We get the following unexpected changes when deploying.

# module.cluster.google_container_node_pool.cluster["non-preemptible"] will be updated in-place
  ~ resource "google_container_node_pool" "cluster" {
        id                          = "projects/serlo-staging/locations/europe-west3-a/clusters/serlo-staging-cluster/nodePools/serlo-staging-cluster-non-preemptible"
        name                        = "serlo-staging-cluster-non-preemptible"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          - location_policy      = "BALANCED" -> null
            # (4 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

  # module.cluster.google_container_node_pool.cluster["preemptible"] will be updated in-place
  ~ resource "google_container_node_pool" "cluster" {
        id                          = "projects/serlo-staging/locations/europe-west3-a/clusters/serlo-staging-cluster/nodePools/serlo-staging-cluster-preemptible"
        name                        = "serlo-staging-cluster-preemptible"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          - location_policy      = "ANY" -> null
            # (4 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

Probably related to the upgrade of GKE
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-26

Redis server down due to disc usage

On 19th May 2022 between 10 am and 2 pm redis server stopped to work, what led to API out of service.
The cause was too much disc usage : the /data mount was at 100% (8Gi).
After removing the heavy apendonly.aof file, it worked again.

TODO

• Redo all manual changes:
• reset storage increase of pv, pvc and pod
• Put a limit for apendonly.aof
• Investigate the cause of so much disc comsumption (at prod it was at 328Mi)
• Create recovery workflows for such cases

Remove legacy from staging

Investigate why registration emails are not being sent

In staging the verification email after registration are not being sent/received. In kratos logs you can see it sends the email. In sparkpost I can't find though any corresponding event

Script for importing users into kratos table without sensitive data

Constant `Pod errors: NodeAffinity` in several workloads

Every day some workloads in staging get Pod errors: NodeAffinity.

It should be related to https://stackoverflow.com/questions/68560484/predicate-nodeaffinity-failed.

It is not a big deal because it is staging and the pods are autohealing. But it should anyway be solved

Enable Enmeshed PoC again

See branch user-journey
Choose strategy:
a. We rollback the staging for a while to a state in which the first user journey was presented. (advantage: it is simple, disadvantage: it blocks our tests in staging)
b. We adapt the current code to reintroduce the enmeshed backend service. ( advantage: it doesn't block our tests in staging, disadvantage: it is more complex than a, possibily introducing bugs and unnecessary code)

Assuming a, the following has to be made:

• Enmeshed Relationship Template was created when API was in version 0.27.0 and due to refactorings the template creation was somehow skipped. As long as it was in the database, it was ok, but recreating everything led to HttpServer - POST /api/v1/RelationshipTemplates/Own/.../Token 404. Maybe write a script for easily redoing it or create an new api version, where this problem is fixed. Anyhow, it is good to have it documented.
• After recreating enmeshed relationship template, upgrade API to a more recent version (in which the enmeshed middleware was still available, of course. v0.36.0 is a good candidate).
• Make a tag in this repo for that (better than leaving a stale branch, but it would be also ok)
• cloudflare worker: test /user-journey in main branch
• Due to a huge upgrade in Keycloak helm chart without the corresponding configuration, its deployment is buggy, leading to 5 min timeout while deploying. Fix that (possible solution is simply removing Keycloak).