mattermost/mattermost-operator

Mattermost Operator for Kubernetes

mattermost-operator's Issues

Kubernetes won't pull down the right mattermost-operator version

When I install mattermost-operator on a kubernetes cluster that already have pulled down the image before, it won't get the latest version of mattermost-operator. In this documentation file you use both image: mattermost/mattermost-operator:latest and imagePullPolicy: IfNotPresent.

This results in that kubernetes will pull down the image just the first time a user run this yaml file. Then you make an update to mattermost-operator and I want that update. So if the user then run this yaml again, then kubernetes will use the cache of mattermost-operator image instead of pulling down the latest because the imagePullPolicy says it should only pull down the image if the image doesn't exists in the cache. Which it does, because it's the same name and the same tag.

Proposed solution would be to start versioning the mattermost-operator image and put that in the image tag. Then Kubernetes won't use the cache if the image is updated. A user can also decide which version they want to use.

Another solution would be to remove the imagePullPolicy and still use latest as the image tag. Kubernetes would then use Always as the pull policy. However this will make Kubernetes to always pull down the image from docker hub, even if the image hasn't been changed.

panic when doing `dep init`

I am trying to follow the README.md to set up the dev environment. I am getting the a bunch of errors while doing dep init right before step make dep. One of them is:

panic: version queue is empty, should not happen: gopkg.in/fsnotify.v1

goroutine 1 [running]:
github.com/golang/dep/gps.(*solver).findValidVersion(0xc00059e000, 0xc001165220, 0xc0004ca9e0, 0x1, 0x1, 0x0, 0xc0004ca9e0)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/gps/solver.go:915 +0x55a
github.com/golang/dep/gps.(*solver).createVersionQueue(0xc00059e000, 0xc0004f31a0, 0x14, 0x0, 0x0, 0xc0004ca9e0, 0x1, 0x1, 0x0, 0x0, ...)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/gps/solver.go:902 +0x472
github.com/golang/dep/gps.(*solver).solve(0xc00059e000, 0x16e3660, 0xc0000b0058, 0x5, 0xc00055a708, 0xc0000ff698)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/gps/solver.go:505 +0x543
github.com/golang/dep/gps.(*solver).Solve(0xc00059e000, 0x16e3660, 0xc0000b0058, 0xc0001886f0, 0xc0000260c2, 0x29, 0xc000464bd0)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/gps/solver.go:440 +0x1eb
main.(*initCommand).Run(0xc00017768a, 0xc0000e8880, 0xc000188680, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/cmd/dep/init.go:155 +0x5ad
main.(*Config).Run(0xc0000bcea0, 0x0)
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/cmd/dep/main.go:212 +0x111c
main.main()
	/private/tmp/dep-20190614-48387-1iw5fry/src/github.com/golang/dep/cmd/dep/main.go:84 +0x636

Another overall question: should Gopkg.toml be checked in (currently it is missing)?

Allow defining Service type

Summary

Currently the CRD doesn’t allow defining the Service type, defaulting to a type of ClusterIP.

This is needed in order to support GKE Ingress, which do not allow to bound to a Service of type ClusterIP when container native load balancing is not used.

Steps to reproduce

• Create a GKE cluster without container native load balancing
• Deploy mattermost through the operator with following spec related to ingress:

ingressAnnotations:
  kubernetes.io/ingress.class: "gce"
ingressName: mattermost.mydomain.com

• Check created Ingress to see the following failure:

$ kubectl -n mattermost describe ing mattermost
Name:             mattermost
Namespace:        mattermost
Address:
Default backend:  default-http-backend:80 (x.x.x.x:8080)
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  mattermost.mydomain.io
                          /   mattermost:8065 (x.x.x.x:8065)
Annotations:              kubernetes.io/ingress.class: gce
                          mattermost.com/last-applied: ...
                          nginx.ingress.kubernetes.io/proxy-body-size: 1000M
Events:
  Type     Reason     Age                From                     Message
  ----     ------     ----               ----                     -------
  Normal   Sync       91s (x2 over 91s)  loadbalancer-controller  Scheduled for sync
  Warning  Translate  9s (x15 over 91s)  loadbalancer-controller  Translation failed: invalid ingress spec: service "mattermost/mattermost" is type "ClusterIP", expected "NodePort" or "LoadBalancer"

Operator version used: 1.14.0
Mattermost version used: 5.37.2
Environment: GKE

Expected behavior

Being able to specify mattermost Service type, GKE Ingress would work.

Possible fixes

I’m not a Go developer, but I think this is the relevant code.

I thought at first that this feature might be enabled by the recent changes to Ingress management allowing to disable Ingress creation (triggered from this issue), but the problem is that the operator will continue to set a service of ClusterIP type.

Possible solutions that I can think of:

• allow disabling Service creation, falling back on user to create the service (easiest)
  • something like a boolean serviceEnabled
• allow setting Service type to NodePort, without trying to set ClusterIP to None (preferred)
  • serviceType property that can be set to:
    • headless (default)
    • clusterIP
    • nodePort
• allow setting Ingress parameters and useServiceLoadBalancer at the same time (this is not currently possible as the useServiceLoadBalancer takes precedence over Ingress parameters)

Support pod labels/annotations

Summary

Currently there is no support for adding annotations on the server pods. Labels can be added via resourceLabels but any labels added there are applied to all resources, rather than just the server pods.

This support would be beneficial for grouping things together for ownership and provide for better integration with externally managed resources (i.e. networkPolicy pod selectors).

Possible fixes

Add support via:

spec:
  podTemplate:
    labels: {}
    annotations: {}

As mentioned here I'd propose the behavior with the default labels/annotations be as follows:

• Annotations: Prometheus defaults will always override user specified annotations when they conflict
• Labels: Highest precedence would be the defaults set by the operator, then user specified podTemplate.labels, then resourceLabels

In my opinion this does not change any default behaviors so shouldn't require a major version bump. Annotations have never been exposed to the end user so that is just a new feature. Previously overriding one of the default labels would actually cause the operator to error since different spots in the code handle label precedence differently. SelectorLabels will give precedence to the default operator provided app label (see here), whereas labels will always give precedence to the user's labels (see here). In other words it would've been impossible to deploy with an overridden app label, so this is effectively a "new feature" as well (or a bug fix depending on how you look at it).

Add ability to specify Image Pull Secrets on Mattermost resource

Summary

As discussed in #221 we would like to be able to specify ImagePullSecrets for Mattermost images.
For that we need to add new field to Mattermost spec and propagate it when creating to Deployment.

Field should be optional and empty by default.

Add error information to Mattermost CR status

Summary

Currently, the only way to see that there are some issues with processing Mattermost CR is to check logs of the Operator. While this should still be a way to go for a detailed error report, adding the error section to Mattermost CR status would be helpful to signal to the user that something is not right.

Possible solution

We can add a new Error field to MattermostStatus that we populate in case of an error. As we already try to update the status anytime the reconciliation finishes it should be pretty straightforward to set an error on the new status before the update.

Operator image validation check failing with sidecar

Summary

In summary, when using an injected sidecar with Mattermost and the operator the "image validation health check" could erroneously print that the wrong image is being used, depending on the container order.

Steps to reproduce

Deploy the operator + something that will inject a sidecar to pods, in the first container position. In my case I was using Istio when I ran into this issue. Deploy a Mattermost CR and watch as it reconciles. The pod should come to healthy with 2/2 containers running but when you check the CR status it will show as "reconciling". Going a step further, check the operator logs and you will see:

time="2022-05-13T17:03:48Z" level=info msg="[opr.controllers.Mattermost.health-check] mattermost pod mattermost-xyz is running incorrect image" Request.Name=mattermost Request.Namespace=mattermost

Expected behavior

I would expect that the operator identifies and checks against the mattermost container, regardless of container order / existing sidecars.

Observed behavior

The first container in the pod is always checked, and when a sidecar is present as that first container the health check errors.

Possible fixes

The problematic line is here where the operator checks the first container (containers[0]) image.

The fix in my mind would be to have this line instead check a specific container, potentially checking by name since the name seems to be a variable we could access (also hardcoded to "mattermost" if I'm not mistaken).

kubernetes mattermost installation failure and unable to change fake ssl

I'm following the default guide to launch mattermost on Kub:
post installation I'm getting 503 error and the SSL is not generated for the url required

minio operator is very old

minio operator is now 3.0.28, while you use 1.0.7

any plans to upgrade it?

Mattermost install-kubernetes TLS Ingress Controller

We have installed Mattermost with the k8s manual. Successful! :) After that we tried to configure a TLS certificate on the Mattermost Ingress Controller, but there is no option in the mattermost-installation.yaml.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
    - sslexample.foo.com
    secretName: testsecret-tls

Will this feature come or can we solve this differently?

Clarify BYO SQL instructions

The FAQ indicates that you can bring your own instance of MySQL to use with the Operator, instead of relying on the (broken for me) mysql operator.

I can also see that docs/examples/full.yaml seems to imply that an 'externalSecret' value can be set to indicate the password of the external database, but there is nowhere to put the address, and that value does not appear to be referenced anywhere else in the codebase...

Yamls are wrong

Summary

Yamls are defined in a wrong way

Problem

In the docs, the operator are applied using this URL:

https://docs.mattermost.com/install/install-kubernetes.html

$ kubectl apply -n mattermost-operator -f https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mattermost-operator/mattermost-operator.yaml

The mattermost-operator.yaml define a service account that is not namespaced:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mattermost-operator
---

but the clusterrolebinding assume that the SA is in the namespace mattermost-operator

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mattermost-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mattermost-operator
subjects:
- kind: ServiceAccount
  name: mattermost-operator
  namespace: mattermost-operator
---

Also the deployment and the service does not have a namespace defined, but use the SA that suppose to be in mattermost-operator namespace.

Deployment has serviceAccountName: mattermost-operator

Possible fixes

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mattermost-operator
  namespace: mattermost-operator

---

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: mattermost-operator
    name: mattermost-operator
  name: mattermost-operator
  namespace: mattermost-operator
spec:
  ports:
  - name: metrics
    port: 8383
    protocol: TCP
    targetPort: metrics
  selector:
    name: mattermost-operator
  type: ClusterIP
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mattermost-operator
  namespace: mattermost-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mattermost-operator
  template:
    metadata:
      labels:
        name: mattermost-operator
    spec:
      containers:
      - args:
        - --enable-leader-election
        - --metrics-addr=0.0.0.0:8383
        command:
        - /mattermost-operator
        env:
        - name: MAX_RECONCILING_INSTALLATIONS
          value: "20"
        - name: REQUEUE_ON_LIMIT_DELAY
          value: 20s
        image: mattermost/mattermost-operator:v1.15.0
        imagePullPolicy: IfNotPresent
        name: mattermost-operator
        ports:
        - containerPort: 8383
          name: metrics
      serviceAccountName: mattermost-operator

Operator projects using the removed APIs in k8s 1.22 requires changes.

Problem Description

Kubernetes has been deprecating API(s), which will be removed and are no longer available in 1.22. Operators projects using these APIs versions will not work on Kubernetes 1.22 or any cluster vendor using this Kubernetes version(1.22), such as OpenShift 4.9+. Following the APIs that are most likely your projects to be affected by:

• apiextensions.k8s.io/v1beta1: (Used for CRDs and available since v1.16)
• rbac.authorization.k8s.io/v1beta1: (Used for RBAC/rules and available since v1.8)
• admissionregistration.k8s.io/v1beta1 (Used for Webhooks and available since v1.16)

Therefore, looks like this project distributes solutions in the repository and does not contain any version compatible with k8s 1.22/OCP 4.9. (More info). Following some findings by checking the distributions published:

• mattermost-operator.v0.5.0: this distribution is using APIs which were deprecated and removed in v1.22. More info: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. Migrate the API(s) for CRD: (["clusterinstallations.mattermost.com"])

NOTE: The above findings are only about the manifests shipped inside of the distribution. It is not checking the codebase.

How to solve

It would be very nice to see new distributions of this project that are no longer using these APIs and so they can work on Kubernetes 1.22 and newer and published in the community-operators collection. OpenShift 4.9, for example, will not ship operators anymore that do still use v1beta1 extension APIs.

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

If this projects only need to migrate the API for CRDs and it was built with OperatorSDK versions lower than 1.0.0 then, you maybe able to solve it with an OperatorSDK version >= v0.18.x < 1.0.0:

$ operator-sdk generate crds --crd-version=v1
INFO[0000] Running CRD generator.
INFO[0000] CRD generation complete.

Alternatively, you can try to upgrade your manifests with controller-gen (version >= v0.4.1) :

If this project does not use Webhooks:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..."

If this project is using Webhooks:

1. Add the markers sideEffects and admissionReviewVersions to your webhook (Example with sideEffects=None and admissionReviewVersions={v1,v1beta1}: memcached-operator/api/v1alpha1/memcached_webhook.go):

2. Run the command:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..."

For further information and tips see the comment.

Disabling ingress still requires `ingress.host`

Summary

The new ingress.enabled spec does allow users to disable ingress but it seems like the ingress.host value in the spec is used for much more than just the ingress. Disabling ingress without providing an "accurate" value to ingress.host causes a number of issues.

Steps to reproduce

1. Deploy the operator.
2. Deploy a Mattermost resource with ingress.enabled set to false and nothing set under ingress.host or ingressName.
3. Attempt to navigate to the UI via alternative means (i.e. port-forward or using a service mesh ingress).
4. You should see the UI load a single white screen, with some errors in the browser console.

Expected behavior

Ingress should be able to be fully disabled with the UI still working.

Observed behavior (that appears unintentional)

Ingress is able to be disabled, but unless you provide an ingress.host the UI does not function.

Possible fixes

I'm not 100% sure but I suspect it is due to a line here where it is used for the siteUrl - which affects things in the config.json.

Assuming this is the spot that causes the issue, this is clearly a needed variable/config to set. Perhaps the solution is to do a number of things:

1. A new spec.siteURL (name/place wherever it makes sense) that is used to populate the config.json when available, but defaults to ingress.host when not set.
2. Error handling for the operator to reflect this flow ^. Since the ingress was made optional the operator should perform some validation that the siteURL is populated with a value otherwise users can deploy, pods will go to running, but the UI will be unusable.
3. Better document this (maybe not needed if you add a new siteUrl value). Currently examples mention that ingressName is required, but only when using an ingress.

While there is a workaround, I think the naming of the values in spec could get confusing and at minimum this should be documented.

Mattermost can't reach mysql

I've followed the instructions to install mattermost via operators on kubernetes, but my mattermost init pod 1 can never reach the mysql instance. I'm seeing this repeated in the logs:

init-check-mysql waiting for mysql
init-check-mysql   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
init-check-mysql                                  Dload  Upload   Total   Spent    Left  Speed
init-check-mysql   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to db-3366d4-mysql-master.mattermost port 3306: Connection refused
init-check-mysql waiting for mysql

I've tested the connection to mysql by connecting to it from another pod to two different services in the mattermost namespace:

$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -n mattermost -- mysql -h db-3366d4-mysql-master.mattermost -p<password>

ERROR 2003 (HY000): Can't connect to MySQL server on 'db-3366d4-mysql-master.mattermost' (111)

$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -n mattermost -- mysql -h mysql.mattermost -p<password> 

mysql> 

But mattermost seems to want to connect to it through the db-3366d4-mysql-master service rather than the mysql service. Is there any way to specify this? Or is it something in the operator code that needs to be changed?

Let me know if you need any additional information from me, thanks!

Do not return errors from reconciliation when waiting for update job

Summary

Before changes are rolled out to Mattermost deployment, the update job is deployed to verify that the image is correct. The Operator waits for the job to complete before starting deployment update.

From reconciler perspective, when the job is still running the error is returned to requeue reconciliation request. If error occur enough number of times, in rare cases this may significantly delay rolling out the deployment due to error reconciliation back off.

We should modify logic that waits for the update job to finish to return some indication that it is not yet finished rather than returning an error and use constant time for requeue delay. For now it should be enough to return bool alongside error and propagate it from checkUpdateJob function to reconciler.

Feature Request: Add mattermostEnv environment variables to init containers

Add the mattermostEnv environment variables to the Init Contianers. This would allow setting the database username and password from a secret in the form:

username: mmuser
password: mmpassword

With a database connection string using $DB_USER and $DB_PASSWORD, with shell expansion to the correct values. This is helpful for setups using a postgres-operator, where the user's password is automatically generated and stored in a secret like above.

Mattermost | Ingress TLS | Cert-manager | ClusterIssuer not working

Dear Mattermost Operator Team and the community,

Hope everyone is doing well. Thank you for the awesome work on the Mattermost operator. We really appreciate it; it helps us to make our life easier. Thank you :).

Summary

We are trying to deploy Mattermost on a Kubernetes Cluster. Cluster already has cert-manager (cluster-issuer) and Nginx ingress controller. However, when the certificate is issued, it is from Kubernetes Ingress Controller Fake Certificate rather than the cluster-issuer. We did test if our cluster-issuer is issuing certificates or not, and it does. We are unsure what is wrong or how to debug it; some guidance would be appreciated. Thank you :)

Steps to reproduce

• Reproduce the issue using by installing the Mattermost on on-prem Kubernetes with the below

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
metadata:
  name: mm-bitvijays                         # Chose the desired name
spec:
  size: 200users                               # Adjust to your requirements
  ingress:
    enabled: true
    host: mattermost.bitvijays.local        # Adjust to your domain
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      cert-manager.io/cluster-issuer: vault-issuer-4
      cert-manager.io/common-name: mattermost.bitvijays.local
  version: 6.0.1
  licenseSecret: ""                             # If you have created secret in step 1, put its name here

• Mattermost image is mattermost/mattermost-operator:v1.18.0

Expected behavior

• Mattermost should have requested the certificate from the ClusterIssuer and the clusterissuer issued certificate should be displayed when accessing the mattermost website.

Observed behavior (that appears unintentional)

Certificate is issued by the Kubernetes Ingress Controller Fake Certificate rather than by the ClusterIssuer

Possible fixes

It might be because of the fact that the kubernetes.io/ingress.class annotation is deprecated and replaced by the ingressClassName parameter in the latest ingress spec.

ArgoCD Operator is also probably having the same issue. May refer argoproj-labs/argocd-operator#626

Add owner reference to update-check job so that it points to Mattermost resource

Description

When modifying spec.image or spec.version of Mattermost resource the update-check job is run to ensure image is correct.
However, the job is not owned by the Mattermost resource which causes Operator to not to pick up when the job completes and carry on with update.

Possible solution

Very simple and reliable solution would be to add OwnerReference to the job, so that it is owned by Mattermost resource.
We already have function that adds owner references when creating objects. We could use it when lunching the update job

Allow custom db check init container image

Summary

Currently the DB check hardcodes the image to either a curl image or a postgres 13 image depending on the DB connection string. It would be great to have the ability to customize the image being used here (for example, specify a different postgres image that has more security hardening or a different version postgres).

Possible fixes

Add a value in the mattermost spec that is consumed when creating the deployment and specifying the init container.

Example of what I'm thinking:

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
...
spec:
  database:
    external:
      initDbCheck:
        image: mycustompostgres
        tag: 11.2
...

Help Wanted Request

Note: this section was added later by the repository owners

Proposed solution for this feature from comments below:

...
spec:
  database:
    readinessCheck:
       initContainers: [Full Containers definition here allowing user to override everything (in Go type []v1.Container)]

We should allow for the init containers to be completely flexible for user's needs so a full container spec would be ideal here.

Adding support for custom CA certificates

For assuring that the machine will not get signed by unkown authority if has custom ca certifcates
And using skip ssl verify is not supported on some components (s3, jenkins plugin etc..)

suggesting adding to the cr
certificates
secret:
- secret 1
- secret 2

and in the mattermost.go

add something like this - creating one big volume and volume mounts
havent figured out where and how to get secret data exactly

	volumes := []corev1.Volume{}
	volumeMounts:= []corev1.VolumeMount{}
	// Mattermost extra secret
	if mattermost.Spec.customCA {
		volumeMount= append(volumeMounts, corev1.VolumeMount{
			MountPath: "/mattermost-license",
			Name:      "mattermost-license",
			ReadOnly:  true,
		})

		volume = append(volumes, corev1.Volume{
			Name: "certificates",
			VolumeSource: corev1.VolumeSource{
				Secret: &corev1.SecretVolumeSource{
					SecretName: mattermost.Spec.customCA.secret,
				},
			},
		}
	}


	// Mattermost License

	podAnnotations := map[string]string{}
	if len(mattermost.Spec.MattermostLicenseSecret) != 0 {
		envVarGeneral = append(envVarGeneral, corev1.EnvVar{
			Name:  "MM_SERVICESETTINGS_LICENSEFILELOCATION",
			Value: "/mattermost-license/license",
		})

		volumeMount= append(volumeMounts, corev1.VolumeMount{
			MountPath: "/mattermost-license",
			Name:      "mattermost-license",
			ReadOnly:  true,
		})

		volume = append(volumes, corev1.Volume{
			Name: "mattermost-license",
			VolumeSource: corev1.VolumeSource{
				Secret: &corev1.SecretVolumeSource{
					SecretName: mattermost.Spec.MattermostLicenseSecret,
				},
			},
		})

this is from my helm chart that I added to my installation

  extraVolumes: 
    - name: certificates
      secret:
        secretName: certificates
  extraVolumeMounts: 
    - name: certificates
      mountPath: /etc/ssl/certs/cert1.pem
      subPath: cert1.pem
    - name: certificates
      mountPath: /etc/ssl/certs/cert2.pem
      subPath: cert2.pem

Retreive information that were used during installation?

I do not have access to mattermost-installation.yaml that was used when installing but would like to upgrade the version.

Where and how can I find these information?

size: xxxxXusers is not working when you want to scale

---

Begin Edit by @gabrieljackson

The Mattermost Operator currently offers two ways to set resources for the Mattermost app pods and supporting database and filestore pods. One method is to use the Size configuration option and the other is to set the Resources and Replicas configuration.

Size configuration will set the Resources and Replicas configuration to some default values when used. By setting Resources and Replicas, users can override defaults set by the Size configuration.

This issue outlined below shows an issue with behavior where setting a given Size and then later changing it will not update the underlying Resource and Replica values. This is due to Resource and Replica overrides being possible as mentioned above.

A solution to this is desired where updating the Size value is able to scale the Mattermost application up or down in an intelligent way with respect to possible overrides.

If you're interested in working on this enhancement, please comment here and come join our "Contributors" community channel on our daily build server, where you can discuss questions with community members and the Mattermost core team. For technical advice or questions, please join our "Developers" and "Developers: Cloud" community channels.

New contributors please see our Developer's Guide.

JIRA: https://mattermost.atlassian.net/browse/MM-22241

End Edit

---

Original Issue

Hi,

size: 5000users # Size of the Mattermost installation, typically based on the number of users. Automatically sets the replica and resource limits for Minio, databaes and app servers based on the number provided here. Accepts 100users, 1000users, 5000users, 10000users, or 25000users. Manually setting replicas or resources will override the values set by 'size'.

I am wondering, if size option is working only when you start mattermost first time. Thus, you cannot scale using that option afterwards? I started mattermost with 100users size so only 1 replica of everything was created. Then I changed the file to 5000users, applied config and nothing changed. Same goes if I decided to create mattermost to 10k users and then scale out to 100 - operator does not react on that. Could you explain what is the right way to set replicas if I need that?

Thank you

How to add custom annotations field?

values.yaml add metadata.annotations not effect.

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mattermost
  annotations:
    reloader.stakater.com/auto: "true"

Unable to load configuration: failed to create store (externaldb)

Summary

Unable to store external postgres configuration on version 6 and above with read only error

Using this config:

  database:
    external:
      secret: mattermost

kubectl logs -f -n mattermost pod/mattermost-7cc6dbcdfb-9bmd2
Error: failed to load configuration: failed to create store: unable to load on store creation: failed to persist: failed to write file: open /mattermost/config/postgresql:/user:password@mypostgresdb:25060/defaultdb?sslmode=require: no such file or directory
Usage:
  mattermost [flags]
  mattermost [command]

Available Commands:
  db          Commands related to the database
  export      Export data from Mattermost
  help        Help about any command
  import      Import data.
  jobserver   Start the Mattermost job server
  server      Run the Mattermost server
  version     Display version information

Flags:
  -c, --config string   Configuration file to use.
  -h, --help            help for mattermost

Use "mattermost [command] --help" for more information about a command.

Steps to reproduce

• Latest mattermost operator
• Deploy mattermost using helm operator and external DB secret

Expected behavior

mattermost starts

Mattermost K8 / Reconciling State

Hi,

I followed the official documentation to deploy MM on K8s; however my installation process is stuck with custom resource state at "reconciling"

mattermost-operator does not provide appropriate configuration for securityContext in mattermost deploymets.

Summary

Although mattermost-operator deployment via chart allows securityContext configuration, it is not applied to upstream mattermost installations.
I reviewed CRD definition and it does not appear a securityContext key to configure this secure behavior, therefore mattermost installations fail on secure environments (Pod Security Policies enabled not allowing root containers/specify runAs user). This just appears for special initContainers.

Steps to reproduce

Deploy the mattermost-operator and create a new mattermost instance.

Expected behavior

I expect to be able to specify the securityContext key for upstream mattermost instances created using mattermost-operator.

Observed behavior (that appears unintentional)

CreateContainerConfigError arraise as soon as you create a new mattermost instance due to not being able to especify RunAs user key and image defined user is not represented by its ID.

Editing created deployment manually works just by adding requiredsecurityContext configuration, but first it is required to change ownerReferences's controller key to false to disable the management of this resource by the mattermost-operator.

Possible fixes

Add securityContext option for being able to configure Kubernetes environment security for both, mattermost-operator and upstream created mattermost instances using it.

AWS ALB Ingress option

Summary

At the moment, only a Service LoadBalancer, and a Nginx Ingress is supported.
It would be great to be able to use an AWS ALB.

AWS only provides a Classic LB if we use Service LoadBalancer.

Expected behavior

Add option to create AWS ALB Ingress instead of Nginx Ingress

Configure custom.json for e.g. Plugins using the Operator with a mounted ConfigMap

Hi,

Does the new mattermost operator somehow supports the configuration of custom.json ?

https://docs.mattermost.com/administration/config-settings.html#load-custom-configuration-defaults

I'm looking for a way to configure the welcomebot in a fully ansible managed setup which seems to require edditing the config.json file. Any ideas on how to achieve this using the operator?

A custom volume mount with a ConfigMap may seems the way to go. But I can't get it to work yet:

  mattermostEnv:
    - name: MM_CUSTOM_DEFAULTS_PATH
      value: "/mattermost/config/custom-config-map/custom.json"
  volumeMounts:
  - mountPath: "/mattermost/config/custom-config-map"
    name: mattermost-custom-json-config
  volumes:
    - name: mattermost-custom-json-config
      configMap:
        # Provide the name of the ConfigMap containing the files you want
        # to add to the container
        name: mattermost-custom-json-config

something like this seems promising, but unfortunatelly the mounted user is root..

Are there any plans to add an initContainer Config or a PodSecurityContext Config to the operator? So I can manually modifiy the mounted file permissions. e.g.:

securityContext:
fsGroup: 2000

Allow overrides to update-check job spec

Summary

From a quick scan through the code I don't believe there is currently any way to override parts of the spec for the update-check job that runs with version updates. It may be desirable to override various parts of this job, for example:

• securityContext: Enforce best practices for security such as runAsNonRoot, dropping all capabilities, etc
• labels/annotations: To better group things together for ownership and provide for integration with externally managed resources (i.e. networkPolicy pod selectors)
• startup command: This is a very niche use-case but when deploying in an istio-injected environment the sidecar needs to be terminated by running additional commands against its endpoint (can provide more details if needed)

Ultimate it would be great to be able to override different portions of the pod spec.

Possible fixes

Add a new section to the Mattermost CRD spec that allows for update check pod spec overrides. This would honestly be beneficial for the primary Mattermost pods as well, since currently you have to utilize resourcePatch to add labels/annotations to the pods (which is not very beginner friendly or easy to navigate, especially with multiple changes.

Proposed Change:

spec:
  podTemplate:
    <allow anything for podspec, or allow subset including annotations/labels/securitycontext
  updateCheck:
    podTemplate:
      <allow anything for podspec, or allow subset including annotations/labels/securitycontext

Local File store

Summary

Why in the CRD definition is not present Local File store? Its not possible to deploy mattermost in k8s using the operator (or the helm chart) with local file store?

In the docs say that is possible to use it: https://docs.mattermost.com/configure/configuration-settings.html#file-storage-system

Expected behavior

Use Local File store in the mattermost deployed in k8s.

Image pull secrets

I have the Mattermost image hosted in private registry that requires authentication.

When building a Mattermost instance using the operator I don't see a way to provide my authentication (aka image pull secrets).

If there is a way to either specify this secret or specify the service account to use for Mattermost which already exists, that would be a great help to me! If not, can we get this added? I'm working on v1.12.0.

Using external mysql

Hey,

Is it possible to use and configuration for a mysql running externally outside the cluster?
There's seems to be some information here:
https://github.com/mattermost/mattermost-operator/blob/master/docs/mattermost-operator/mattermost-operator.yaml#L768
but I cant figure it out.

Thank you.

Configure to use postgresql

There doesn't seem to be way to configure to use postgresql.

Can I use this operator for Mattermost community installation?

Abstraction for operator dependencies

I'm trying to push that upstream of kubernetes
kubernetes/enhancements#706

I guess you understand what it is about :) Do you think you could help me?

Thanks!

Service port standard naming for Istio

Summary

Currently the service that is created by the operator has two ports, app and metrics (code ref).

Istio (service mesh) utilizes the port names for automatic protocol detection for its sidecars. The important part here is If the protocol cannot automatically be determined, traffic will be treated as plain TCP traffic. I ran across this issue with monitoring Mattermost metrics. When you point a prometheus instance at the Mattermost metrics port, it returns a 404 (if Prometheus has Istio sidecars for traffic) due to the "non-standard" port name.

Steps to reproduce

This will be a challenging one to reproduce without knowledge and ability to deploy Istio, Prometheus, and Mattermost. I'm happy to walk through a deployment where I'm seeing this issue if that'd be beneficial (video call or similar might be best for this given the amount of detail and complexity).

Expected behavior

I would expect that Mattermost metrics can be monitored in any situation, including when my Prometheus instance is Istio injected.

Observed behavior (that appears unintentional)

Prometheus returns 404s on the Mattermost endpoints.

Possible fixes

I think there's a couple routes that could be taken. For initial testing to validate that this is the issue I worked on this branch, with the changes essentially just renaming the two port names in the service generation (to http-app and http-metrics so that Istio knows they are HTTP). Confirmed that this works, the only changes required are to the "external" port names, the targetPort doesn't need to be changed.

I'd assume this wouldn't be an ideal solution since it could break things for someone potentially. I think ideally the operator provides a way to override the default names of these ports. Example of a simple way to do this:

spec:
  service:
    appPortName: "http-app" # defaults to app
    metricsPortName: "http-metrics" # defaults to metrics

If we wanted to go further than that and allow more customization on the service (if I'm remember right serviceAnnotations are already a value?), we could refactor things into a service section in the spec similar to what was done with Ingress such as:

spec:
  service:
    annotations: ...
    ports:
      - name: "http-app"
         port: 8065
      - name: "http-metrics"
         port: 8067

I don't know if that exactly makes sense to do but hopefully that gives an idea of what I'm thinking. Ultimately just looking for a way that makes sense to allow port name overriding and is maintainable/extensible for the future.

how to launch a command at start up

hello

i d like to launch a mmctl command at startup
how could i implement it in my mattermost deployement ?

in the apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
i cannot add command there ?

thanks

Minio InitContainer fails with custom cluster DNS

mattermost-operator/pkg/controller/clusterinstallation/minio.go

Line 129 in 018b4f3

connectionString := fmt.Sprintf("%s.%s.svc.cluster.local:%d", minioService.Name, mattermost.Namespace, minioService.Spec.Ports[0].Port)

"%s.%s.svc.cluster.local:%d" is used for the minio connection string.

I have a custom DNS for my cluster so hard-coding the cluster.local in there makes it unable start for me.

Changing service type for Mattermost managed by Operator requires deleting service

Summary

When changing the service type used by Mattermost with useServiceLoadBalancer either true -> false or false -> true the manual intervention of deleting the Service is required as some immutable fields change.

Mattermost Operator should be able to handle it either by doing smart update omitting immutable fields (if possible) or recreating the Service automatically.

Steps to reproduce

• Create any Mattermost instance with useServiceLoadBalancer set to true.
• Wait until Service is created.
• Edit Mattermost manifest setting useServiceLoadBalancer to false.

The reverse change (from false to true) also produces the error and should be handled properly.

Expected behavior

Service is updated and Mattermost reaches a stable state.

Observed behavior (that appears unintentional)

Mattermost is stuck in reconciling state and Mattermost Operator logs errors.

Possible fixes

Perform Service update omitting immutable fields (if possible) or recreate the Service automatically.

Allow disabling of Ingress creation

Summary

Currently the operator/CRD do not allow creation of a Mattermost instance without an ingress. The value spec.ingressName is always required to the best of my knowledge.

Our use case for this is deploying MM without an ingress and using "alternative ingress" via a service mesh and would prefer that this ingress is not created.

Possible fixes

Provide a new value (defaulted to true) that toggles the creation of Ingress. When false, it would also "disable the requirement" to provide a spec.ingressName. There could be numerous ways to implement this, but probably the most backwards compatible would be with a new spec.ingressEnabled flag:

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
...
spec:
  ingressEnabled: false # defaults to true
...

If open to changing the spec slightly, it would seem to make more logical sense and better match other operators to do it with a spec.ingress block:

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
...
spec:
  ingress:
    enabled: true # defaults to true, just added for example
    name: my-mm-ingress
    annotations:
      kubernetes.io/ingress.class: nginx
...

Totally understand if the second is off the table because it would introduce "breaking" changes. Either of these proposed solutions (or really anything that allows us to disable the Ingress) would work for our needs.

Websockets do not work by default on >1 replica

Summary

Websockets do not work when using default installation instructions from here and manual.

Steps to reproduce

Set size: 5000users to enable >1 replica, which creates problem. Set size: 100users to enable only 1 replica, which solves problem.

Observed behavior (that appears unintentional)

When there is more than one replica, using mattermost for any length of time results in missed messages and need to manually refresh to update channels and messages.

Possible fixes

The configuration line above

I have tried the following to try to help:

      nginx.ingress.kubernetes.io/affinity: "cookie"
      nginx.ingress.kubernetes.io/session-cookie-name: "route"
      nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
      nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

and separately, I tried removing the above and instead trying:

      nginx.ingress.kubernetes.io/upstream-hash-by: $remote_addr

Neither of these worked, and it appears websockets is the culprit.

I am currently testing the following, but still too early to say anything:

nginx.ingress.kubernetes.io/websocket-services: "mattermost"

Other issues that might be related:

kubernetes/ingress-nginx#5539

*deleted*

deleted

Proper way to backup/restore config

Summary

Regarding the several different ways config can be handled when using the operator, I’m wondering if there is a proper way to backup the config file? And also restore it?

Expected behavior

Provide a way (or several ways) to handle config backup/restore in Operator documentation.

Observed behavior

Nothing related to config backup/restore is present in Operator documentation

Possible fixes

Configure through environment variables, but this is mostly an alternative solution, not a fix.

It would be great if it was possible to create a configMap from a Mattermost configuration file, and then give it to the operator to mount it in the right place inside the pod(s).

Edit: as a temporary fix, could we use the cp command of kubectl to backup and restore config file?

Add end-to-end test working with external database

Description

Current end-to-end tests of Mattermost Operator run using MySQL Operator and MinIO Operator. We should add another test which runs with an external database configuration.

This could be achieved by creating separate Postgres deployment on the Kind cluster on which the tests are run, and treat it as an external database.

Setting up of the deployment can be done either in bash script or in setup Go code.

We can also explore ways to test external file store in a similar way.

Mattermost Taint/Toleration Support

Summary

The Mattermost Operator does not expose tolerations as part of the scheduling spec:

https://github.com/mattermost/mattermost-operator/blob/master/apis/mattermost/v1beta1/mattermost_types.go#L96

Steps to reproduce

Create Kubernetes nodes with a Taint and try and schedule the mattermost pods on those nodes.

Expected behavior

Given a taint, we should be able to tell the mattermost pods to tolerate the taint via:

apiVersion: installation.mattermost.com/v1beta1
kind: Mattermost
metadata:
  name: mattermost
  namespace: mattermost
spec:
  ... #omited for brevity
  scheduling:
    tolerations:
    - key: "key1"
      operator: "Equal"
      value: "value1"
      effect: "NoSchedule"

Observed behavior (that appears unintentional)

Possible fixes

1. Add toleration here: https://github.com/mattermost/mattermost-operator/blob/master/apis/mattermost/v1beta1/mattermost_types.go#L96

// Scheduling defines the configuration related to scheduling of the Mattermost pods
// as well as resource constraints.
type Scheduling struct {
	// Defines the resource requests and limits for the Mattermost app server pods.
	// +optional
	Resources v1.ResourceRequirements `json:"resources,omitempty"`
	// NodeSelector is a selector which must be true for the pod to fit on a node.
	// Selector which must match a node's labels for the pod to be scheduled on that node.
	// More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
	// +optional
	NodeSelector map[string]string `json:"nodeSelector,omitempty"`
	// If specified, affinity will define the pod's scheduling constraints
	// +optional
	Affinity *v1.Affinity `json:"affinity,omitempty"`
        // +optional
        Tolerations *v1.Tolerations `json:"tolerations,omitempty"`
}

Toleration Spec

2. Pass tolerations into

   mattermost-operator/pkg/mattermost/mattermost.go

   Line 411 in 8df0b9e

   NodeSelector: mattermost.Spec.NodeSelector,

Custom TLS options

Goal

The operator should allow for custom TLS certificates for Ingress.
Currently, the expected secret name is the Ingress manifest's name + -tls-cert.

When using LetsEncrypt via Cert-Manager, the secrets are named however the user requests it to be.

Actions steps

I plan to:

• modify the CRD to support custom TLS certs (similar to Ingress)
• update the resource definitions

Docker image: Executable file not found in $PATH

When moving pods to new nodes (forcing a new image pull), the mattermost-operator deployment fails with ErrImagePull and CrashLoopBackoff, with events like:

Error: failed to start container "mattermost-operator": Error response from daemon: OCI runtime create failed: 
container_linux.go:349: starting container process caused "exec: \"mattermost-operator\": executable file not found in $PATH":
unknown

Specifying the image digest to the last known good image provided a workaround to the issue:
image: mattermost/mattermost-operator@sha256:45fdb00b2e7623ab12f3b8023aa6966db6a7ebd1f4f89409692e657ed9f2d0d6

It seems like the latest built image available has an issue with its CMD.

Environment: Azure AKS 1.18.8, kubectl 1.18.2

Websocket forwarding not working with Nginx ingress

Hi folks,

I've been further tinkering and the websocket can't make a connection from outside my cluster to inside.

First off, everything works fine when I'm posting messages or connecting bots or whatever. The only thing that is a problem is the websocket.

The config has these:

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mattermost
spec:
  image: mattermost/mattermost-enterprise-edition
  size: 100users
  ingressName: matter.p.fqdn.com
  ingressAnnotations:
    kubernetes.io/ingress.class: public
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Connection "upgrade";
      proxy_set_header Upgrade "websocket";
      proxy_http_version 1.1;
      proxy_read_timeout 61s;
  useIngressTLS: true
  version: 5.19.1
  mattermostLicenseSecret: ""
  database:
    storageSize: 55Gi
  minio:
    storageSize: 100Gi
  elasticSearch:
    host: ""
    username: ""
    password: ""

When I check on whether that flowed through from the operator to the ingress, I'm delighted to find:

$ kubectl describe ing mattermost                                                            
Name:             mattermost
Namespace:        default
Address:          10.2.2.210
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  matter-p-fqdn-com-tls-cert terminates matter.p.terharlab.com
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  matter.p.fqdn.com  
                          /   mattermost:8065 (10.233.84.195:8065)
Annotations:              cert-manager.io/cluster-issuer: letsencrypt
                          kubernetes.io/ingress.class: public
                          mattermost.com/last-applied:
                            {"metadata":{"annotations":{"cert-manager.io/cluster-issuer":"letsencrypt","kubernetes.io/ingress.class":"public","nginx.ingress.kubernete...
                          nginx.ingress.kubernetes.io/configuration-snippet:
                            proxy_set_header Connection "upgrade";
                            proxy_set_header Upgrade "websocket";
                            proxy_http_version 1.1;
                            proxy_read_timeout 61s;
                          nginx.ingress.kubernetes.io/proxy-body-size: 1000M
Events:
  Type    Reason          Age                 From                      Message
  ----    ------          ----                ----                      -------
  Normal  AddedOrUpdated  7m3s (x4 over 30h)  nginx-ingress-controller  Configuration for default/mattermost was added or updated

But alas, when I check the actual running config on the ingress controller, I says otherwise.

$ kubectl exec -it ingress-pod-30393  -- cat /etc/nginx/conf.d/default-mattermost.conf
# configuration for default/mattermost
upstream default-mattermost-matter.p.fqdn.com-mattermost-8065 {
	zone default-mattermost-matter.p.fqdn.com-mattermost-8065 256k;
	random two least_conn;
	
	server 10.233.84.195:8065 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
	listen 80;
	listen 443 ssl;
	ssl_certificate /etc/nginx/secrets/default-matter-p-fqdn-com-tls-cert;
	ssl_certificate_key /etc/nginx/secrets/default-matter-p-fqdn-com-tls-cert;
	real_ip_header proxy_protocol;
	server_tokens on;
	server_name matter.p.fqdn.com;

	if ($scheme = http) {
		return 301 https://$host:443$request_uri;
	}
	location / {
		proxy_http_version 1.1;

		proxy_connect_timeout 60s;
		proxy_read_timeout 60s;
		proxy_send_timeout 60s;
		client_max_body_size 1m;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_buffering on;
		
		proxy_pass http://default-mattermost-matter.p.fqdn.com-mattermost-8065;	
	}
}

Honestly I expected to see all of those annotations reflected in the configuration.

When I look at the ingress logs, I see:

10.15.113.47 - - [09/Jun/2020:02:31:35 +0000] "GET /api/v4/websocket HTTP/1.1" 400 192 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15" "-"
10.15.113.47 - - [09/Jun/2020:02:35:44 +0000] "GET /api/v4/websocket HTTP/1.1" 400 192 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15" "-"
10.15.113.47 - - [09/Jun/2020:02:36:35 +0000] "GET /api/v4/websocket HTTP/1.1" 400 192 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15" "-"

And in the mattermost container, I see:

{"level":"error","ts":1591670144.52219,"caller":"api4/websocket.go:28","msg":"websocket connect err.","error":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}
{"level":"error","ts":1591670144.5222652,"caller":"mlog/log.go:174","msg":"Failed to upgrade websocket connection","path":"/api/v4/websocket","request_id":"5u6mx33ib7bh3ksrszdby7autc","ip_addr":"10.15.113.47","user_id":"b8itqkaxwpnr9geee3smx1gbdc","method":"GET","err_where":"connect","http_code":500,"err_details":""}

emojis are always represented as ?

We use default Mattermost 5.18 on Kubernetes and get all emojis only shown as ?.

When I look at the database, it is displayed correctly:

Anybody got any ideas?