Inconsistent server behavior with several replicas (k8s)

[baznikin]

Inconsistent server behavior with several replicas (k8s)

[agnivade]

I'll ping our cloud platform team to chime in @fmartingr @mirshahriar @gabrieljackson

[gabrieljackson]

Hey @baznikin, let's see if we can get to the bottom of this. First off, can you confirm that you are running Mattermost with a license? The issue you described is similar to behavior I have seen in the past where Mattermost server clustering wasn't working as expected. If it is licensed, can you review the logs and/or system console and verify that the multiple servers seem to be talking correctly to one another?

[baznikin]

Hello @gabrieljackson! We run without license; we activate trial license to look around and cancel it couple of weeks later. It was 2 weeks ago and users start to complain 1-1.5 weeks ago. So, yes, it can be correlated.
What messages should I look for? This one, I suppose?..

"This server is not licensed to run in High Availability mode."

[gabrieljackson]

Yep, that's exactly the log line we are looking for. The behavior you are seeing is what can occur when you are running multiple Mattermost server instances, but server clustering is not activated due to not being licensed or other issues. When clustering is not enabled you will see temporary message "missing" behavior. If you refresh your client/webapp the messages will be there, but you may not receive indicators that they have arrived.

[baznikin]

Thanks for clarification! However it is very very bad user experience since MM Operator creates 2 replicas by default (also no "red alarm" fires in my head while I read documentation on deploying, validated architecture topics, but possibly its my fault). If we stumbled over it during testing most likely we just drop MM and choose another competitor. I suppose we should propose MM operator to handle this situation and set replicas count to 1 if no license at moment (i.e. trial period ended or smth else).