How to validate internal services?

[chaostoolkit]

How to validate internal services?

[Lawouach]

The microservice_available_and_healthy action is probably something to be mindful of. It's our first action and it tried to be clever by assuming a microservice was a clearly defined thing. Today, the approach tend to be to consider your pods/deployments as their own thing and consider what you can learn at that level.

You can learn from anywhere, in your case, because your services aren't directly visible to the internet, maybe they still have an impact in the overall workflow your user is following? Maybe at a higher level?

For instance, assume I'm killing a cache server, I'm not going to learn if the cache is up or not? But I am interested in understand the increased latency for my users when they connect to the application.

Try to locate the level at which you are interested in learning something new and look there, even if you impact something deeper and lower level.

does this make sense?

[SidGrundfos]

I guess so. But as these services are used as internal endpoints to be consumed by a service in the same cluster, it might be a good idea to know about the status of the endpoints even if the pods are down.

[Lawouach]

You could run the experiment from the cluster itself (run chaostoolkit as a pod/job?)

[SidGrundfos]

Yes, I will try the alternative. Thanks!

[Lawouach]

No pb. Another solution, use a probe to query your monitoring solution for deleted pods?

[SidGrundfos]

yes, that seems easy :)