Traffic got routed to an endpoint that does not belong to the target service

[Wenliang-CHEN]

Traffic got routed to an endpoint that does not belong to the target service

[kflynn]

@Wenliang-CHEN Keeping fingers crossed for you -- enjoy the holiday! 🙂

[kflynn]

@Wenliang-CHEN Happy new year!! Just wanted to make sure this was still on your radar. 🙂

[adleong]

Hi @Wenliang-CHEN

This sounds a bit similar to an issue we had where the destination controller could become locked and stop processing service discovery updates. However, this bug was fixed in stable-2.14.2 and should not affect you in stable-2.14.3. In order to rule out that possibility, you could take a look at the endpoints_updates counter metric exposed by the destination controller:

linkerd diagnostics controller-metrics | grep endpoints_updates

You should see this counter incremented when the endpoints of a service change. If, instead, this counter remains at the same value, it means that the destination controller is not processing updates for some reason.

In stable-2.14.4 we added *_informer_lag_secs histogram metrics to the destination controller for even more visibility. If you upgrade to stable-2.14.4 or later you can use these histograms to see if there is a substantial lag between when endpoints are updated in Kubernetes vs when the destination controller processes those updates.

[Wenliang-CHEN]

Hey @adleong , thanks for the reply.

And yes, I do see the endpoints_updates counter incremented after the deployment of the target service: service A. With that I guess the destination controller was processing.

A couple of things worth mentioning:

• the issue happens about 20mins after the deployment of the target service.
• If we restart the deployment that owns the outbound pod, the issue is solved

Does it change anything?

And as action item, I think we will try to update to stable-2.14.4 and take a look at *_informer_lag_secs as well.

Meanwhile, if we found anything new, we will report in the thread again.

Thanks!

[kflynn]

@Wenliang-CHEN Any joy trying with stable-2.14.4? 🙂

[Wenliang-CHEN]

Hey @kflynn not yet...around Christmas holiday. I will let you know 😄

But there has not been another instance since I reported the issue. But to be safe, we are still observing...

[Wenliang-CHEN]

Hey @kflynn happy new year!

And yes, we have not forgotten this. We just upgraded to v2.14.9. And so far we did not get any report about the same issue.

Hopefully the upgrade somehow fixes it. We will monitor it through out Feb. If there is no further report, I think we can close it for now. Thanks!

[Wenliang-CHEN]

Okay, the issue happens again.

We are able to get the linkerd.endpoints_updates, linkerd.endpointslices_informer_lag_seconds.bucket and linkerd.endpoints_informer_lag_seconds.bucket

It seems they go in patterns: the linkerd.endpointslices_informer_lag_seconds.bucket goes with linkerd.endpoints_updates:

And the linkerd.endpoints_informer_lag_seconds.bucket is aways 0

We are not sure how to understand this. Do they mean anything particular? Or are they totally normal?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.