ServiceMesh 503 when Envoy and App are Co-located with BPF Host Routing

[jschwartzy]

ServiceMesh 503 when Envoy and App are Co-located with BPF Host Routing

[eudoxiabanderbear]

When looking at the flow using tcpdump, we did see that the application pod did receive the SYN call from the ingress. However when the application pop was attempting to send the SYN-ACK back, the packet appears to be unable to route back, to the ingress. Followed by the ingress, attempting to restart the TCP flow.

19:23:00.745947 IP ip-100-66-155-240.us-west-2.compute.internal.46283 > ip-100-66-156-47.us-west-2.compute.internal.7654: Flags [S], seq 2885681828, win 62727, options [mss 8961,sackOK,TS val 340353580 ecr 0,nop,wscale 7], length 0
19:23:00.745967 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149138160 ecr 340353580,nop,wscale 7], length 0
19:23:00.745984 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149138160 ecr 340353580,nop,wscale 7], length 0
19:23:01.755860 IP ip-100-66-155-240.us-west-2.compute.internal.46283 > ip-100-66-156-47.us-west-2.compute.internal.7654: Flags [S], seq 2885681828, win 62727, options [mss 8961,sackOK,TS val 340354590 ecr 0,nop,wscale 7], length 0
19:23:01.755887 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755904 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755906 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0
19:23:01.755909 IP ip-100-66-156-47.us-west-2.compute.internal.7654 > ip-100-66-155-240.us-west-2.compute.internal.46283: Flags [S.], seq 3184591693, ack 2885681829, win 62643, options [mss 8961,sackOK,TS val 3149139170 ecr 340353580,nop,wscale 7], length 0

[sayboras]

Thanks for your issue.

As mentioned in your comment, the workaround is to set bpf.hostLegacyRouting=true. You can probably set endpointRoutes.enabled=true as well.

Based on my previous investigation, most likely it's due to below block, especially the variable ENABLE_SKIP_FIB.

cilium/pkg/datapath/linux/config/config.go

Lines 1048 to 1059 in 451c3b2

	if !option.Config.EnableHostLegacyRouting && option.Config.DirectRoutingDevice != "" {
	directRoutingIface := option.Config.DirectRoutingDevice
	directRoutingIfIndex, err := link.GetIfIndex(directRoutingIface)
	if err != nil {
	return err
	}
	fmt.Fprintf(fw, "#define DIRECT_ROUTING_DEV_IFINDEX %d\n", directRoutingIfIndex)
	if len(option.Config.GetDevices()) == 1 {
	if e.IsHost() || !option.Config.EnforceLXCFibLookup() {
	fmt.Fprintf(fw, "#define ENABLE_SKIP_FIB 1\n")
	}
	}