BGPv1 and BGPv2 - Updating a peer's ASN has no effect

[dswaffordcw]

BGPv1 and BGPv2 - Updating a peer's ASN has no effect

[dswaffordcw]

This issue is also present when running only the BGPv1 Control Plane. For this next round of testing, I nuked my local Kube stack and brought up Kube and Cilium in a fresh state.

Then, I applied my BGPv1 bgpp.yaml file, shown below. For this test, I used the same approach of configuring a peer intentionally with the wrong ASN.

I observed the peers being configured after running k apply -f bgpp.yaml by viewing cilium bgp peers. I then edited that bgpp.yaml to configure the correct ASN, and re-ran k apply -f bgpp.yaml. The same issue occurs. Cilium never applies the change and shows a reconciliation error on it's own internal pod's logs:

time="2024-06-17T02:19:37Z" level=info msg="Serving cilium health API at unix:///var/run/cilium/health.sock" subsys=health-server
time="2024-06-17T02:19:51Z" level=info msg="Registering BGP servers for policy with local ASN 65001" component=manager.registerBGPServer subsys=bgp-control-plane
time="2024-06-17T02:19:51Z" level=info msg="Adding peer 10.0.1.1/32 65500 to local ASN 65001" component=NeighborReconciler subsys=bgp-control-plane
time="2024-06-17T02:19:51Z" level=info msg="Add a peer configuration" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:19:51Z" level=info msg="Successfully registered GoBGP servers for policy with local ASN 65001" component=manager.registerBGPServer subsys=bgp-control-plane
time="2024-06-17T02:19:58Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:20:12Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:20:23Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:20:36Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:20:43Z" level=info msg="Adding peer 10.0.1.1/32 65000 to local ASN 65001" component=NeighborReconciler subsys=bgp-control-plane
time="2024-06-17T02:20:43Z" level=error msg="Encountered error reconciling virtual router with local ASN 65001" component=manager.reconcile error="reconciliation of virtual router with local ASN 65001 failed: failed while reconciling neighbor 10.0.1.1/32 65000: failed while adding peer 10.0.1.1 with ASN 65000: can't overwrite the existing peer: 10.0.1.1" subsys=bgp-control-plane
time="2024-06-17T02:20:50Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:21:04Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
time="2024-06-17T02:21:14Z" level=warning msg="sent notification" Data="as number mismatch expected 65500, received 65000" Key=10.0.1.1 Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane

The BGPv1 bgpp.yaml used to first reproduce this issue was:

---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: control-plane
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: bgp-cplane-dev-v4-control-plane
  virtualRouters:
  - localASN: 65001
    neighbors:
    - peerASN: 65500
      peerAddress: 10.0.1.1/32
---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: worker
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: bgp-cplane-dev-v4-worker
  virtualRouters:
  - localASN: 65002
    #exportPodCIDR: true
    serviceSelector:
      #matchLabels:
      #  name: hello-node
      matchExpressions:
       - { key: app, operator: In, values: [hello-node, hello-node2, hello-node3] }
    serviceAdvertisements:
      - LoadBalancerIP # <-- default
      #- ClusterIP      # <-- options
      #- ExternalIP     # <-- options
    neighbors:
    - peerASN: 65000
      peerAddress: 10.0.2.1/32
      #holdTimeSeconds: 180
      #keepAliveTimeSeconds: 60
      #eBGPMultihopTTL: 1
      gracefulRestart:
        enabled: true
        restartTimeSeconds: 60
      advertisedPathAttributes:
      - selectorType: CiliumLoadBalancerIPPool
        selector:
          matchLabels:
            environment: mgmt
        communities:
          standard:
          - 65002:100
          large:
          - 65002:100:1
          #wellknown:
          #- no-export
#      - selectorType: CiliumLoadBalancerIPPool
#        selector:
#          matchLabels:
#            name: vrf-mgmt
#        communities:
#          standard:
#          - 65001:100
#      - selectorType: CiliumLoadBalancerIPPool
#        selector:
#          matchLabels:
#            name: vrf-core-services
#        communities:
#          standard:
#          - 65001:200
#      - selectorType: CiliumLoadBalancerIPPool
#        selector:
#          matchLabels:
#            name: vrf-customers
#        communities:
#          standard:
#          - 65001:300
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "mgmt"
  labels:
    environment: mgmt
spec:
  blocks:
    - cidr: "100.64.8.0/22"
  cidrs:
    - cidr: "100.64.12.0/22"
  #serviceSelector:
  #  matchExpressions:
  #    - {key: app, operator: In, values: [hello-node]}
---
#apiversion: "cilium.io/v2alpha1"
#kind: ciliumloadbalancerippool
#metadata:
#  name: "vrf-core-services"
#spec:
#  blocks:
#  - cidr: "100.64.20.0/24"
#  serviceselector:
#    matchexpressions:
#      - {key: app, operator: in, values: [hello-node2]}
#---
#apiversion: "cilium.io/v2alpha1"
#kind: ciliumloadbalancerippool
#metadata:
#  name: "vrf-customers"
#spec:
#  blocks:
#  - cidr: "100.64.30.0/24"
#  serviceselector:
#    matchexpressions:
#      - {key: app, operator: in, values: [hello-node3]}
---
#apiversion: v1
#kind: service
#metadata:
#  name: test-lb
#  namespace: default
#  annotations:
#    io.cilium/lb-ipam-ips: "100.64.10.1,100.64.20.1,100.64.30.1"
#spec:
#  loadbalancerclass: io.cilium/bgp-control-plane
#  type: loadbalancer
#  ipfamilies:
#    - ipv4
#  ports:
#  - port: 80
#    protocol: tcp
#    targetport: 80
#  selector:
#    app: nginx
#---
#apiversion: v1
#kind: pod
#metadata:
#  name: nginx
#  labels:
#    app: nginx
#spec:
#  containers:
#    - name: nginx
#      image: nginx:1.14.2
#      ports:
#       - containerPort: 80

Viewing Cilium's bgp peers output after initially configuring the first peer with the incorrect ASN (65500 instead of 65000):

$  cilium bgp peers
Node                              Local AS   Peer AS   Peer Address   Session State   Uptime   Family         Received   Advertised
bgp-cplane-dev-v4-control-plane   65001      65500     10.0.1.1       active          0s       ipv4/unicast   0          0
                                                                                               ipv6/unicast   0          0
bgp-cplane-dev-v4-worker          65002      65000     10.0.2.1       established     10s      ipv4/unicast   1          2
                                                                                               ipv6/unicast   0          1

After running k apply -f bgpp.yaml, with the following changed section:

---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: control-plane
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: bgp-cplane-dev-v4-control-plane
  virtualRouters:
  - localASN: 65001
    neighbors:
    - peerASN: 65500
      peerAddress: 10.0.1.1/32

Then, viewing Cilium BGP peers again:

$  cilium bgp peers
Node                              Local AS   Peer AS   Peer Address   Session State   Uptime   Family         Received   Advertised
bgp-cplane-dev-v4-control-plane   65001      65500     10.0.1.1       idle            0s       ipv4/unicast   0          0
                                                                                               ipv6/unicast   0          0
bgp-cplane-dev-v4-worker          65002      65000     10.0.2.1       established     57s      ipv4/unicast   1          2
                                                                                               ipv6/unicast   0          1

At this point, there is no peer present in the applied bgpp.yaml with the Peer AS of 65500. The one configured / applied was changed to 65000.

This appears to be a bug as Cilium is now in a stuck state. One is unable to remove or modify the former peer's configuration.