Gateway: Combining listeners with HTTPS and TLS protocol

[vehagn]

Gateway: Combining listeners with HTTPS and TLS protocol

[youngnick]

Thanks for the follow-up @vehagn. Yeah, we shouldn't really support this by the Gateway API spec, although it will work in Cilium's implementation now.

We actually configure all the TLS or HTTPS rules on a single Listener with SNI matching first, then HTTP termination if it's a terminated HTTP connection (which TLSRoute is not). Because of this, the config you've given will actually work with Cilium, even though by the Gateway API spec it should not be allowed.

If you could open another issue and mention me, I would really appreciate it. I suspect we may need to add some upstream conformance tests of the Protocol -> Route type mappings visible on https://gateway-api.sigs.k8s.io/guides/tls/ (although I don't know if we've documented that requirement anywhere else, sigh).

But all of that is for me to worry about upstream - having a clear issue here will help me discuss this with the upstream community.

Thanks again for your follow-up!

(Once you've created the new issue, I can link it here and close this one out.)

[youngnick]

HI @vehagn, can you give some more info about where you got that initial error from? I had a look at this and couldn't reproduce it with Cilium 1.15.1 and either Kubernetes 1.24 or 1.29. I created a similar Gateway and it was accepted by Cilium okay:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: default
spec:
  gatewayClassName: cilium
  listeners:
    - name: https-gateway
      protocol: HTTPS
      port: 443
      hostname: "*.test"
      tls:
        certificateRefs:
          - kind: Secret
            name: selfsigned
      allowedRoutes:
        namespaces:
          from: All
    - name: proxmox-tls-passthrough
      protocol: TLS
      port: 443
      hostname: "proxmox.test"
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All

[vehagn]

Thank you for looking into this @youngnick!

Seeing that you couldn't reproduce the error made me investigate further.
I now think the issue lies with Cert-manager and not Cilium.

I tried to create the same Gateway as you did and did not experience any issues before I added

  annotations:
    cert-manager.io/issuer: cloudflare-issuer

can you give some more info about where you got that initial error from?

I got the error from the events for the Gateway. I initially only looked at them in the Argo CD Web GUI, but looking again now using kubectl describe I see that the source is cert-manager-gateway-shim.

I'll report the issue over at the Cert-manager repo and think this issue can be closed.

[vehagn]

I did some further digging and it appears that I'm able to connect a TLSRoute to a Gateway which only supports HTTPRoute kinds. I would expect this to throw some kind of error.

The involved resources are

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test
  namespace: gateway
spec:
  gatewayClassName: cilium
  listeners:
    - protocol: HTTPS
      port: 443
      name: https-gateway
      hostname: "*.example.com"
      tls:
        certificateRefs:
          - kind: Secret
            name: test-cert
      allowedRoutes:
        namespaces:
          from: All

(Notice I've removed the TLS-listener)

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: test
  namespace: test
spec:
  parentRefs:
    - name: test
      namespace: gateway
  hostnames:
    - "test.example.com"
  rules:
    - backendRefs:
        - name: test-svc
          port: 443

Looking at the status of both they appear to have successfully linked

❯ kubectl -n test get tlsroute test -oyaml | yq '.status'

parents:
  - conditions:
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Accepted TLSRoute
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Service reference is valid
        observedGeneration: 1
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
    controllerName: io.cilium/gateway-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: test
      namespace: gateway

❯ kubectl -n gateway get gateway test -oyaml | yq '.status'

addresses:
  - type: IPAddress
    value: 192.168.1.224
conditions:
  - lastTransitionTime: "2024-05-04T08:35:43Z"
    message: Gateway successfully scheduled
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-05-04T08:35:43Z"
    message: Gateway successfully reconciled
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
listeners:
  - attachedRoutes: 1
    conditions:
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Invalid CertificateRef
        reason: Invalid
        status: "False"
        type: Programmed
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Listener Accepted
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Invalid CertificateRef
        reason: InvalidCertificateRef
        status: "False"
        type: ResolvedRefs
    name: https-gateway
    supportedKinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute

Ignoring the Invalid CertificateRef messages the TLSRoute appears to have been successfully accepted by the Gateway.

Is this the intended behaviour, or should I open a new issue?