Envoy external auth and UDS example tutorial does't work

[Sanskarzz]

Envoy external auth and UDS example tutorial does't work

[ashutosh-narkar]

Thanks for reporting this @Sanskarzz. If you'd like to contribute a fix that would be great! Thanks.

[Sanskarzz]

Hey @ashutosh-narkar Yes, I would like to contribute. However, I have tried debugging and checking the Envoy logs to identify the problem, but I couldn't find a solution. It would be great if you could guide me or provide me with the steps to follow to resolve this issue.

Here are the logs of envoy when i made curl request

[2024-05-25 06:47:51.313][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:47:51.313][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:47:51.313][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:47:51.313][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:47:51.377][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:47:56.309][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:47:56.309][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:47:56.309][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:47:56.309][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:47:56.378][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:47:56.723][22][debug][conn_handler] [source/server/active_tcp_listener.cc:140] [C2] new connection from 10.244.0.1:46174
[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/conn_manager_impl.cc:274] [C2] new stream
[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/conn_manager_impl.cc:867] [C2][S1304557893114863817] request headers complete (end_stream=true):
':authority', '192.168.49.2:32286'
':path', '/people'
':method', 'GET'
'user-agent', 'curl/7.81.0'
'accept', '*/*'
'authorization', 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU'

[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/filter_manager.cc:835] [C2][S1304557893114863817] request end stream
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:363] Finish with grpc-status code 0
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:215] notifyRemoteClose 0 
[2024-05-25 06:47:56.726][22][debug][http] [source/common/http/filter_manager.cc:947] [C2][S1304557893114863817] Sending local reply with details ext_authz_denied
[2024-05-25 06:47:56.726][22][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C2][S1304557893114863817] encoding headers via codec (end_stream=true):
':status', '403'
'date', 'Sat, 25 May 2024 06:47:56 GMT'
'server', 'envoy'

[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:404] Stream cleanup with 0 in-flight tags
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:393] Deferred delete
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:165] GoogleAsyncStreamImpl destruct
[2024-05-25 06:47:56.726][22][debug][connection] [source/common/network/connection_impl.cc:640] [C2] remote close
[2024-05-25 06:47:56.726][22][debug][connection] [source/common/network/connection_impl.cc:249] [C2] closing socket: 0
[2024-05-25 06:47:56.726][22][debug][conn_handler] [source/server/active_stream_listener_base.cc:120] [C2] adding to cleanup list
[2024-05-25 06:48:01.314][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:48:01.314][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:48:01.314][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:48:01.314][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:48:01.383][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:48:06.310][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:48:06.310][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:48:06.310][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:48:06.310][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:48:06.382][1][debug][main] [source/server/server.cc:229] flushing stats

[srenatus]

If you control the envoy CLI args, try adding --component-log-level ext_authz:trace and see what it logs then.

[Sanskarzz]

Hey @ashutosh-narkar Are you sure grpc_service.google_grpc.target_uri field in envoy config supports unix domain sockets (uds) , just asking because i have not found any documentation on that .

- name: envoy.ext_authz
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
                  transport_api_version: V3
                  with_request_body:
                    max_request_bytes: 8192
                    allow_partial_message: true
                    pack_as_bytes: true
                  failure_mode_allow: false
                  grpc_service:
                    google_grpc:
                      stat_prefix: ext_authz
                      target_uri: unix:///run/opa/sockets/auth.sock
                    timeout: 0.5s

[ashutosh-narkar]

It was working before so I would imagine the target_uri setting should be fine. I would check if there's some new config setting in Envoy that maybe causing this or if something changed in the Envoy config. Also I would check if OPA's getting the expected request.

[Sanskarzz]

Thank you, @ashutosh-narkar , for your response. If there is an opportunity for me to contribute, please let me know where the issue lies, and I would be happy to assist. Actually i'm LFX mentee currently working on the kyverno-envoy-plugin, I have learned a great deal from your work on the OPA-envoy-plugin. I appreciate your contributions to open source; they have been incredibly helpful and inspiring. Thanks for doing open source.

[Sanskarzz]

@srenatus @ashutosh-narkar
Here the request/log info after adding --component-log-level ext_authz:trace in args

sanskar@sanskar-HP-Laptop-15s-du1xxx:~$ kubectl logs "$(kubectl get pod -l app=example-app -o jsonpath={.items..metadata.name})" -c envoy -f
[2024-05-29 12:26:19.095][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] Sending CheckRequest: attributes {
  source {
    address {
      socket_address {
        address: "10.244.0.1"
        port_value: 31005
      }
    }
  }
  destination {
    address {
      socket_address {
        address: "10.244.0.5"
        port_value: 8000
      }
    }
  }
  request {
    time {
      seconds: 1716985579
      nanos: 93830000
    }
    http {
      id: "6536821954363734"
      method: "GET"
      headers {
        key: ":authority"
        value: "192.168.49.2:31814"
      }
      headers {
        key: ":method"
        value: "GET"
      }
      headers {
        key: ":path"
        value: "/people"
      }
      headers {
        key: ":scheme"
        value: "http"
      }
      headers {
        key: "accept"
        value: "*/*"
      }
      headers {
        key: "authorization"
        value: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU"
      }
      headers {
        key: "user-agent"
        value: "curl/7.81.0"
      }
      headers {
        key: "x-forwarded-proto"
        value: "http"
      }
      headers {
        key: "x-request-id"
        value: "65f25374-f403-4fbf-8840-008f5e490844"
      }
      path: "/people"
      host: "192.168.49.2:31814"
      scheme: "http"
      protocol: "HTTP/1.1"
    }
  }
  metadata_context {
  }
}

[2024-05-29 12:26:19.102][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:48] Received CheckResponse: status {
  code: 7
}
dynamic_metadata {
  fields {
    key: "decision_id"
    value {
      string_value: "ac1824e9-6d1f-43f2-931f-69aa8f106e40"
    }
  }
}

curl request

sanskar@sanskar-HP-Laptop-15s-du1xxx:~/opa-envoy-plugin/examples/envoy-uds$ curl -i -H "Authorization: Bearer "$ALICE_TOKEN"" http://$SERVICE_URL/people
HTTP/1.1 403 Forbidden
date: Wed, 29 May 2024 12:26:19 GMT
server: envoy
content-length: 0

[srenatus]

[2024-05-29 12:26:19.102][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:48] Received CheckResponse: status {
  code: 7
}
dynamic_metadata {
  fields {
    key: "decision_id"
    value {
      string_value: "ac1824e9-6d1f-43f2-931f-69aa8f106e40"
    }
  }
}

So that's definitely a response from opa-envoy-plugin, meaning the UDS communication works. The problem thus has something to do with you config and policy. Can you share them?

[Sanskarzz]

I don't think so this proves the response from opa-envoy-plugin it can be sent by envoy filter also.
I am using same config and policy as provided in example demo checkout this.

[srenatus]

Are you sure? That decision ID in dynamic metadata is generated by opa-envoy-plugin and sent as part of the response. Envoy doesn't make this up.

[Sanskarzz]

Not fully sure leave it. I found where was the error the ALICE_TOKEN was provided in the already expired I will PR this soon.
But can you please help me with these log error in envoy I could not find where is the problem i don't have much experience of envoy .

sanskar@sanskar-HP-Laptop-15s-du1xxx:~$ kubectl logs "$(kubectl get pod -l app=testapp -o jsonpath={.items..metadata.name})" -c envoy -f
[2024-05-29 18:36:58.404][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:111] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter calling authorization server
[2024-05-29 18:36:58.405][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] Sending CheckRequest: attributes {
  source {
    address {
      socket_address {
        address: "10.244.0.1"
        port_value: 40835
      }
    }
  }
  destination {
    address {
      socket_address {
        address: "10.244.0.6"
        port_value: 7000
      }
    }
  }
  request {
    time {
      seconds: 1717007818
      nanos: 389537000
    }
    http {
      id: "566655657751400563"
      method: "GET"
      headers {
        key: ":authority"
        value: "192.168.49.2:32430"
      }
      headers {
        key: ":method"
        value: "GET"
      }
      headers {
        key: ":path"
        value: "/book"
      }
      headers {
        key: ":scheme"
        value: "http"
      }
      headers {
        key: "accept"
        value: "*/*"
      }
      headers {
        key: "authorization"
        value: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0Iiwic3ViIjoiWVd4cFkyVT0ifQ.ja1bgvIt47393ba_WbSBm35NrUhdxM4mOVQN8iXz8lk"
      }
      headers {
        key: "user-agent"
        value: "curl/7.81.0"
      }
      headers {
        key: "x-forwarded-proto"
        value: "http"
      }
      headers {
        key: "x-request-id"
        value: "eecc8745-23d9-4851-9e6b-13d274972058"
      }
      path: "/book"
      host: "192.168.49.2:32430"
      scheme: "http"
      protocol: "HTTP/1.1"
    }
  }
  metadata_context {
  }
  route_metadata_context {
  }
}

[2024-05-29 18:36:58.406][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:116] CheckRequest call failed with status: Internal
[2024-05-29 18:36:58.406][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:468] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter rejected the request with an error. Response status code: 403
[2024-05-29 18:36:58.412][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:221] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter has 0 response header(s) to add and 0 response header(s) to set to the encoded response:

[srenatus]

But can you please help me with these log error in envoy I could not find where is the problem i don't have much experience of envoy .

Now it looks like a problem calling opa-envoy-plugin. Or rather, opa-envoy-plugin seems to have hit some error. Can you check and share its logs, too?

Also, it might help to enable decision logs with opa-envoy-plugin: you'll see exactly when it has gotten a request, with which inputs, and what the result was.

decision_logs:
  console: true