Traffic Management: ALL https requests work (even without a ServiceEntry)

[filipre]

Traffic Management: ALL https requests work (even without a ServiceEntry)

[louiscryan]

Hey. Take a look at

https://istio.io/docs/setup/kubernetes/helm-install/

The values for the fields come from the annotations block of the config-map itself. If you don't have an annotation there for excludeIP then we'll have to look elsewhere.

You can also see what the value is being expanded to by looking at the podspec of a pod that was mutated by the webhook

[louiscryan]

Hmm. Its possible someone has configured the ip whitelist for bypass

https://istio.io/docs/tasks/traffic-management/egress/

[filipre]

Thanks for the answer! This seems likely. However, I am a bit confused by the documentation. It explains how to set the value during the Istio installation but not how to read it out. I took a look (kubectl describe) at the ConfigMap of the istio-sidecar-injector but it only shows the template and not the actual values. Is there a simple way to read out global.proxy.includeIPRanges without the usage of helm (we didn't even use helm for the installation nor any specified cloud provider there).

We use this istio-init-image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15.

Here is the output of the describe command:

➜  kubectl describe ConfigMap istio-sidecar-injector -n istio-system -R
Name:         istio-sidecar-injector
Namespace:    istio-system
Labels:       app=istio
              chart=istio-0.8.0
              heritage=Tiller
              istio=sidecar-injector
              release=RELEASE-NAME
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"config":"policy: enabled\ntemplate: |-\n  initContainers:\n  - name: istio-init\n    image: gcr.io/istio-release/proxy_init...

Data
====
config:
----
policy: enabled
template: |-
  initContainers:
  - name: istio-init
    image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always

  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    gcr.io/istio-release/proxyv2:release-0.8-20180531-09-15
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
Events:  <none>

Or do I have to inspect a different resource to find out whether someone could configure a bypass?

Greetings
René

[filipre]

Hi,

ok, I am starting to understand it. So for each pod there will be an istio-init and istio-proxy container which will be initialised by the template defined in this ConfigMap I pasted above. If you use helm and you set global.proxy.includeIPRanges, then it will set the .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges" annotation with the value you put into it. However, if you don't provide this annotation (or use helm), then Istio will use '*' which means, ALL outbound traffic is redirected to Envoy.

Here are the initContainer-arguments of one of our services:

  initContainers:
  - args:
    - -p
    - "15001"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - 3000,
    - -d
    - ""
    image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

However, that means that all outbound traffic (like you have when you do curl https://www.facebook.com/ inside that container) for this service should go to Envoy first. But, doing the curl command gives me the full HTML document of Facebook, even though we didn't whitelist it. Can you confirm that this is right what I wrote and that it is not bypassing the service?

Maybe reinstalling Istio on our cluster solves this problem. Though, it would be nice to identify the root cause.

[louiscryan]

That all looks right. I think I know what is going on here. Your request is HTTPS not HTTP and because TLS is originated by curl the traffic is opaque to the proxy, i.e. it looks like TCP. By default Istio acts like a transparent TCP proxy, primarily because we don't want your apps to break when you install. We configure Envoy to forward traffic to the original target ip & port <https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/service_discovery#arch-overview-service-discovery-types-original-destination> in this mode. IIRC you should be able to see metrics about TCP connections being created that line up with your testing in Prometheus (Its also possible but less likely that someone has defined a ServiceEntry resource that captures the resolved IP for www.facebook.com in the ServiceEntry.addresses field) Instead of curl-ing HTTPS, try HTTP instead. If you do this outside Istio you will get a 302 response. If you do this with Istio it doesn't know how to resolve facebook.com (no ServiceEntry) and you'll get either 404/503 Aside, the fact that HTTPS will pass through transparently and HTTP does not is somewhat inconsistent and something we frequently discuss. If you're feeling adventurous you can create a ServiceEntry for ' www.facebook.com' and define its IPs, & ports. Then when you use HTTP to address it inside the mesh it should succeed.

…

On Wed, Jun 20, 2018 at 2:35 AM René Filip ***@***.***> wrote: Hi, ok, I am starting to understand it. So for each pod there will be an istio-init and istio-proxy container which will be initialised by the template defined in this ConfigMap I pasted above. *If* you use helm *and* you set global.proxy.includeIPRanges, then it will set the .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges" annotation with the value you put into it. However, if you don't provide this annotation (or use helm), then Istio will use '*' which means, ALL outbound traffic is redirected to Envoy. Here are the initContainer-arguments of one of our services: initContainers: - args: - -p - "15001" - -u - "1337" - -m - REDIRECT - -i - '*' - -x - "" - -b - 3000, - -d - "" image: gcr.io/istio-release/proxy_init:release-0.8-20180531-09-15 imagePullPolicy: IfNotPresent name: istio-init resources: {} securityContext: capabilities: add: - NET_ADMIN privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File However, that means that all outbound traffic (like you have when you do curl https://www.facebook.com/ inside that container) for this service should go to Envoy first. But, doing the curl command gives me the full HTML document of Facebook, even though we didn't whitelist it. Can you confirm that this is right what I wrote and that it is not bypassing the service? Maybe reinstalling Istio on our cluster solves this problem. Though, it would be nice to identify the root cause. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#389 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIoKPICyEIXHplux5Eh8RxjRObuyzHfwks5t-hdrgaJpZM4Ul_PZ> .

[filipre]

Hi,

By default Istio acts like a transparent TCP proxy, primarily because we don't want your
apps to break when you install.

Are you sure about that? Because then I don't understand the documentation. If Istio acts like a transparent TCP proxy, why do you have to add this ServiceEntry to make https://www.google.com work?

cat <<EOF | istioctl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google-ext
spec:
  hosts:
  - www.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
EOF

In the documentation you do exactly the thing I do (notice that you use HTTPS aswell):

$ curl https://www.google.com

Also, when I tested it on Minikube, https://www.google.com only worked after I added the ServiceEntry entry above. Once I removed it, it didn't work again (which I expected!).

Regarding our problem: We surely didn't whitelist www.facebook.com. It turned out, that the curl requests only sometimes work (80% maybe?). This makes us believe, that there might be some "zombie" process running or somehow two different versions of envoy or something like that. The best solution would be to reinstall Istio. Still, it would be nice if you could resolve my confusion regarding the "transparent TCP proxy" :)

[filipre]

ok. For anyone having the same issue: It turned out that the resolution field of the ServiceEntry field was missing. If it is missing it defaults to NONE.

REQUIRED: Service discovery mode for the hosts. Care must be taken when setting the resolution mode to NONE for a TCP port without accompanying IP addresses. In such cases, traffic to any IP on said port will be allowed (i.e. 0.0.0.0:).
https://istio.io/docs/reference/config/istio.networking.v1alpha3/