Minikube with Istio Gateway Connection Refused

[istio]

Minikube with Istio Gateway Connection Refused

[sdelatorre]

Istio will run on minikube if I skip the rbac files. However, the ingress component becomes unresponsive after a minikube restart (e.g. connection refused). The only way to make it responsive again is to destroy the VM and reinstall everything from scratch. Reinstalling the ingress by deleting the component and re-adding it works, however, is loses the ability to communicate with prometheus and service graph (zipkin somehow still works) so a fresh VM install is required. Are there any logs that I can provide that will help solve this issue?

I do see a bunch of these errors in the logs:

W0615 19:38:35.063655 1 ingress.go:86] failed to fetch http://istio-pilot:8080/v1alpha/secret/istio-proxy/ingress Get http://istio-pilot:8080/v1alpha/secret/istio-proxy/ingress: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[rrekas]

I too get this same problem with minikube and the bookinfo sample. Usually running
kubectl delete pods --all
then re running
export GATEWAY_URL=$(kubectl get po -l istio=ingress -o 'jsonpath={.items[0].status.hostIP}'):$(kubectl get svc istio-ingress -o 'jsonpath={.spec.ports[0].nodePort}') fixes the problem.

I notice that usually when this happens the IP address/port changes in the gateway url.

[nprice1]

I tried deleting my .minikube directory and reinstalling from scratch both with the rbac-alpha and the rbac-beta yaml files, still getting the error. I'll keep investigating and I'll update if I figure out the issue. Thank you for your help!

[DCRichards]

I've found just deleting the ingress pod seems to resolve all issues, which also seems to be more evidence towards the above being the issue.

[rshriram]

Has this VM been up and running when you put your machine to sleep ? We have seen this issue on our macs when we put it to sleep. If that's the case, the only recourse is to possibly recreate the minikube VM. @kyessenov @frankbu ?? On Wed, Jun 7, 2017 at 12:45 PM nprice1 <[email protected]> wrote:

[nprice1]

I've tried deleting the VM using minikube delete and then removing the ~/.minikube/ directory, then reinstalling from scratch. That does not resolve the issue unfortunately.

[frankbu]

Maybe your networking is screwed up (I've seen that before, never really figured out why - but fixed itself somehow), but I noticed inconsistency in your values, above. You showed the running istio-ingress service like this:

istio-ingress   10.0.0.96    <pending>     80:31126/TCP,443:30916/TCP    14m

and said your GATEWAY_URL is this: 192.168.99.100:31448

It should be 192.168.99.10x:31126 where the x depends on the number minikube delete/restarts (0, 1, 2 ...) and the port is the nodeport associated with port 80 of istio-ingress.

Make sure you rerun the command to reset GATEWAY_URL whenever you restart istio.

[nprice1]

Sorry that was my fault, when I originally posted the question I didn't include the actual value of the GATEWAY_URL, so I reinstalled and added that one. I just did a fresh install of everything, here is the result of kubectl get pods:

NAME                              READY     STATUS    RESTARTS   AGE
details-v1-931928059-2d2dk        2/2       Running   0          6m
grafana-1261931457-v1ztm          1/1       Running   0          12m
istio-egress-703620233-vb4bd      1/1       Running   0          12m
istio-ingress-925683244-bmhpr     1/1       Running   0          12m
istio-manager-2910860705-sqjvr    2/2       Running   0          12m
istio-mixer-2335471611-n5vz9      1/1       Running   0          12m
productpage-v1-3825081635-4339z   2/2       Running   0          6m
prometheus-3067433533-w7zgm       1/1       Running   0          12m
ratings-v1-4018280497-t1p1n       2/2       Running   0          6m
reviews-v1-2660898904-qjcp6       2/2       Running   0          6m
reviews-v2-3326941274-gb4cw       2/2       Running   0          6m
reviews-v3-3992983644-w16qf       2/2       Running   0          6m
servicegraph-3127588006-7fzsc     1/1       Running   0          12m

Here is the result of kubectl get services:

NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)                       AGE
details         10.0.0.115   <none>        9080/TCP                      6m
grafana         10.0.0.45    <pending>     3000:32496/TCP                13m
istio-egress    10.0.0.84    <none>        80/TCP                        13m
istio-ingress   10.0.0.32    <pending>     80:30332/TCP,443:30673/TCP    13m
istio-manager   10.0.0.227   <none>        8080/TCP,8081/TCP             13m
istio-mixer     10.0.0.247   <none>        9091/TCP,9094/TCP,42422/TCP   13m
kubernetes      10.0.0.1     <none>        443/TCP                       13m
productpage     10.0.0.120   <none>        9080/TCP                      6m
prometheus      10.0.0.69    <pending>     9090:30818/TCP                13m
ratings         10.0.0.22    <none>        9080/TCP                      6m
reviews         10.0.0.108   <none>        9080/TCP                      6m
servicegraph    10.0.0.139   <pending>     8088:31741/TCP                13m

Here is my $GATEWAY_URL: 192.168.99.100:30332

The ports line up, but I'm still getting the connection refused error.

[frankbu]

Make sure that the IP (192.168.99.100) is equal to minikube ip.

The only other thing I can think of is what version of minikube are you using? 0.19.0 has a bug with TPRs, that's fixed in 0.19.1, so if you're not already using it try upgrading.

[nprice1]

I'm using 0.19.1 of minikube, and it appears my minikube IP matches: 192.168.99.100. This seems like a weird one.

[frankbu]

@rrekas that is working as expected. Whenever you restart Istio it will use a new randomly generated nodeport, so you need to set the GATEWAY_URL environment variable to the new value or you will get a connection refused error.

@nprice1 try removing your .minikube directory and then reinstall from scratch (with minikube 0.9.1). The rbac instructions (as documented) may be off though - I noticed that with 0.9.1, kubectl api-versions | grep rbac seems to be returning both alpha and beta now

rbac.authorization.k8s.io/v1alpha1
rbac.authorization.k8s.io/v1beta1

but even though the instructions said to use rbac-beta in this case, I needed to use rbac-alpha instead. But ultimately, it all came up and worked.

Please let us know if you get to the bottom of your problem, so we can add it to the troubleshooting docs, in case anyone else hits it. Thanks.

[nprice1]

@sdelatorre sorry for the late reply, I haven't had a chance to dive back into this issue. I have tried complete removal of the VM and deleting the minikube directory and starting from scratch, but the problem persists. I've tried looking through the minikube logs to see if I can find any issues but I don't see anything obvious, is there something you would recommend looking for or a log file I may not be aware of?

[sdelatorre]

@nprice1 I'm not totally sure either, I'm still trying to figure it out. When I start from a fresh VM I make sure to run the following:

minkube stop
minkube delete
rm -rf ~/.minikube
rm -rf ~/.kube

That usually starts me off with a clean slate.

[DCRichards]

As @sdelatorre's comment shows, it looks to be that the ingress pod initialises before the pilot, which is causing these errors and preventing the ingress functioning properly, it seems it doesn't recover, which I guess is the bug here.

[ldemailly]

any chance you can try with code from master: (only try this if comfortable with each step):

# on a mac (on linux change the istioctl download url)
git clone https://github.com/istio/istio.git
source istio.VERSION
kubectl apply -f install/kubernetes/istio.yaml
curl $ISTIOCTL_URL/istioctl-osx > ./istioctl-osx
chmod 755 istioctl-osx
./istioctl-osx kube-inject --tag $PILOT_TAG --hub $PILOT_HUB samples/bookinfo/bookinfo.yaml | kubectl apply -f -

and see if you still get ingress issues / can reproduce that init order problem ?

[nprice1]

@ldemailly I tried running those commands using master (and I also updated to version v0.22.2 of minikube and deleted everything from my initial minikube install). When I try and apply istio.yaml I get some errors, here are the results:

namespace "istio-system" created
clusterrole "istio-pilot-istio-system" created
clusterrole "istio-initializer-istio-system" created
clusterrole "istio-mixer-istio-system" created
clusterrole "istio-ca-istio-system" created
clusterrole "istio-sidecar-istio-system" created
clusterrolebinding "istio-pilot-admin-role-binding-istio-system" created
clusterrolebinding "istio-initializer-admin-role-binding-istio-system" created
clusterrolebinding "istio-ca-role-binding-istio-system" created
clusterrolebinding "istio-ingress-admin-role-binding-istio-system" created
clusterrolebinding "istio-egress-admin-role-binding-istio-system" created
clusterrolebinding "istio-sidecar-role-binding-istio-system" created
clusterrolebinding "istio-mixer-admin-role-binding-istio-system" created
configmap "istio-mixer" created
service "istio-mixer" created
serviceaccount "istio-mixer-service-account" created
deployment "istio-mixer" created
customresourcedefinition "rules.config.istio.io" created
customresourcedefinition "attributemanifests.config.istio.io" created
customresourcedefinition "deniers.config.istio.io" created
customresourcedefinition "listcheckers.config.istio.io" created
customresourcedefinition "memquotas.config.istio.io" created
customresourcedefinition "noops.config.istio.io" created
customresourcedefinition "prometheuses.config.istio.io" created
customresourcedefinition "stackdrivers.config.istio.io" created
customresourcedefinition "statsds.config.istio.io" created
customresourcedefinition "stdios.config.istio.io" created
customresourcedefinition "svcctrls.config.istio.io" created
customresourcedefinition "checknothings.config.istio.io" created
customresourcedefinition "listentries.config.istio.io" created
customresourcedefinition "logentries.config.istio.io" created
customresourcedefinition "metrics.config.istio.io" created
customresourcedefinition "quotas.config.istio.io" created
customresourcedefinition "reportnothings.config.istio.io" created
namespace "istio-config-default" created
configmap "istio" created
service "istio-pilot" created
serviceaccount "istio-pilot-service-account" created
deployment "istio-pilot" created
service "istio-ingress" created
serviceaccount "istio-ingress-service-account" created
deployment "istio-ingress" created
service "istio-egress" created
serviceaccount "istio-egress-service-account" created
deployment "istio-egress" created
serviceaccount "istio-ca-service-account" created
deployment "istio-ca" created
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=attributemanifest
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=attributemanifest
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=stdio
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=logentry
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=rule
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=metric
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=prometheus
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=rule
unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=rule

[nprice1]

Also I tried using minikube v0.22.2 with Istio 1.5.0 and I have a problem initializing the istio-manager pod for some reason:

istio-manager-1384464148-pdwz6    0/2       ImagePullBackOff   0          1m

Not sure if that is related at all.

[ldemailly]

if you use a 0.1.x release you must use the last one which 0.1.6, not 0.1.5
(I am curious where you got a reference to 0.1.5 from btw given 0.1.6 is the latest stable release since early june, everything (should) point(s) to it)

cc @douglas-reid for the config error with master and @frankbu who has experience with minikube

[nprice1]

@ldemailly sorry for the confusion, I used 1.5.0 because that is what I was using when I originally posted this bug. I have updated my versions of minikube, kubernetes-cli, and istio. I can still reproduce the problem with istio 1.6.0, however my above problem with the master branch is resolved probably with my update of kubernetes-cli. I was able to run the commands you gave me successfully (however I did have to change the last one, hopefully it is still equivalent):

git clone https://github.com/istio/istio.git
source istio.VERSION
kubectl apply -f install/kubernetes/istio.yaml
curl $ISTIOCTL_URL/istioctl-osx > ./istioctl-osx
chmod 755 istioctl-osx
kubectl apply -f <(./istioctl-osx kube-inject --tag $PILOT_TAG --hub $PILOT_HUB -f samples/bookinfo/kube/bookinfo.yaml)

Everything appears to have run successfully. However now when I run the kubectl get pods command the istio pods don't appear. Here are my pods:

NAME                             READY     STATUS    RESTARTS   AGE
details-v1-2202528699-fpqbb      2/2       Running   0          5m
productpage-v1-849198989-8bncp   2/2       Running   0          5m
ratings-v1-217391091-6dtpz       2/2       Running   0          5m
reviews-v1-3397570414-69vb6      2/2       Running   0          5m
reviews-v2-755907806-0mvx1       2/2       Running   0          5m
reviews-v3-1808066742-nhbnm      2/2       Running   0          5m

This means my original command for getting the $GATEWAY_URL doesn't work. Is there another method for retrieving that? Also is it expected for the istio pods to not appear?

[nprice1]

@ldemailly nevermind it looks like the istio pods and services are now in the istio-system namespace. I wasn't able to use the same method to get my $GATEWAY_URL, but I used the IP address I got from the minikube dashboard and the port number from the istio-ingress service and it looks like it is working! I'm able to see the /productpage service in my broswer. I'm going to mark this issue as closed, thank you so much for your help! I would still appreciate a description of what was actually occurring if possible, but this seemed like a weird issue that could have a huge number of causes.

[ldemailly]

you are welcome! if the instructions on istio.io still don't work in a few days, do hit the "Report a Doc Issue" at the bottom of the page