Unable to push app to builder (deis-workflow)

[davidchua]

Unable to push app to builder (deis-workflow)

[bacongobbler]

I believe this error is usually attributed to networking issues (usually DNS-related) within slugbuilder to the outside world. IIRC Vagrant somehow inherits the host's DNS server information. Can you try using the kubectl troubleshooting guide and check the DNS setup in your cluster?

[davidchua]

Is there something specific I should be looking for?

Just for reference, I've then tried to create a new app via the deis pull method (https://deis.com/docs/workflow/quickstart/deploy-an-app/) which works.

$ deis pull deis/example-go -a global-zirconia
Creating build... done
$ curl http://global-zirconia.10.247.16.91.nip.io
Powered by Deis

[bacongobbler]

deis pull should work normally because the commands are run from the host's docker socket which inherits the node's networking. I believe this issue crops up every so often in Vagrant due to pod networking.

Is there something specific I should be looking for?

Yes, you can look in /etc/resolv.conf in the builder container with the following:

kubectl exec -it --namespace=deis deis-builder-somerandomstring bash
cat /etc/resolv.conf
curl --verbose google.com  # to determine if the pod's resolving DNS lookups correctly

[bacongobbler]

For reference here's when I hit the same issue: deis/builder#55

[davidchua]

thanks @bacongobbler!

Looks like I can't curl google.com as you expected.

How does the pod inherit its nameservers from and where should I be modifying the resolv.conf. Directly on the pod itself?

[bacongobbler]

So for myself, my ISP was providing my computer with a DNS search domain called bacongobbler.com, making all requests from https://google.com to become https://google.com.bacongobbler.com. I edited the host machine's /etc/resolv.conf (as in my physical host, not the virtual machines), then destroyed my cluster and re-provisioned from scratch.

deis/builder#55 (comment) goes into a little more detail on how I resolved this.

[davidchua]

@bacongobbler done! got builder curling google.com successfully.

Unfortunately, still getting a pre-receive hook error.

➜  2048 git:(master) git push deis master
Counting objects: 768, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (329/329), done.
Writing objects: 100% (768/768), 446.45 KiB | 0 bytes/s, done.
Total 768 (delta 435), reused 768 (delta 435)
Starting build... but first, coffee!
remote: 2016/10/10 04:59:15 Error running git receive hook [watching events for builder pod startup (Giving up; pod went into failed status:
remote: []:)]
To ssh://deis-builder.10.247.16.91.nip.io:2222/ironic-hacienda.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/ironic-hacienda.git'

[bacongobbler]

@davidchua interesting that kubernetes didn't post any debug information about why the pod went into failed status. If you're still seeing this issue I'd suggest diving into the kubernetes logs or watching the events api in the deis namespace to investigate why slugbuilder is failing.

[bacongobbler]

looks like this issue has since been resolved, so I'm closing this due to inactivity. If you're still seeing this issue, please post a new issue at https://github.com/deis/builder as this repo is intended for v1 issues. Thanks!