run distributed training with kubernetes: worker wait for ps reponse forever

[tensorflow]

run distributed training with kubernetes: worker wait for ps reponse forever

[limjoe]

I have the same problem.

[scotthuang1989]

@limjoe I figured it out. This is a problem of "pod network" . you can check this link to find out more. I choose Calico as my pod network. but it don't work well on my system(ubuntu 16.04), actually the Calico official document also said that it requires some manual configuration on ubuntu. So I switch to Canal . Then it works well.

[limjoe]

Thanks a lot, I used flannel network add-on before. And Canal works well indeed.

[LynnaPan]

hi, I encountered the same problem:

2017-10-20 05:03:41.425370: I tensorflow/core/distributed_runtime/master.cc:209] CreateSession still waiting for response from worker: /job:ps/replica:0/task:0 

2017-10-20 05:03:41.425421: I tensorflow/core/distributed_runtime/master.cc:209] CreateSession still waiting for response from worker: /job:worker/replica:0/task:0

Also I switch network to canal, the problem still there.
Is there any way to troubleshooting?