Timeout waiting for host : mongo

[gushil]

Timeout waiting for host : mongo

[noliveleger]

@gushil ,
Can you tell us which OS you are using?
Version of docker and docker-compose?
Which interface did you choose during setup?
Does the corresponding ip match mongo.kobo.private in extra_hosts in docker-compose.frontend.override.yml?

[raphaelmerx]

Having the same issue with Ubuntu 18.04, docker version 18.09.5, docker-compose version 1.21.2, and a server mode installation. The extra_hosts section in docker-compose.frontend.override.yml does match the public ip of my machine: mongo.kobo.temp.private:<machine public ip>.

Interestingly when I curl <machine ip>:27017 from the host, mongodb is responding, but after attaching to the kpi container, the same curl command times out. So it seems that this container cannot open an outward connection on port 27017. Yet it can access the outside world (curl google.com works)

[noliveleger]

@raphaelmerx, nothing else is installed?
Would you mind to paste the output of docker ps command?

[raphaelmerx]

Nothing else is installed on this machine. I started from a fresh Ubuntu 18.04 install. Here's the output of docker ps:

enketo@kobo ~ % docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS                              NAMES
08f2c296f53a        kobotoolbox/kobocat:2.019.13                      "/sbin/my_init"          2 days ago          Up 2 minutes        8000/tcp                           kobo-docker_kobocat_1
fdef9a6c2d54        kobotoolbox/nginx:latest                          "/sbin/my_init"          2 days ago          Up 2 days           0.0.0.0:80->80/tcp, 443/tcp        kobo-docker_nginx_1
a72515b6f53e        kobotoolbox/enketo-express-extra-widgets:1.75.3   "/sbin/my_init"          2 days ago          Up 2 days           8005/tcp                           kobo-docker_enketo_express_1
6df7112f21ca        kobotoolbox/kpi:2.019.13                          "/sbin/my_init"          2 days ago          Up 6 seconds        8000/tcp                           kobo-docker_kpi_1
b8b4963583c2        redis:3.2                                         "docker-entrypoint.s…"   2 days ago          Up 2 days           6379/tcp, 0.0.0.0:6380->6380/tcp   kobo-docker_redis_cache_1
1c5c5200e545        redis:3.2                                         "docker-entrypoint.s…"   2 days ago          Up 2 days           0.0.0.0:6379->6379/tcp             kobo-docker_redis_main_1
4837d35051d6        mdillon/postgis:9.5                               "docker-entrypoint.s…"   2 days ago          Up 2 days           0.0.0.0:5432->5432/tcp             kobo-docker_postgres_1

As you can see above kpi and kobocat keep restarting, because they are waiting for mongo, which they cannot access.

[noliveleger]

@raphaelmerx I don't see the mongo container? Did you paste all the output or it's missing? I'm wondering why it didn't start with others?

[raphaelmerx]

Indeed the mongo container was gone one hour ago, yet it was there and accepting connections 3 days ago... Anyways, just did another python run.py, and here's the docker ps output:

enketo@kobo ~/kobo-install % docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS                              NAMES
6e247f9e3a37        kobotoolbox/nginx:latest                          "/sbin/my_init"          6 minutes ago       Up 6 minutes        0.0.0.0:80->80/tcp, 443/tcp        kobo-docker_nginx_1
328ddf9f34bd        kobotoolbox/kpi:2.019.13                          "/sbin/my_init"          6 minutes ago       Up 6 seconds        8000/tcp                           kobo-docker_kpi_1
d59cd029a2bb        kobotoolbox/kobocat:2.019.13                      "/sbin/my_init"          6 minutes ago       Up 6 seconds        8000/tcp                           kobo-docker_kobocat_1
93ed694c2613        kobotoolbox/enketo-express-extra-widgets:1.75.3   "/sbin/my_init"          6 minutes ago       Up 6 minutes        8005/tcp                           kobo-docker_enketo_express_1
8923410e6f82        redis:3.2                                         "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        0.0.0.0:6379->6379/tcp             kobo-docker_redis_main_1
a8db11109a15        mdillon/postgis:9.5                               "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        0.0.0.0:5432->5432/tcp             kobo-docker_postgres_1
866cc4bb3fbd        mongo:3.4                                         "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        0.0.0.0:27017->27017/tcp           kobo-docker_mongo_1
0944ad661f01        redis:3.2                                         "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        6379/tcp, 0.0.0.0:6380->6380/tcp   kobo-docker_redis_cache_1

Mongo is here, yet kpi and kobocat keep restarting because they can't connect to mongo.

[noliveleger]

I gave it a try with these versions:

• ubuntu 18.04.2
• docker: version 18.09.5, build e8ff056
• docker-compose version 1.24.0, build 0aa59064

Using docker with a normal user. not sudo

Even I don't think it's related to your issue, it did not work for me at the first run because of AppArmor. I had to disable it. It blocked docker to stop/down containers.
With AppArmor disabled, it worked as expected. You can check if it changes something for you.

Would you mind to share your .run.conf (can be found in kobo-install directory). Don't forget to remove passwords and other private credentials (aws, smtp etc...) before sharing it.

You can also try advanced options and use another port for Mongo.

[raphaelmerx]

Thanks for suggesting changing the mongo port, it made me remember to check if ufw was running on this machine, which it was. Disabling ufw fixed the connection between kobocat/kpi and mongo!

@gushil please try sudo systemctl stop ufw and see if that works for you.

[gushil]

@raphaelmerx I still got the issue, especially when using in centos in server.
It looks like that either kpi or kobocat cannot connect to mongo using extra_hosts that mapped mongo.kobo.private to host ip address.

If I try with local development host, which is macos, this issue comes intermittently, but most of the time works.

[noliveleger]

@gushil: Which version do you have of docker, docker-compose and CentOS.
Do you have other things installed on the server?

[gushil]

Docker version 18.09.5, build e8ff056
docker-compose version 1.23.1, build b02f1306
CentOS Linux release 7.6.1810 (Core)

I don't think there are many things installed in the server because it's a recent one.

[noliveleger]

I can't say. It works on our side. You may have already checked but as raphael suggested, any firewalls/iptables rules activated?
You can ask for help at https://community.kobotoolbox.org/