Giter Site home page Giter Site logo

Comments (38)

kennethjiang avatar kennethjiang commented on June 30, 2024

We are a little overwhelmed at this moment. Will spin up a Fedora VM and reproduce this when I get a chance. It'll be great if someone can take a stab at it before that.

from obico-server.

raymondh2 avatar raymondh2 commented on June 30, 2024

Issue is caused by the ml_api/Dockerfile. Fedora is not a Debian based distro.

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

@RaymondHimle Fedora is the host whereas the command was running inside the container, which is a Ubuntu. Note the line:

Status: Downloaded newer image for docker.io/nvidia/cuda:9.0-devel-ubuntu16.04

from obico-server.

arhi avatar arhi commented on June 30, 2024

@RaymondHimle I don't have too much experience with docker but from what I gather host and docker image do not have to be same distro, isn't that one of the points of docker container?! you are running the same docker image on windows :D

@kennethjiang anything I can do to help? From what I see it fails on

RUN apt update

if I comment that one out it fails with

Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libsm6
E: Unable to locate package libxrender1
E: Unable to locate package libfontconfig1
E: Unable to locate package python3-pip
E: Unable to locate package vim
E: Unable to locate package ffmpeg
E: Unable to locate package wget
E: Unable to locate package curl

looks to me like image is either not seeing internet or something else is messed up there :( ... I'll try to figure it out and get back to you... maybe docker system on that server needs configuring before I try to setup this :)

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

@arhi I tested it on a Fedora29 VM and couldn't reproduce this problem.

Commenting out RUN apt update won't be the solution.

Iid a bit search and found this: moby/moby#20025 . One comment in that issue was to do update-ca-certificates -f. Although I'm not sure if it's the solution, you can try it out with:

docker run --rm --it c24bd4961e81 bash # Get you inside the container
update-ca-certificates -f #This needs to run inside the container, not on the host
apt update

Let me know if apt update still craps out.

Also you have a way to get me on your Fedora server I can poke around a bit more.

from obico-server.

arhi avatar arhi commented on June 30, 2024

it's def configuration issue on docker side ...

  • killed firewall
  • added dns option to docker (no clue why it does not use default one from system
[root@gedora sysconfig]# cat /etc/sysconfig/docker-network
# /etc/sysconfig/docker-network
DOCKER_NETWORK_OPTIONS="--dns 8.8.8.8 "

and had to add

RUN apt-get clean

to the ml_api/Dockerfile (as it was ignoring apt-get update 'cause it had cache from unsuccessful run)

and then it semiworked

[root@gedora TheSpaghettiDetective]# docker-compose up -d
Building ml_api
Step 1/13 : FROM nvidia/cuda:9.0-devel-ubuntu16.04
 ---> c24bd4961e81
Step 2/13 : RUN apt-get clean
 ---> Running in 86c7828053dd
 ---> 4698b7511875
Removing intermediate container 86c7828053dd
Step 3/13 : RUN apt-get update
 ---> Running in d5df8cbd9cde
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
  Operation timed out after 0 milliseconds with 0 out of 0 bytes received
Err:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
  Operation timed out after 0 milliseconds with 0 out of 0 bytes received
Ign:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Ign:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Ign:6 http://security.ubuntu.com/ubuntu xenial-security Release
Ign:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:10 http://archive.ubuntu.com/ubuntu xenial Release
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:13 http://archive.ubuntu.com/ubuntu xenial-updates Release
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:16 http://archive.ubuntu.com/ubuntu xenial-backports Release
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Ign:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Err:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
  Connection failed [IP: 91.189.91.26 80]
Ign:9 http://security.ubuntu.com/ubuntu xenial-security/main all Packages
Ign:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
Ign:12 http://security.ubuntu.com/ubuntu xenial-security/restricted all Packages
Ign:14 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
Ign:15 http://security.ubuntu.com/ubuntu xenial-security/universe all Packages
Ign:17 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
Ign:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse all Packages
Ign:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Ign:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Get:37 http://archive.ubuntu.com/ubuntu xenial-backports/restricted amd64 Packages [40 B]
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Get:41 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse amd64 Packages [40 B]
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Ign:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Ign:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Ign:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Ign:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Ign:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Ign:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Ign:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Ign:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Ign:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Ign:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Ign:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Ign:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Err:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
  Connection failed [IP: 91.189.88.161 80]
Ign:20 http://archive.ubuntu.com/ubuntu xenial/main all Packages
Ign:21 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages
Ign:22 http://archive.ubuntu.com/ubuntu xenial/restricted all Packages
Ign:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Ign:24 http://archive.ubuntu.com/ubuntu xenial/universe all Packages
Ign:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages
Ign:26 http://archive.ubuntu.com/ubuntu xenial/multiverse all Packages
Err:27 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
  Connection failed [IP: 91.189.88.161 80]
Ign:28 http://archive.ubuntu.com/ubuntu xenial-updates/main all Packages
Ign:29 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages
Ign:30 http://archive.ubuntu.com/ubuntu xenial-updates/restricted all Packages
Ign:31 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
Ign:32 http://archive.ubuntu.com/ubuntu xenial-updates/universe all Packages
Ign:33 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages
Ign:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse all Packages
Err:35 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
  Connection failed [IP: 91.189.88.161 80]
Ign:36 http://archive.ubuntu.com/ubuntu xenial-backports/main all Packages
Ign:38 http://archive.ubuntu.com/ubuntu xenial-backports/restricted all Packages
Ign:39 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages
Ign:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe all Packages
Ign:42 http://archive.ubuntu.com/ubuntu xenial-backports/multiverse all Packages
Fetched 80 B in 1h 44min 26s (0 B/s)
Reading package lists...
W: The repository 'http://security.ubuntu.com/ubuntu xenial-security Release' does not have a Release file.
W: The repository 'http://archive.ubuntu.com/ubuntu xenial Release' does not have a Release file.
W: The repository 'http://archive.ubuntu.com/ubuntu xenial-updates Release' does not have a Release file.
W: The repository 'http://archive.ubuntu.com/ubuntu xenial-backports Release' does not have a Release file.
W: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/InRelease  Operation timed out after 0 milliseconds with 0 out of 0 bytes received
W: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/InRelease  Operation timed out after 0 milliseconds with 0 out of 0 bytes received
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64/Packages  Connection failed [IP: 91.189.91.26 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/main/binary-amd64/Packages  Connection failed [IP: 91.189.88.161 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-amd64/Packages  Connection failed [IP: 91.189.88.161 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/main/binary-amd64/Packages  Connection failed [IP: 91.189.88.161 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
ERROR: Service 'ml_api' failed to build: The command '/bin/sh -c apt-get update' returned a non-zero code: 100

so compared to previous attempts fetched way more data but still failed ...

now I'm drawing blank :(

from obico-server.

arhi avatar arhi commented on June 30, 2024

looks like my config is ok now it's just that tuturubuntu is unable to fetch updated list of packages :( .. will try from some debian vm but..

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

I'll reset my Fedora29 VM and start over to compare notes with you.

from obico-server.

arhi avatar arhi commented on June 30, 2024

looks like it has issue connecting to that 91.189.88.161 server

docker run --rm --it c24bd4961e81 bash -> I get error that inknown --it

changed the Dockerfile to

FROM nvidia/cuda:9.0-devel-ubuntu16.04

RUN apt-get clean
RUN update-ca-certificates -f
RUN apt-get update
RUN apt install -y libsm6 libxrender1 libfontconfig1 python3-pip vim ffmpeg wget curl
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 10

ADD . /app
WORKDIR /app
RUN wget --quiet -O model/model.weights $(cat model/model.weights.url | tr -d '\r')
RUN pip install --upgrade pip
RUN pip install -r requirements.txt

EXPOSE 3333

so added clean and update-ca-certificate ... running it again .. weird how it takes this long to dl everything on 200mbps link :(

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Here is the console output of my run: https://pastebin.com/rZe8z0HU

The bunch of Ign:x ... in your log still doesn't look right to me.

If you want you can post your SSH pub key here (or email to me at [email protected] if you want to be extra cautious) and I'll give you ssh access to my fedora VM to compare it with your own.

from obico-server.

arhi avatar arhi commented on June 30, 2024

first error I don't think I can solve

[root@gedora TheSpaghettiDetective]# docker-compose up -d
Building ml_api
Step 1/13 : FROM nvidia/cuda:9.0-devel-ubuntu16.04
 ---> c24bd4961e81
Step 2/13 : RUN apt-get clean
 ---> Using cache
 ---> 4698b7511875
Step 3/13 : RUN update-ca-certificates -f
 ---> Running in 79725a615f5a
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
148 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
 ---> bd7a9ffb0202
Removing intermediate container 79725a615f5a
Step 4/13 : RUN apt-get update
 ---> Running in b24d7b0a0664
Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
  Operation timed out after 0 milliseconds with 0 out of 0 bytes received
Err:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
  Operation timed out after 0 milliseconds with 0 out of 0 bytes received

as there is no inRelease file neither on
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 nor on https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64

docker installed

[root@gedora ~]# rpm -qa | grep docker
docker-distribution-2.6.2-7.git48294d9.fc28.x86_64
python3-dockerpty-0.4.1-12.fc29.noarch
python3-docker-pycreds-0.3.0-3.fc29.noarch
cockpit-docker-189-1.fc29.x86_64
python3-docker-3.5.0-1.fc29.noarch
docker-common-1.13.1-65.git1185cfd.fc29.x86_64
docker-rhel-push-plugin-1.13.1-65.git1185cfd.fc29.x86_64
docker-1.13.1-65.git1185cfd.fc29.x86_64
docker-compose-1.22.0-2.fc29.noarch
[root@gedora ~]#

from obico-server.

arhi avatar arhi commented on June 30, 2024

@kennethjiang weirdf ... you have

Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]
Get:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release [564 B]
Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Packages [193 kB]
Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages [39.0 kB]

I'm failing these .. it's done from inside the docker so host should be irrelevant ...

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Yes I'm puzzled why we are seeing different errors here...

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

One explanation could be we are hitting different CDN servers... just a wild guess.

from obico-server.

arhi avatar arhi commented on June 30, 2024

most probably, I'm deleting all docker storage and images and will restart the process .. maybe I even do it trough vpn trough USA so I get different CDN's

from obico-server.

arhi avatar arhi commented on June 30, 2024

the whole docker thing was supposed to make this easier :D :D :D :D (btw tried now on bionic 18.04 and it complains that compose file is using "version" and that's not supported?!)

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

I'd say docker just made things easier :)

Can you do docker-compose --version to check the version you are running?

from obico-server.

arhi avatar arhi commented on June 30, 2024

[root@gedora ~]# docker-compose --version
docker-compose version 1.22.0, build f46880f

from obico-server.

arhi avatar arhi commented on June 30, 2024

[root@gedora ~]# docker --version
Docker version 1.13.1, build 1185cfd/1.13.1

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Did you install docker-compose by following https://docs.docker.com/compose/install/ ? Somehow mine is:

[kenneth@fedora TheSpaghettiDetective]$ docker-compose --version
docker-compose version 1.23.2, build 1110ad01

from obico-server.

arhi avatar arhi commented on June 30, 2024

nope, I did "dnf install docker-compose"
lemme check that url

from obico-server.

arhi avatar arhi commented on June 30, 2024

took the one from the git, now is 1.23.2 let's see where we go now :D running it again

[root@gedora ~]# docker-compose --version
docker-compose version 1.23.2, build 1110ad01

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Also your docker version doesn't look right either. On both of my Ubuntu and Fedora vms, docker version is Docker version 18.09.3, build 774a1f4

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

You may want to follow https://docs.docker.com/install/linux/docker-ce/fedora/ for Fedora and https://docs.docker.com/install/linux/docker-ce/ubuntu/ for Ubuntu. Docker is changing a lot so the version in any distro's official repo is likely to be too old.

from obico-server.

arhi avatar arhi commented on June 30, 2024

that might be the issue... thx updating

from obico-server.

arhi avatar arhi commented on June 30, 2024

holy ... the pulling from nvidia/cuda took 1% of time then with older version ?!?!?!?!? (and yes I removed the whole /var/lib/docker before installing new version)

but same problem :(

[root@gedora TheSpaghettiDetective]# docker-compose up -d
Creating network "thespaghettidetective_default" with the default driver
Building ml_api
Step 1/11 : FROM nvidia/cuda:9.0-devel-ubuntu16.04
9.0-devel-ubuntu16.04: Pulling from nvidia/cuda
7b722c1070cd: Pull complete
5fbf74db61f1: Pull complete
ed41cb72e5c9: Pull complete
7ea47a67709e: Pull complete
52efd3da8bcd: Pull complete
eea82f174227: Pull complete
0d7845ca9ae6: Pull complete
cfbd609f9a85: Pull complete
ed1cb7fbcbd9: Pull complete
Digest: sha256:64b79f57b0f0ddfdb21f6f0c45c900c1a9a3751e87c3dce681b361bbe4163fe1
Status: Downloaded newer image for nvidia/cuda:9.0-devel-ubuntu16.04
 ---> c24bd4961e81
Step 2/11 : RUN apt update
 ---> Running in 3c73c06d2e97

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
  Operation timed out after 0 milliseconds with 0 out of 0 bytes received

will setup some vpn to USA to test tomorrow 5am here now need some zzzzzzz

btw when I go to https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 from here there is no InRelease file

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Progress!

Neither can I find InRelease at https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64. So hopefully this error is harmless. But I'm still bothered by the fact we are seeing different errors messages here...

https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release is a valid link for me. And I can find the needed cuda_xx files there too. How about you?

Time to head off to bed for me too...

from obico-server.

arhi avatar arhi commented on June 30, 2024

had to try vpn pre-bed ... it "goes" now trough vpn .. weird since this is take some time I'm off and let it run but it picked up all the nvidia stuff where it was erroring without vpn... weird..

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Great. And thank you for going through this with me. This will definitely make it to the FAQ section of README.md

from obico-server.

arhi avatar arhi commented on June 30, 2024

and finished trough vpn ?!!?!?!?!?!?!?!?!

Step 8/8 : RUN python manage.py collectstatic --noinput -c
 ---> Using cache
 ---> 46e02c57229a

Successfully built 46e02c57229a
Successfully tagged thespaghettidetective_tasks:latest
WARNING: Image for service tasks was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating thespaghettidetective_ml_api_1 ... done
Creating thespaghettidetective_redis_1  ... done
Creating thespaghettidetective_tasks_1  ... done
Creating thespaghettidetective_web_1    ... done
[root@gedora TheSpaghettiDetective]#

kill me if I understand why it didn't want to go from here (Belgrade, Serbia), and I can open those url's from the same server using links, why it was failing to fetch them from docker - beats me, but opening openvpn to my usa server solved that final issue ...

I'm closing the bug now as it's not a bug, but yes, I do believe part of this process outta go into FAQ :D and the existance of this report might also help someone solve similar problem :)

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

WIN!!!

Yes I'm pretty sure other users will run into the same issue and hopefully this thread can save them some time. And I'm glad to have a technically competent user like you to help us sort this out @arhi !

Can you do me a favor by running some additional command to further determine the root cause of it and the best solution for user without the luxury of VPN'ing to US?

docker-compose exec ml_api bash  # <-- shell into container
curl -v 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release'  # <--- Run this in the container

Let me know what's the output of the curl command.

from obico-server.

arhi avatar arhi commented on June 30, 2024

@kennethjiang glad I can be of any assistance...

running this, no vpn, no docker, directly on host

[root@gedora tmp]#  curl -v 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release'
*   Trying 192.229.211.70...
* TCP_NODELAY set
* Connected to developer.download.nvidia.com (192.229.211.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Santa Clara; O=NVIDIA Corporation; OU=Tech Ops; CN=*.download.nvidia.com
*  start date: Aug 15 00:00:00 2016 GMT
*  expire date: Aug 20 12:00:00 2019 GMT
*  subjectAltName: host "developer.download.nvidia.com" matched cert's "*.download.nvidia.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x55666e9d5470)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /compute/cuda/repos/ubuntu1604/x86_64/Release HTTP/2
> Host: developer.download.nvidia.com
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=604800
< content-type: application/octet-stream
< date: Sat, 23 Mar 2019 15:05:13 GMT
< etag: "1713150023"
< expires: Sat, 30 Mar 2019 15:05:13 GMT
< last-modified: Wed, 27 Feb 2019 23:13:37 GMT
< server: ECAcc (bsa/EAEA)
< x-cache: HIT
< content-length: 564
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
Origin: NVIDIA
Label: NVIDIA CUDA
Architecture: x86_64
Date: Wed, 27 Feb 2019 23:13:36 +0000
MD5Sum:
 a6aca54f39a630f5a74e10db81ec9fdf          1075813 Packages
 171b9aa266eadbc2c8f42aacd8e55f58           192796 Packages.gz
SHA1:
 ab3eacbf0507d9af6c2e8f5ec095a7943c77f52b          1075813 Packages
 be802da709743ca5ae4605d6367fa3845a81c455           192796 Packages.gz
SHA256:
 99c7126df4a429faa72312ae8b92c94dcfc3b9f065770a915bca1306ecc0796a          1075813 Packages
 f7f4a3df6870aa137dfe47ba79782feaa2649fa91781c07e5c7d7c72532de5a0           192796 Packages.gz
* Connection #0 to host developer.download.nvidia.com left intact
[root@gedora tmp]#

now running the same thing in docker it's "stuck" for a while now

[root@gedora TheSpaghettiDetective]# docker-compose exec ml_api bash
root@ml_api:/app# curl -v 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release'
*   Trying 192.229.211.70...
* Connected to developer.download.nvidia.com (192.229.211.70) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1



from obico-server.

arhi avatar arhi commented on June 30, 2024

looks like nothing https works "normally" if vpn is not up... this looks like host configuration mess

root@ml_api:/app# curl -v 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release'
*   Trying 192.229.211.70...
* Connected to developer.download.nvidia.com (192.229.211.70) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
^C
root@ml_api:/app# curl -v 'https://www.thespaghettidetective.com/img/gauge.png'
*   Trying 40.84.232.193...
* Connected to www.thespaghettidetective.com (40.84.232.193) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1

^C
root@ml_api:/app# curl -v 'http://www.thespaghettidetective.com/img/gauge.png'
*   Trying 40.84.232.193...
* Connected to www.thespaghettidetective.com (40.84.232.193) port 80 (#0)
> GET /img/gauge.png HTTP/1.1
> Host: www.thespaghettidetective.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.10.3 (Ubuntu)
< Date: Sat, 23 Mar 2019 15:41:54 GMT
< Content-Type: text/html
< Content-Length: 194
< Connection: keep-alive
< Location: https://www.thespaghettidetective.com/img/gauge.png
<
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>
* Connection #0 to host www.thespaghettidetective.com left intact
root@ml_api:/app# curl -v 'https://www.thespaghettidetective.com/img/gauge.png'
*   Trying 40.84.232.193...
* Connected to www.thespaghettidetective.com (40.84.232.193) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1

from obico-server.

arhi avatar arhi commented on June 30, 2024
root@ml_api:/app# curl -v 'https://www.thespaghettidetective.com/img/gauge.png'
*   Trying 40.84.232.193...
* Connected to www.thespaghettidetective.com (40.84.232.193) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Error in the pull function.
* Closing connection 0
curl: (35) gnutls_handshake() failed: Error in the pull function.
root@ml_api:/app#

from obico-server.

arhi avatar arhi commented on June 30, 2024

found this: https://serverfault.com/questions/785768/https-request-fails-in-docker-1-10-with-virtualized-network

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

hmmm this is super bizarre. Not a networking expert myself but I have a feeling this is a routing and/or firewall issue triggered by how Docker does its networking... #headscratching

from obico-server.

arhi avatar arhi commented on June 30, 2024

I found the problem and it has nothing to do with TSD but it's a weird behavior of docker system network ... looks like HTTPS requests fail if your docker MTU is lower then your system MTU, I had my eth0 (enp4s0 really but irrelevant) have MTU of 1420 due to back in the day my ADSL provider used 1:1 network mapping and a tunnel so that was the highest MTU that would normally work trough ADSL. I moved to a different provider long time ago but MTU on that server stayed at 1420 (I actually forgot about it). Normally everything works with that setup and normally apps adapt but for some reason docker configures network under it at 1500. Even when I run dockerd with --mtu=1400 that changes nothing, in the docker ifconfig shows (different from what internet say) 1500 and not 1400. Maybe a new image created after this config is created will do that but don't know, makes little sense. Also changing MTU inside docker does not work for me, I get "SIOCSIFMTU: Operation not permitted" error if I try to change it from inside docker. Even with docker0 interface at MTU 1400 the docker container is still 1500 ?!?! On fedora I see a br-b9a846ed861f interface (bridge between docker0 and bunch of veth* interfaces), setting that one to mtu 1400 - no changes .. changing all veth* intefaces to 1400 .. no changes :D :D :D this is getting interesting ..

[root@gedora TheSpaghettiDetective]# ifconfig
br-b9a846ed861f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet 172.19.0.1  netmask 255.255.0.0  broadcast 172.19.255.255
        inet6 fe80::42:1cff:fee5:8de4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:1c:e5:8d:e4  txqueuelen 0  (Ethernet)
        RX packets 28  bytes 2463 (2.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55  bytes 50612 (49.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1400
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:31:15:25:50  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1420
        inet 192.168.89.1  netmask 255.255.255.0  broadcast 192.168.89.255
        inet6 fd88:71be:2f7a:2959::1  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::21e:8cff:fe4b:17e  prefixlen 64  scopeid 0x20<link>
        ether 00:1e:8c:4b:01:7e  txqueuelen 1000  (Ethernet)
        RX packets 74034  bytes 9853891 (9.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4908  bytes 551081 (538.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 267  bytes 25328 (24.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267  bytes 25328 (24.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth8344569: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::e046:88ff:fe11:45a0  prefixlen 64  scopeid 0x20<link>
        ether e2:46:88:11:45:a0  txqueuelen 0  (Ethernet)
        RX packets 28  bytes 2463 (2.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55  bytes 50612 (49.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethc9b0aba: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400
        inet6 fe80::ccef:73ff:febd:4604  prefixlen 64  scopeid 0x20<link>
        ether ce:ef:73:bd:46:04  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29  bytes 5137 (5.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@gedora TheSpaghettiDetective]# systemctl restart docker
[root@gedora TheSpaghettiDetective]# docker-compose exec ml_api ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:ac:13:00:03
          inet addr:172.19.0.3  Bcast:172.19.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4067 (4.0 KB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[root@gedora TheSpaghettiDetective]# ps ax | grep dockerd
 8967 ?        Ssl    0:00 /usr/bin/dockerd -H fd:// --mtu=1450 --containerd=/run/containerd/containerd.sock

so no matter what I do indide docker it's 1500 :D :D :D and dockerd is started with --mtu=1450 so by all the documentation I read it should be 1450 or 1400 ... no way it could be 1500 :D

here: http://atodorov.org/blog/2017/12/08/how-to-configure-mtu-for-the-docker-network/
it explains that docker0 interface must have lower MTU then eth0 (it is, but it does not show inside docker)

anyhow this really is docker network configuration related, not TSD related and I think there's more then enough info in the report for anyone suffering same problem to solve it :)

from obico-server.

kennethjiang avatar kennethjiang commented on June 30, 2024

Now this is WAY over my head! @arhi Again I'm grateful to have you helping us sorting this tricky problem out so that other users will save a lot of time in the future!

from obico-server.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.