Xenial docker image creates/observes bogus hash sum mismatches.

[tianon]

Xenial docker image creates/observes bogus hash sum mismatches.

[Alan-R]

Attn @tianon, @benbc: It's broken again...
Step 20 : RUN apt-get update && apt-get -y install --no-install-recommends neo4j=3.0.0
---> Running in 84d76aa818b3
Ign:1 http://debian.neo4j.org/repo stable/ InRelease
Get:2 http://debian.neo4j.org/repo stable/ Release [1479 B]
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:4 http://debian.neo4j.org/repo stable/ Release.gpg [819 B]
Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Get:6 http://debian.neo4j.org/repo stable/ Packages [28.6 kB]
Err:6 http://debian.neo4j.org/repo stable/ Packages
Hash Sum mismatch
Hit:7 http://archive.ubuntu.com/ubuntu xenial-security InRelease
Fetched 30.9 kB in 0s (50.9 kB/s)
Reading package lists...
W: http://debian.neo4j.org/repo/stable/Release.gpg: Signature by key 66D34E951A8C53D90242132B26C95CF201182252 uses weak digest algorithm (SHA1)
E: Failed to fetch http://debian.neo4j.org/repo/stable/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update && apt-get -y install --no-install-recommends neo4j=3.0.0' returned a non-zero code: 100

[Alan-R]

Here's how to reproduce this problem:

It turns out that the problem only seems to exist for Xenial Xerus Docker image only - not in Xenial outside of Docker. Below is how to reproduce the problem: Just to make sure I had the good version of Xenial, I deleted my locally cached image and pulled a fresh copy. It made no difference. The relevant Dockerfile is below:

FROM ubuntu:xenial
RUN apt-get update && apt-get install -y wget
RUN wget -O - http://debian.neo4j.org/neotechnology.gpg.key | apt-key add - 
RUN echo "deb http://debian.neo4j.org/repo stable/" > /etc/apt/sources.list.d/neo4j.list
RUN apt-get update && apt-get -y install --no-install-recommends neo4j=3.0.0

Here's the resulting error message:

W: http://debian.neo4j.org/repo/stable/Release.gpg: Signature by key 66D34E951A8C53D90242132B26C95CF201182252 uses weak digest algorithm (SHA1)
E: Failed to fetch http://debian.neo4j.org/repo/stable/Packages.gz  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

This is something wrong with the Docker image - since the problem only occurs under Docker - not on "normal" OS installs. Maybe a missing dynamic library??

[benbc]

I'm one of the Neo4j Docker image maintainers. I'm very interested to understand what's going on here and happy to collaborate on any changes that are needed. @tianon do you have any ideas?

[Alan-R]

@benbc I'm tied up for the next 10 days or so - but here's a thought about how to diagnose the problem if you have time. Do an strace of the apt-get install both inside the container and on a VM or bare metal machine, and look for failed calls that occur inside the container that don't occur in the VM. You're probably looking for a failed stat or open. If you filtered for that particular circumstance, then that's likely to be a file that's missing. Not all missing files cause programs to fail outright. My guess is that there's a missing library that's dynamically loaded. This is a pretty effective technique - but it can be tedious :-(.

[tianon]

Here's what I've done so far to try and narrow this down:

1. reproduced the fail in Docker
2. reproduced the success in a simple chroot (debootstrap --variant minbase xenial)
3. compared the output of dpkg -l in both -- completely identical

Next thing I'm trying is exploding the Docker rootfs to compare to the chroot bit by bit and see what falls out.

[tianon]

Ooh, it might be that we have this in our apt.conf for Docker:

Acquire::GzipIndexes "true";
Acquire::CompressionTypes::Order:: "gz";

This forces gz to be preferred, where normally bz2 is (we do this because until very, very recent APT versions, Acquire::GzipIndexes required .gz in order to store the list files compressed on-disk). How is that repo managed? apt-ftparchive, reprepro, etc?

[benbc]

@tianon As of today, I can't reproduce this problem. Maybe the latest ubuntu:16.04 builds are slightly different now.

@Alan-R Do you have a minute to check this yourself?

[Alan-R]

It looks fixed to me too. Neo4j gets installed, my tests run. Two fail - but not for things related to that. @tianon: Did you make a change to that apt.conf file?

[tsde]

Hi,

I'm experiencing the same issue with a "real" installation of Ubuntu 16.04 (desktop) - which is actually an upgrade from 15.10. I don't remember having this issue last week. Now I have this:

Get:9 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages [4 815 B]
Err:9 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
  Hash Sum mismatch
E: Failed to fetch https://apt.dockerproject.org/repo/dists/ubuntu-xenial/main/binary-amd64/Packages  Hash Sum mismatch

I tried to clean using apt-get clean and rm -rf /var/lib/apt/lists/* with no luck. I also tried to force apt to use gz compression, again with no luck.

Any hint on this one ?

Cheers

[yosifkit]

@tsde, that looks like moby/moby#23203, which is fixed 🎉

@Alan-R and @benbc looks like the original issue is fixed now?

[Alan-R]

The original issue we were having seems to have been fixed for over a week. I'm guessing that @tsde's problem was separate. Since I'm currently in the process of putting out a release, I'll notice if it's broken again, and let you know. It seems unlikely.

Speaking of which - does anyone know what went wrong here, and what was done to fix it?

"Problems that go away by themselves, come back by themselves" - Sarah Kiefhaber

[tsde]

My bad for polluting this thread :/ It's indeed related to the issue @yosifkit mentionned. It's now fixed. And thx @yosifkit for pointing it out

[Alan-R]

FWIW: I got the same result with 3.0.1.

I removed my local copy of xenial before doing the first one above.

[tianon]

This is an issue with the hashes of the http://debian.neo4j.org/repo repository, not this image (and is likely fixed by now, given the age of the previous comments 😅).

The default configuration of this image helps to show the problem more readily by preferring the Packages.gz artifact, but it is not a "bug" with this image if a repository has an invalid hash for said file in their Release file.

Please file any further issues with the maintainers of the repository which exhibits the issue. Thanks! 👍