v2.0.x containerd docker remote bug: sometimes gets "cannot reuse body, request must be retried"

[estesp]

v2.0.x containerd docker remote bug: sometimes gets "cannot reuse body, request must be retried"

[estesp]

Hmm, strange. I tested v2.0.0-beta.0 with GCR, but my normal test image uses repos not tags for differentiating the platforms. I'll have to test with tags in case that changes behavior in some way.

Just tried again to make sure:

$ ./manifest-tool --version
./manifest-tool version 2.0.0-beta.0 (commit: ff5aefe0f43b89efc21373c5514ef98a62c63e8a)
[long push from-args command]
.... response.status="201 Created" size=966 url="https://gcr.io/v2/[REDACTED]/alpine/manifests/latest"
Digest: sha256:493661ff8ba145bbe641dc56c6da3071b7761dbb22cd42fdf15b31b6f170eaa8 966

[estesp]

Any chance you've been able to try this again? Especially now that maybe you are using v2 :) I have never been able to reproduce an issue pushing to GCR.

[thockin]

I have not had need to push a manifest lately, but I will try to fake one up and let you know.

…

On Wed, Sep 15, 2021 at 9:41 AM Phil Estes ***@***.***> wrote: Any chance you've been able to try this again? Especially now that maybe you are using v2 :) I have never been able to reproduce an issue pushing to GCR. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDZMRBWKK5EVC32K63UCDEB5ANCNFSM47HAVRQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[estesp]

Just released v2.0.0 and it seems to be working fine with GCR in my tests. Going to close this for now but feel free to re-open if you are able to reproduce.

[thockin]

I'm baaaaaack

platforms=$(echo linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x | sed 's/ /,/g');  \
manifest-tool                                         \
    --username=oauth2accesstoken                      \
    --password=$(gcloud auth print-access-token)     \
    push from-args                                    \
    --platforms "$platforms"                         \
    --template gcr.io/k8s-staging-git-sync/git-sync:v3.5.0__OS_ARCH \
    --target gcr.io/k8s-staging-git-sync/git-sync:v3.5.0
FATA[0004] Error pushing manifest list/index to registry: sha256:98821a216b62b6ee5dccdc2682c87f85d64e905d61872d0ab19073f9c4463623: failed commit on ref "manifest-sha256:6850b0792b249e2562a2b622e1938cb35747174030aef27d5274319811d9f150": cannot reuse body, request must be retried 

I have no idea where to even start

[thockin]

FWIW manifest-tool 1.0.3 still works:

platforms=$(echo linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x | sed 's/ /,/g');  \
manifest-tool                                         \
    --username=oauth2accesstoken                      \
    --password=$(gcloud auth print-access-token)     \
    push from-args                                    \
    --platforms "$platforms"                         \
    --template gcr.io/k8s-staging-git-sync/git-sync:v3.5.0__OS_ARCH \
    --target gcr.io/k8s-staging-git-sync/git-sync:v3.5.0
Digest: sha256:d16f5b2bca94cdbb4e40b256bfe639450a6f0577dbd8b3fcaf126a2261822fcd 1665

[estesp]

You may not be alone; looks like this might be a codepath my testing just isn't hitting (and we have a recent report in containerd proper as well); also #156 (comment)

In the #156 issue, I see a 404 response that my tries with Quay aren't getting, and that potentially is trying to reuse the body on the response asking for auth? Need to dig deeper now that it doesn't seem to be a random one-off.

[thockin]

Let me know if I can help with debug traces or something...

…

On Fri, Mar 11, 2022 at 12:27 PM Phil Estes ***@***.***> wrote: Reopened #122 <#122>. — Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDORI3NFX5PSWVIMKLU7OUEPANCNFSM47HAVRQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <estesp/manifest-tool/issue/122/issue_event/6227467648 <(622)%20746-7648>@ github.com>

[estesp]

Thanks; just found a reproduce scenario and found that, aside from the fact containerd is working on fixing the "body reuse" problem, there is a different flow of registry interaction when the source repos are != target repo (e.g. use "names" not "tags" to differentiate OS/arch).

When all source images are in the same repo namespace as the target manifest list, that flow ends in a 404 existence check on the final target name:tag which appears to cause a 401 Unauthorized, which leads to the retry with same body error in the auth flow. Digging into that now; I think it's related to registry token scopes (which are set at the library/auth level, not in my code, but need to find out why it's causing different behavior)

[estesp]

Not sure if you have the chance to test the changes in #159, but I'm 99% convinced this will workaround the issue until containerd has a solution to the req/resp retry with a 401. I've verified it fixes the same issue hit on Quay. Interestingly it requires that you have a public repo for all the sources; if your repo is private or any of the sources come from a private repo, then you don't hit the bug as the 401 auth challenge comes during a "fetch" which means you don't get the body reuse error.

[thockin]

Works for me, commented in #159

…

On Tue, Mar 15, 2022 at 12:14 PM Tim Hockin ***@***.***> wrote: will try On Tue, Mar 15, 2022 at 9:49 AM Phil Estes ***@***.***> wrote: > Not sure if you have the chance to test the changes in #159 > <#159>, but I'm 99% > convinced this will workaround the issue until containerd has a solution to > the req/resp retry with a 401. I've verified it fixes the same issue hit on > Quay. Interestingly it requires that you have a public repo for all the > sources; if your repo is private or any of the sources come from a private > repo, then you don't hit the bug as the 401 auth challenge comes during a > "fetch" which means you don't get the body reuse error. > > — > Reply to this email directly, view it on GitHub > <#122 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABKWAVD42SWH4N57ICEMFLTVAC5PRANCNFSM47HAVRQA> > . > Triage notifications on the go with GitHub Mobile for iOS > <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> > or Android > <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. > > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[thockin]

will try

…

On Tue, Mar 15, 2022 at 9:49 AM Phil Estes ***@***.***> wrote: Not sure if you have the chance to test the changes in #159 <#159>, but I'm 99% convinced this will workaround the issue until containerd has a solution to the req/resp retry with a 401. I've verified it fixes the same issue hit on Quay. Interestingly it requires that you have a public repo for all the sources; if your repo is private or any of the sources come from a private repo, then you don't hit the bug as the 401 auth challenge comes during a "fetch" which means you don't get the body reuse error. — Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVD42SWH4N57ICEMFLTVAC5PRANCNFSM47HAVRQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>