docker-library/meta-scripts

Scripts to generate metadata (source IDs, build IDs)

meta-scripts's Issues

scaling issues in deploy

With #5 and #16, our builds.json and sources.json generators are now really fast (and respectful of our upstream registry rate limits), but we're still very slow at deploying images (to the point that deploying often fails before it gets through the full set 😭). The deploy job is currently effectively "generate a bunch of crane cp commands and run them", which isn't terribly efficient:

meta-scripts/jenkins.jq

Lines 13 to 17 in 80bb556

	| if length == 1 then
	@sh "crane copy \(.) \($target)"
	else
	@sh "crane index append --tag \($target) " + (map("--manifest " + @sh) | join(" ")) + " --flatten"
	end

I'm working on a new version of this that is more native to our process so that things like auth tokens can be shared between API invocations more correctly/effectively, but wanted to file an issue to track the problem (especially because it's now the main blocker to our scaling up much more).

Launchpad does not allow fetching arbitrary commits, which now breaks our tests here

+ .test/test.sh
failed fetching git repo for "ubuntu" (tags "22.04, jammy-20240111, jammy, latest" on arch "amd64")
failed fetching "e6e3490ad3f[5](https://github.com/docker-library/meta-scripts/actions/runs/7936297386/job/21671258830?pr=24#step:5:6)24ccaa072edafe525f8ca8ac5490:refs/remotes/temp2901481880/temp": server does not support exact SHA1 refspec
failed fetching "refs/tags/dist-jammy-amd[6](https://github.com/docker-library/meta-scripts/actions/runs/7936297386/job/21671258830?pr=24#step:5:7)4-20240111-e6e3490a:refs/remotes/temp29014[8](https://github.com/docker-library/meta-scripts/actions/runs/7936297386/job/21671258830?pr=24#step:5:9)1880/temp": couldn't find remote ref "refs/tags/dist-jammy-amd64-20240111-e6e3490a"

We can fix this in a few different ways. The easiest/quickest temporary fix would be to mirror those old commits to a GitHub repository somewhere and update our test library file to reference it from GitHub instead (I still have these commits locally, so that would be pretty easy -- choosing/naming a repo is much harder; we used to use https://github.com/docker-library/commit-warehouse for this, before GitHub added support for fetching arbitrary commits and before it grew too large to even use anymore 😇).

A better long-term fix would probably be to finally convert busybox over to use oci-import (now that we have #20) which would also allow us to test #20 properly. 😄