Missing Docker image for 1.11.3

[chrisohaver]

Missing Docker image for 1.11.3

[chrisohaver]

Thanks @yongtang !

IMO we should still pursue moving to a bot account for GitHub image publishing.

[chrisohaver]

For some reason, this is also not being tagged as the latest release when you go to the releases page.

This because 1.11.3 is not fully released. The docker images are not published, so the release is not available. The latest release actually available is still 1.11.1.

[heiko-braun]

Thanks @chrisohaver, I only tested the login part of the release actions, but this seems to work:
#6723

[chrisohaver]

If a maintainer with admin permissions in the coredns dockerhub org could generate a token and then update the coredns/coredns secrets: DOCKERHUB_USERNAME (to yours) and DOCKERHUB_PASSWORD (to be your token instead of password), that would be a great help!

(I am not a coredns dockerhub org admin)

[grumps]

update:

I gave up on using goreleaser for this effort. It was way to hard to shoe horn in just docker builds.

I've been able to setup the workflow to push to ghcr.io and docker. Its sub-optimal at the moment because it builds the images twice.

To complete my changes:

• I'll likely need to do something to tag the builds properly.
• The removing arch tags, doesn't work at the moment. (I'm wondering if pushing arch tags is even needed. I'll investigate further)

To completely resolve the release

In the end someone with higher privileges on github and docker will be needed for the following:

• someone with docker privileges needs to create a token
• since coredns doesn't use github packages today there's likely some configuration needed (I'll document the best as possible)

Future looking statement

It took me a bit to follow the flow of the release for this project. I do think using goreleaser would greatly simplify this release process. I'd be happy to take that on, potentially. I'll have to look into the project changes process, eg. RFCs or whatever to get a signoff and gather requirements for releases. If there's any maintainers on this list, it would be good to know if that's something project is open to.

[chrisohaver]

@johnbelamaric has shared the list of dockerhub coredns project admins...

@johnbelamaric
@yongtang
@miekg
@prologic

I'm told that the number of admins allowed per project is curiously limited to 4, so we cannot add new admins.

Some concern was expressed over sharing a password or user account access token because it enables access to all of a user's projects, not just the coredns project. It was suggested that we create a shared "coredns" account for the purpose of publishing images, but due to the 4 seat limit this would require one person to relinquish their admin membership.

To the current dockerhub coredns project admins: @johnbelamaric, @yongtang , @miekg , @prologic, would any of you be comfortable creating an access token to be stored as a project secret in GitHub to be used when publishing images during releases. If not, would any of you be willing to relinquish your dockerhub coredns project admin membership to make space for a shared coredns account?

[johnbelamaric]

My screenshot is from dockerhub.

Oops, ok, thanks.

But I think I'd still rather do it at the org level. @chrisohaver or @yongtang do you have access to the coredns.io email management? We need to:

• Create a group like, "[email protected]" or "[email protected]". If we don't have access to create those, then maybe we can this on the CNCF list server instead.
• Add at least the steering committee members to that group, and maybe the security mailing list. This way, those people will receive receive notifications about that account.
• Create a docker hub account with that email address, and let me know the user name of that account.
• I can then remove all the admins except myself, and add that new hub account as an admin.
• Once we have that account as an admin, we can login to Docker Hub as that user and generate the necessary token.
• If we know the CNCF docker admin user name, we can then also remove me as an admin and add the CNCF docker account.

[yongtang]

I don't have coredns.io access as well. Though I think I can try reset the dockerhub username/password to see if it works.

[jauderho]

For some reason, this is also not being tagged as the latest release when you go to the releases page.

[dafanasiev]

NOTE: right now there is no way to detect latest release using github api releases-list endpoint (https://api.github.com/repos/coredns/coredns/releases) because both draft and prerelease are false:

# curl https://api.github.com/repos/coredns/coredns/releases | jq '.[] | select(.name=="v1.11.3")'

{
  "url": "https://api.github.com/repos/coredns/coredns/releases/153656321",
  "assets_url": "https://api.github.com/repos/coredns/coredns/releases/153656321/assets",
  "upload_url": "https://uploads.github.com/repos/coredns/coredns/releases/153656321/assets{?name,label}",
  "html_url": "https://github.com/coredns/coredns/releases/tag/v1.11.3",
  "id": 153656321,
  "author": {....},
  "node_id": "RE_kwDOAzt_0s4JKJwB",
  "tag_name": "v1.11.3",
  "target_commitish": "a7ed346585e30b99317d36e4d007b7b19a228ea5",
  "name": "v1.11.3",
  "draft": false,
  "prerelease": false,
  "created_at": "2024-04-26T19:08:47Z",
  "published_at": "2024-05-01T12:01:11Z",
  "assets": [....],
  "tarball_url": "https://api.github.com/repos/coredns/coredns/tarball/v1.11.3",
  "zipball_url": "https://api.github.com/repos/coredns/coredns/zipball/v1.11.3",
  "body": "This release contains some new features, bug fixes, and package updates. Because of the deployment issues with the previous release, all changed features from 1.11.2 have been included in this release.\r\nNew features include:\r\n* When the _forward_ plugin receives a malformed upstream response that overflows,\r\n  it will now send an empty response to the client with the truncated (TC) bit set to prompt the client\r\n  to retry over TCP.\r\n* The _rewrite_ plugin can now rewrite response codes.\r\n* The _dnstap_ plugin now supports adding metadata to the dnstap `extra` field.\r\n\r\n## Brought to You By\r\n\r\nAmila Senadheera,\r\nBen Kochie,\r\nBenjamin,\r\nChris O'Haver,\r\nGrant Spence,\r\nJohn Belamaric,\r\nKeita Kitamura,\r\nMarius Kimmina,\r\nMichael Grosser,\r\nOndřej Benkovský,\r\nP. Radha Krishna,\r\nRahil Bhimjiani,\r\nSri Harsha,\r\nTom Thorogood,\r\nWillow (GHOST),\r\nYong Tang,\r\nYuheng,\r\nZhizhen He,\r\nguangwu,\r\njourney-c,\r\npschou\r\nTed Ford\r\n\r\n## Noteworthy Changes\r\n\r\n* plugin/tls: respect the path specified by root plugin (https://github.com/coredns/coredns/pull/6138)\r\n* plugin/auto: warn when auto is unable to read elements of the directory tree (https://github.com/coredns/coredns/pull/6333)\r\n* plugin/etcd: the etcd client adds the DialKeepAliveTime parameter (https://github.com/coredns/coredns/pull/6351)\r\n* plugin/cache: key cache on Checking Disabled (CD) bit (https://github.com/coredns/coredns/pull/6354)\r\n* plugin/forward: Use the correct root domain name in the forward plugin's health checks (https://github.com/coredns/coredns/pull/6395)\r\n* plugin/forward: Handle UDP responses that overflow with TC bit (https://github.com/coredns/coredns/pull/6277)\r\n* plugin/rewrite: fix multi request concurrency issue in cname rewrite (https://github.com/coredns/coredns/pull/6407)\r\n* plugin/rewrite: add rcode as a rewrite option (https://github.com/coredns/coredns/pull/6204)\r\n* plugin/dnstap: add support for \"extra\" field in payload (https://github.com/coredns/coredns/pull/6226)\r\n* plugin/cache: fix keepttl parsing (https://github.com/coredns/coredns/pull/6250)\r\n* Return RcodeServerFailure when DNS64 has no next plugin (https://github.com/coredns/coredns/pull/6590)\r\n* Change the log flags to be a variable that can be set (https://github.com/coredns/coredns/pull/6546)\r\n* Bump go version to 1.21 (https://github.com/coredns/coredns/pull/6533)\r\n* replace the mutex locks in logging with atomic bool for the \"on\" flag (https://github.com/coredns/coredns/pull/6525)\r\n* Enable Prometheus native histograms (https://github.com/coredns/coredns/pull/6524)\r\n",
  "reactions": {....}
}

# curl https://api.github.com/repos/coredns/coredns/releases | jq '.[] | select(.name=="v1.11.1")'
{
  "url": "https://api.github.com/repos/coredns/coredns/releases/117360174",
  "assets_url": "https://api.github.com/repos/coredns/coredns/releases/117360174/assets",
  "upload_url": "https://uploads.github.com/repos/coredns/coredns/releases/117360174/assets{?name,label}",
  "html_url": "https://github.com/coredns/coredns/releases/tag/v1.11.1",
  "id": 117360174,
  "author": {.....},
  "node_id": "RE_kwDOAzt_0s4G_sYu",
  "tag_name": "v1.11.1",
  "target_commitish": "ae2bbc29be1aaae0b3ded5d188968a6c97bb3144",
  "name": "v1.11.1",
  "draft": false,
  "prerelease": false,
  "created_at": "2023-08-15T19:30:32Z",
  "published_at": "2023-08-15T20:00:30Z",
  "assets": [.....],
  "tarball_url": "https://api.github.com/repos/coredns/coredns/tarball/v1.11.1",
  "zipball_url": "https://api.github.com/repos/coredns/coredns/zipball/v1.11.1",
  "body": "This release fixes a major performance regression introduced in 1.11.0 that affected DoT (TLS) forwarded connections.\r\nIt also adds a new option to _dnstap_ to add metadata to the dnstap extra field, and fixes a config parsing bug in _cache_.\r\n\r\n## Brought to You By\r\n\r\nChris O'Haver,\r\nP. Radha Krishna,\r\nYong Tang,\r\nYuheng,\r\nZhizhen He\r\n\r\n## Noteworthy Changes\r\n\r\n* Revert \"plugin/forward: Continue waiting after receiving malformed responses (https://github.com/coredns/coredns/pull/6014)\" (#6270)\r\n* plugin/dnstap: add support for \"extra\" field in payload (https://github.com/coredns/coredns/pull/6226)\r\n* plugin/cache: fix keepttl parsing (https://github.com/coredns/coredns/pull/6250)\r\n\r\n",
  "reactions": {......}
}

latest release can be fetched using this api : https://api.github.com/repos/coredns/coredns/releases/latest

Maybe coredns team should mark some releases as prerelease or draft....

[chrisohaver]

Done. 1.11.3 now marked pre-release.

[tedaford]

Who has permissions for the Dockerhub credentials? This would be simple to resolve in theory if someone accessible has access to those.

[chrisohaver]

When I get some time, I'll enable actions in my fork and do a release to my personal docker account.
This should identify if the issue is with the build scripts or a bad credential.

[heiko-braun]

Hey @chrisohaver, did you find some time to look into this? Anything I can help with?

[chrisohaver]

@heiko-braun, I had some time a last week to try to work out migrating to docker build/push actions to enable publishing images to both docker and github's container repo. I didn't get very far. I was unable to get the multi-arch build to work completely.

Due to shifting work priorities and personal events, I will not have time to look into this further for the next couple of months.

One way someone can help would be to test/debug a docker release in a coredns fork to a personal docker account to see if it works, or if there is some issue with the existing scripts.

Thanks!

[heiko-braun]

Ah, I thought you have contributor access :)

I bet there is someone who has access to the dockerhub account that's being used?

[heiko-braun]

@chrisohaver do you think we can ask the steering committee for help around this? Is there way to increase visibility?

[chrisohaver]

@chrisohaver do you think we can ask the steering committee for help around this? Is there way to increase visibility?

Steering committee is aware. Thanks!

[mrueg]

@heiko-braun, I had some time a last week to try to work out migrating to docker build/push actions to enable publishing images to both docker and github's container repo. I didn't get very far. I was unable to get the multi-arch build to work completely.

Would you be open to switch the release process to https://goreleaser.com/ and or https://github.com/ko-build/ko ? I believe that could make it more standardized and easier to release.

[chrisohaver]

I would be open to someone taking that up.

[grumps]

@heiko-braun

Would you be open to switch the release process to https://goreleaser.com/ and or https://github.com/ko-build/ko ? I believe that could make it more standardized and easier to release.
@chrisohaver
I would be open to someone taking that up.

Are you referring to just refactoring:

• the docker builds, to use ko or goreleaser
• the entire release process (binaries, notes, "github release", etc)

Either way I'd be interested in taking this up. Refactoring the entire release process seems like a pretty large undertaking, based on my quick review of the current process.

Replacing the docker workflow to use go-releaser or ko seems more straight forward. I've not used ko before but the current dockerfile does do setcap not sure how to achieve that with ko.

[heiko-braun]

@grumps please go ahead. I was merely interested to get the images out. Switching the entire build process to goreleaser is too much work for me atm.

[jauderho]

FWIW, I do have multi-arch images working and v1.11.3 will be marked as latest whenever the release actually happens:

• https://github.com/jauderho/dockerfiles/pkgs/container/coredns
• https://hub.docker.com/r/jauderho/coredns/tags

[grumps]

Update:
Got a POC working with just amd64 docker image

Issues with POC (all resolvable) :

1. goreleaser builds binaries, and I will have to setup prebuilt imports, thankfully this is a feature
2. disable github release creation

Next:

1. resolve POC issues
2. setup mulitarch docker builds
3. cleanup things in Makefile.docker that are no longer needed
4. Open PR

I will continue working on this tonight.

[grumps]

@chrisohaver @heiko-braun unfortunately it looks like using prebuilt bins for go-releaser is a pro only feature. It seems like it would be pretty hard to use goreleaser without building the binaries without it, which would be a re-factor of almost the entire release process.

Seems like getting the current one to work with githubs and dockers container registries are the shortest possible path at the moment. However unclear to me what the current issue is that's blocking the image.

[grumps]

I've opened a PR resolve code issues, I can't resolve the package configuration on Github nor can I resolve the docker token issue. Configuring github to allow the action to push maybe more challenging because the docs seem to not tell you how to setup a package for the first time ever. @chrisohaver @heiko-braun #6783

[chrisohaver]

I've opened a PR resolve code issues, I can't resolve the package configuration on Github nor can I resolve the docker token issue. Configuring github to allow the action to push maybe more challenging because the docs seem to not tell you how to setup a package for the first time ever. @chrisohaver @heiko-braun #6783

Let’s get token issue resolved first. Then we can come back to your PR and evaluate those changes. Thanks!

[grumps]

Do we know if someone is working on that?

…

On Tue, Jul 9, 2024, at 03:45, chrisohaver wrote: > I've opened a PR resolve code issues, I can't resolve the package configuration on Github nor can I resolve the docker token issue. Configuring github to allow the action to push maybe more challenging because the docs seem to not tell you how to setup a package for the first time ever. @chrisohaver @heiko-braun #6783 Let’s get token issue resolved first. Then we can come back to your PR and evaluate those changes. Thanks! -- Reply to this email directly or view it on GitHub: #6661 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[grumps]

You can add an outside collaborator to a repository. Maybe it would be more desirable to create a service account and add it to the repository rather than lose 4 admins?

[johnbelamaric]

I think more of us have to give it up, I think only two are allowed but our 4 were grandfathered in.

I suggest we should give it up, and just add the common account and maybe the CNCF admin account.

[johnbelamaric]

You can add an outside collaborator to a repository. Maybe it would be more desirable to create a service account and add it to the repository rather than lose 4 admins?

That's GitHub, we're taking about Docker Hub, I believe

[grumps]

My screenshot is from dockerhub. I’ve never used it personally but at least the UI makes me think that will work.

…

On Tue, Jul 9, 2024, at 07:27, John Belamaric wrote: > You can add an outside collaborator to a _repository._ Maybe it would be more desirable to create a service account and add it to the repository rather than lose 4 admins? > > > That's GitHub, we're taking about Docker Hub, I believe -- Reply to this email directly or view it on GitHub: #6661 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[grumps]

Also, if you want an end users perspective: 1. Might be nice to get the image updated if switching admins around will take a while logistically 2. Fewer admin accounts the better overall and the project service account and CNCF account seem like good choices assuming good choices are made when creating the project service account.

…

On Tue, Jul 9, 2024, at 07:27, John Belamaric wrote: > You can add an outside collaborator to a _repository._ Maybe it would be more desirable to create a service account and add it to the repository rather than lose 4 admins? > > > That's GitHub, we're taking about Docker Hub, I believe -- Reply to this email directly or view it on GitHub: #6661 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[chrisohaver]

do you have access to the coredns.io email management?

AFAIK, I do not have email mgmt access.

[chrisohaver]

Create a group like, "[email protected]" or "[email protected]". If we don't have access to create those, then maybe we can this on the CNCF list server instead.

It seems that the current maintainers may no longer have direct control over coredns.io email forwarding.

@caniszczyk, do you know how we can get a email provisioned in CNCF to be used as a docker upload account...

named something like: coredns-docker

forwarding to the following:

[email protected]
[email protected]
[email protected]
[email protected]

Thanks!

[johnbelamaric]

I think @yongtang has access to do that.

[chrisohaver]

@yongtang do you have time to assist?