xelalexv / dregsy Goto Github PK

View Code? Open in Web Editor NEW

247.0 9.0 52.0 390 KB

Keep container registries in sync

Home Page: https://buymeacoffee.com/xelalex

License: Apache License 2.0

Go 85.67% Dockerfile 4.04% Makefile 6.45% Shell 3.84%

docker k8s sync-tasks aws-ecr registries skopeo

dregsy's Issues

remove Skopeo submodule

The Skopeo project is phasing out static builds, so the previous approach of building a FROM scratch image for dregsy with just the two binaries no longer works. Instead, we should move to something like Alpine as the base, and install Skopeo via package management during image build.

Build Skopeo binary as part of dregsy build

Recently, it has become difficult to keep the exact same version of Skopeo in both the Ubuntu and Alpine based container images released for dregsy. Also, the Skopeo versions offered in both package managers are far behind the latest Skopeo release, which delays the adoption of new Skoepo features. We should therefore start again to build the Skopeo binary as part of the dregsy build. This should be fairly straightforward using Docker multistage builds.

unauthorized to list catalog

Hello I get a "unauthorized to list catalog",

i want to sync images from a harbor instance, the login is correct (checked several times, i can push to it with the same credentials), the link works too.

level=error msg="error getting catalog page: GET https://hub.domain.com/v2/_catalog?last=&n=100: UNAUTHORIZED: unauthorized to list catalog"

to the config:

i would like to syncronize two namespaces with the help of regex, harbor itself offers this possibility only very limited.

?last=&n=100
what still surprises me is that the number of tags to be retrieved is set to 100, even though 1000 is set in the config.

i use the latest xelalex/dregsy docker image version

in another sync job, i push to the same harbor instance, that works.

But with the list there are problems

I hope you can help me

Support ANSI Coloured Log Output

I use GitHub Actions which has support for ANSI coloured output https://github.blog/2020-09-23-a-better-logs-experience-with-github-actions/

From my perspective it would be cool if we could log using this format. I run this app via GitHub Actions as I have a narrower use case for it making it not worth going through the hassle of doing it natively on k8s for my shop.

Syncing GCP repositories

I'm trying to sync docker images from the Google Container Registry (us.gcr.io) using a service account (https://cloud.google.com/container-registry/docs/advanced-authentication). There is "auth" with encoded json:

_json_key:{
  "type": "service_account",
  "project_id": "...
  ...
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
 ...

Any chance to set up synchronization with GCP repos? I tried skopeo and docker driver without success.

My errors:

2019-09-06T10:22:38Z [ERROR] invalid character '_' looking for beginning of value
2019-09-06T10:22:39Z [ERROR] error listing image tags: time="2019-09-06T10:22:39Z" level=fatal msg="Error reading manifest latest in us.gcr.io/xxx: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"

and with docker as relay:

2019-09-06T10:23:35Z [ERROR] error pulling source image 'us.gcr.io/xxx': Error response from daemon: pull access denied for us.gcr.io/xxx, repository does not exist or may require 'docker login'

docker daemon:

denied: Failed to read tags for host 'us.gcr.io', repository '/v2/xxx/tags/list'"

Thanks.

docker relay: can't use docker.io as source

Tested pulling from DockerHub and pushing to a private GitLab, using docker relay.
It seems the tool has problems with retagging images named docker.io/user/image.
I found the problem in docker.go: client.listImages(): it takes the image from config file docker.io/usr/image and tries to match it with an image on docker host, which is saved as usr/image.

exclude images names matching a pattern

Hi,

Is it possible to to exclude images that match a defined pattern?

Thanks

switch to logrus for logging

Ignore immutable registry errors when source image changed with a flag or configuration

Hi all,

I've encountered an issue uploading images to my immutable registry.

When the source image gets changed for whatever reason dregsy tries reuploading the image but fails because the image tag already exists and can't be overwritten.

Log message:

The image tag '1.20.12' already exists in the 'xxxxxx/kubectl' repository and cannot be overwritten because the repository is immutable.

I think there should be a configuration/flag available to ignore errors like this and simply drop a warning at the end saying something like:

The source image for <insert repo name> was changed in < tag version >. The image was not uploaded because the target registry is immutable.

But this should not be an error just a warning.
I suggest a new field for the configuration of dregsy tasks something like this:

"tasks": {
  [
    "source": {
      "regsitry": "my-registry.com",
      "auth": "xxxxxxxxx"
    },
    "target": {
      "registry": "my-traget-registry.com",
      "auth": "xxxxxxxxx",
      "immutable": true
    },
    ...
  ]
}

or simply a flag called "ignoreImmutable" or something like this.

Let me know what you think.

Using workload identity while using skopeo to push images to GCR from inside a container

Can this use workload identity while using skopeo to push images to GCR from inside a container?

Update ubuntu sha256 to mitigate High security findings

Hello,

thank you very much on this very cool tool :)!

Recently there were some high findings related to ubuntu base images:
https://ubuntu.com/security/CVE-2022-3786
https://ubuntu.com/security/CVE-2022-3602

so to mitigate those, can you please, update sha256 version of ubuntu base image to latest one:
ubuntu@sha256:7cfe75438fc77c9d7235ae502bf229b15ca86647ac01c844b272b56326d56184

and release a new version :)

I have tried out and its working fine + no more security findings.

Add option to enable image scan on push when creating ECR repo

It is great that repos are created if they don't exist, but for ECR it would be nice if we can enable image scanning on push for the created repo:
https://docs.aws.amazon.com/sdk-for-go/api/service/ecr/#ImageScanningConfiguration

Would you consider adding such option?

Support ECR public

Any plans to support ECR public?

the API is seperate from the normal ECR one: https://docs.aws.amazon.com/sdk-for-go/api/service/ecrpublic/#ECRPublic.GetAuthorizationToken so while current ecr auth implementation
won't work implementing ECR public would be very analogous

Add command line flag to limit tasks

I have a dregsy yaml file that syncs to 5 different docker registries. From time to time, I would like to be able to run a manual sync to just one or two of them w/o having to extract that data to a separate yaml file. It would be very useful to be able to specify which tasks to run from the command line. I'm thinking something along the lines of what anisble-playbook uses:
dregsy --config regsync.yml --limit sync-registry2

[feat req/debate] adding support for image digests with the skopeo relay

Hello @ALL, I am currently trying to figure out how to add support for image digests copy/sync in dregsy.

I think this is even simpler to implement than the tags based mapping which supports using regular expression to search through tags. Indeed with image digests, there is no need for search with regular expression, since a digest is unique and uniquely identify one particular image.

Image digest are also more robust than tags in certain situation when there is the need to match a very precise version of an image. And image digest are also used for crypto signature of container images.

What do you think about that? I must say, the dregsy source code is not really well documented. But I think I am starting to understand it a bit. I have several ideas as well.

Description of image digests

An image digest looks like this below: using crane

$ crane digest --platform="linux/amd64" library/busybox:1.29.2

sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd

`skopeo` & digests

A skopeo copy command using digest rather than tags looks like this :

skopeo copy --preserve-digests docker://docker.io/registry@sha256:cc6393207bf9d3e032c4d9277834c1695117532c9f7e8c64e7b7adcda3a85f39      docker-archive:./registry-linux-amd64-2.8.1.tar:myreg.net/mypath/registry:2.8.1

Important note: Skopeo (1.10.0) DOES NOT support using tags and digest at the same time.

dregsy configuration with digests

Below is an example of what could be a dregsy-config.yaml file which uses digests

# test/fixtures/config/skopeo-digest-valid.yaml
relay: skopeo

lister:
  maxItems: 50
  cacheDuration: 30m

tasks:
- name: test-skopeo-digest-valid
  interval: 30
  verbose: true
  source:
    registry: registry.hub.docker.com
  target:
    registry: 127.0.0.1:5000
    auth: something
    skip-tls-verify: true
  mappings:
  # Tags based mapping
  #~~~~~~~~~~~~~~~~~
  - from: library/busybox
    to: skopeo/library/busybox
    tags: ['1.29.2', '1.29.3', 'latest']
    platform: linux/arm/v6

  # Digests based mapping
  #~~~~~~~~~~~~~~~~~~~~
  - from: library/busybox
    digests:
    - sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
    - sha256:76582bc9c59276ea11459bf5ff4f54fd5b7fd23ff622d80479156108fdd26470
    - sha256:6ba3c395f0a2941114d8dcdf80bedcc7e654252f5870dd94daff9cc3188f3eb2
    to: skopeo/library/busybox

  # Using either the WEB GUI at https://hub.docker.com/_/busybox/tags. Or using
  # the tool crane* to get digests from CLI.
  # *crane : https://github.com/google/go-containerregistry/tree/main/cmd/crane
  #
  # $ crane digest --platform="linux/amd64" library/busybox:1.29.2
  # sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
  #
  # $ crane digest --platform="linux/arm/v6" library/busybox:1.29.2
  # sha256:6ba3c395f0a2941114d8dcdf80bedcc7e654252f5870dd94daff9cc3188f3eb2
  #
  # $ crane digest --platform="linux/arm64" library/busybox:1.29.3
  # sha256:76582bc9c59276ea11459bf5ff4f54fd5b7fd23ff622d80479156108fdd26470

It is probably better to differentiate and separate "Tags based mapping" and "Digests based mapping", since skopeo does not support both at the same time.

However it might be possible to have a more compact version where both a tag list and a digest list are part of the same mapping object. The skopeo copy requests are different when they use tags or digest (see above the skopeo example with digest).

Also with digest, you can ignore platform:, since the digest already uniquely identify a precise image.

  mappings:
  # Tags based mapping
  #~~~~~~~~~~~~~~~~~
  - from: library/busybox
    to: skopeo/library/busybox
    tags: ['1.29.2', '1.29.3', 'latest']
    platform: linux/arm/v6
    digests:
    - sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
    - sha256:76582bc9c59276ea11459bf5ff4f54fd5b7fd23ff622d80479156108fdd26470
    - sha256:6ba3c395f0a2941114d8dcdf80bedcc7e654252f5870dd94daff9cc3188f3eb2

But it might be overkill to dedicate task to either tag only or digest only.

support token based authentication for Google Artifact Registry

We are getting issue while copying dockerhub images to google artifact registry repository (LOCATION-docker.pkg.dev)
as per documentation, it only address for Google Container Registry (GCR).
can we add support for google artifact registry repository similarly what we have for GCR
If we use GCP service account json key as auth described in here , we are getting following issue

INFO[0000] dregsy 0.3.6                                 
INFO[0000] skopeo version 1.2.1 commit: 1b6af3b15975f3142462b6affa85af708feb657f 
INFO[0000] relay ready                                   relay=skopeo
INFO[0000] syncing task                                  source=registry.hub.docker.com target=europe-west1-docker.pkg.dev task=dockerhub
INFO[0000] mapping                                       from=/library/alpine to=/<project-id>/upstream
INFO[0000] syncing tag                                   tag=3.10
DEBU[0008] Getting image source signatures              
DEBU[0012] time="2021-04-16T09:53:24Z" level=fatal msg="Error trying to reuse blob sha256:396c31837116ac290458afcb928f68b6cc1c7bdd6963fc72f52f365a2a89c1b5 at destination: Requesting bear token: invalid status code from registry 400 (Bad Request)" 
ERRO[0012] exit status 1

Tags filtering / skip tags that already exist in the target registry

Hello 👋

I have implemented some minimal logic for tag filtering: https://github.com/yannh/dregsy#tags-filtering , support for wildcards, comparison operators, and exclude tag ranges.

I have also implemented an option called "skipExistingTags" - dregsy will skip tags that already exist in the target registry. Since it might cause issues with tags that are not immutable such as 'latest' it is deactivated by default.

To easily implement both these features, I have removed the docker relay - see #21

Do you see a path to merge this back here? I am happy to work on clean PRs. But I am not sure how to implement this efficiently using the docker relay :/

Thanks! 👍

Unable to sync insecure registries

There should be an option to set registries as insecure registries, something like:

tasks:
  - name: task1
    source:
      registry: my_fancy_registry
      insecure: no
    target:
      registry: my_not_so_fancy_registry:5000
      insecure: yes

So the client would do http instead of https for the target registry in my example

exit status 1 when pushing to GAR

hi @xelalexv thx for the awesome tool btw.

when running docker run commands based on the base64 method as shown in #51
docker run --rm -e LOG_LEVEL=debug -e GOOGLE_APPLICATION_CREDENTIALS=base64 -v <absolute path to yaml>:/config.yaml xelalex/dregsy

with the config of:

relay: skopeo


tasks:

  - name: test # required
    verbose: true
    source:
      registry: registry.hub.docker.com
    target:
      registry:  us-central1-docker.pkg.dev
      auth: none
    mappings:
    - from: library/alpine
      to:  registry/docker/library/alpine
      tags: ['latest']

I am receiving 403s

time="2023-02-09T10:37:43Z" level=info msg="dregsy 0.4.4"
time="2023-02-09T10:37:43Z" level=info msg="skopeo version 1.9.3\n"
time="2023-02-09T10:37:43Z" level=info msg="relay ready" relay=skopeo
time="2023-02-09T10:37:43Z" level=info msg="syncing task" source=registry.hub.docker.com target=us-central1-docker.pkg.dev task=test
time="2023-02-09T10:37:43Z" level=info msg=mapping from=/library/alpine to=/registry/docker/library/alpine
time="2023-02-09T10:37:43Z" level=info msg="refreshing credentials" registry=registry.hub.docker.com
time="2023-02-09T10:37:43Z" level=info msg="refreshing credentials" registry=us-central1-docker.pkg.dev
time="2023-02-09T10:37:43Z" level=debug msg="verbatim tags: [latest]"
time="2023-02-09T10:37:43Z" level=debug msg="pruned tags: []"
time="2023-02-09T10:37:43Z" level=debug msg="expanded tags: [latest]"
time="2023-02-09T10:37:43Z" level=info msg="syncing tag" platform= tag=latest
time="2023-02-09T10:37:55Z" level=debug msg="Getting image source signatures"
time="2023-02-09T10:37:55Z" level=debug msg="Copying blob sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9"
time="2023-02-09T10:37:56Z" level=debug msg="time=\"2023-02-09T10:37:56Z\" level=fatal msg=\"copying system image from manifest list: trying to reuse blob sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9 at destination: Requesting bearer token: invalid status code from registry 403 (Forbidden)\""
time="2023-02-09T10:37:56Z" level=error msg="exit status 1"
time="2023-02-09T10:37:56Z" level=error msg="errors during sync"
time="2023-02-09T10:37:56Z" level=debug msg="stopping tasks"

loggin into the destination registry has no problems as shown below:

› cat base64.json | skopeo login -u _json_key_base64 --password-stdin https://us-central1-docker.pkg.dev
Login Succeeded!

and the roles of Cloud Build Service Agent, Storage Object Admin, and Artifact Registry Repository Administrator have been granted to the service account. Is there anything that is missing?

Add unit & e2e tests

There currently are no tests in dregsy. It still gets tested indirectly though, since I use it in a larger project and that project's testing would at least catch major flaws in dregsy. However, a meaningful test set is definitely needed, in particular for proceeding with new feature implementation & refactorings.

Provide a non-alpine image?

I would like to use your docker container with the aws cli so I can perform a ECR login with role based auth. I think this would be a pretty common use case for this tool. Unfortunately AWS do not provide a binary which will work on docker images based on Alpine because they're compiling them against glibc. Hacking this into an alpine image is very cumbersome and not very easy to document aws/aws-cli#4971 aws/aws-cli#4685

It would be great if we could get a non-alpine based image for people wanting to use your tool with AWS.

Omitting tag during sync results in an error when using skopeo, works using docker

The documentation states that if the tag is omitted all tags will be synced.

I am syncing from aws ecr to artifactory, when I provide a tag the sync works, when I don't it does not.

working:

    # paths in the source registry to paths in the destination; 'from' is
    # required, while 'to' can be dropped if the path should remain the same as
    # 'from'. Additionally, the tags being synced for a mapping can be limited
    # by providing a 'tags' list. When omitted, all image tags are synced.
    mappings:
      - from: dev-image1
        to: dev-image1
        tags: ['0snapshot-dev']
#      - from: test/another-image

docker run --rm -v ./config.yaml:/config.yaml  xelalex/dregsy
2020-05-08T15:37:34Z [INFO] 
2020-05-08T15:37:34Z [INFO] skopeo version 0.1.32
2020-05-08T15:37:34Z [INFO] skopeo relay ready
2020-05-08T15:37:34Z [INFO] syncing task 'task1': 'ecr.aws.com' --> 'destination.com'
2020-05-08T15:37:34Z [INFO] mapping '/dev-image1' to '/dev-image1'
2020-05-08T15:37:34Z [INFO] 
2020-05-08T15:37:34Z [INFO] syncing tag '0snapshot-dev':
Getting image source signatures
Skipping fetch of repeat blob sha256:123275d6e508d282237a22fefa5aef822b719a06496444ea89efa65da523fc4b
Skipping fetch of repeat blob sha256:6cd6a943ce277e76a108b1c860c5991d3c9acae2db14865c7a6d4a2231b38ff2
Skipping fetch of repeat blob sha256:a50b5ac4a7fb2ab0d0cc3f2d3c1063d9f7751e32fe038b7ad7545bb239027167
Skipping fetch of repeat blob sha256:a2713a4ab6152352dc7660cb029cda5ba26bb7d7dbeb80ff38678f052dac97e5
Skipping fetch of repeat blob sha256:2db755cb04039e83097e96b20cc3bffb5752498fa1e0105356447b2d9df4f0fd
Skipping fetch of repeat blob sha256:a7febb4cdc5f9be8051af037bf73b9a69644283b223b6866504d67d025a91d5d
Copying config sha256:20d5fe6d7deec98a85813f251ec3264505c4bc6fbc8dda7061c0a3c0a81fed1e
 0 B / 7.22 KB  0s
Writing manifest to image destination
Storing signatures
2020-05-08T15:37:47Z [INFO] 
2020-05-08T15:37:47Z [INFO] all done

not working:

# 'mappings' is a list of 'from':'to' pairs that define mappings of image
    # paths in the source registry to paths in the destination; 'from' is
    # required, while 'to' can be dropped if the path should remain the same as
    # 'from'. Additionally, the tags being synced for a mapping can be limited
    # by providing a 'tags' list. When omitted, all image tags are synced.
    mappings:
      - from: dev-image1
        to: dev-image1
#        tags: ['0snapshot-dev']
#      - from: test/another-image

docker run --rm -v .config.yaml:/config.yaml  xelalex/dregsy
2020-05-08T15:43:11Z [INFO] 
2020-05-08T15:43:11Z [INFO] skopeo version 0.1.32
2020-05-08T15:43:11Z [INFO] skopeo relay ready
2020-05-08T15:43:11Z [INFO] syncing task 'task1': 'ecr.aws.com' --> 'destination.com'
2020-05-08T15:43:11Z [INFO] mapping '/dev-image1' to '/dev-image1'
2020-05-08T15:43:12Z [INFO] 
2020-05-08T15:43:12Z [INFO] all done
2020-05-08T15:43:12Z [ERROR] error listing image tags: time="2020-05-08T15:43:12Z" level=fatal msg="Error reading manifest latest in ecr.aws.com/dev-image1: manifest unknown: Requested image not found" 
, exit status 1

Skopeo version needs push

Currently used version uses and old version using ubuntu 17.10. Build fails on git update due to that.

Can't use ECR without auth-refresh

The documentation implies auth-refresh is an optional feature for ECR registries, but if I use a configuration that does not have auth-refresh set, dregsy still tries to refresh credentials at start up, which doesn't work.

The reason I don't want auth-refresh is that I need separate credentials for source and target registries. Ideally, I would be able to specify a different AWS SDK auth profile for each, but for now, I simply wanted to use my hardcoded credentials without any attempt to refresh them.

Allow use of skopeo's `sync` command instead of `copy`

I am using skopeo to sync my registries, but every time it runs it transfers data, even if it has already copied the image in the prior pass. I see the last modified time in the destination registry update on each run. My expectation was that this would work more like rsync -- i.e. only copying data that has changed on each task run.

Even more importantly, I would expect skopeo to only copy layers that have changed. It looks like the way it is running now it will copy layers that already exist on the destination multiple times if they are used by multiple images.

Is there a way, short of editing the tags list in the yaml file after a successful copy, to only have dregsy sync images that don't exist in the destination?

I am using a local build of dregsy built from the master branch in GitHub and skopeo 1.4.1.

Obfuscated config and log output:

relay: skopeo
skopeo:
  binary: skopeo
docker:
  dockerhost: unix:///var/run/docker.sock
  api-version: 1.24
lister:
  maxItems: 100
  cacheDuration: 1h
tasks:
  - name: sync-lab
    interval: 60
    verbose: true
    source:
      registry: registry.xxxxxx.com
      auth: xxxxxxxxxxx
    target:
      registry: registry1.xxxxxx.com
      auth: xxxxxxxxxxx
    mappings:
      - from: regex:sync/alpine
        to: regex:sync/alpine,synctest-$1

❯ LOG_LEVEL=debug _build/bin/dregsy --config shipsync.yml
INFO[0000] dregsy 0.4.1-dirty
DEBU[0000] lister max items set to 100
DEBU[0000] lister cache duration set to 1h0m0s
INFO[0000] skopeo version 1.4.1
INFO[0000] relay ready                                   relay=skopeo
DEBU[0000] task starts ticking                           task=sync-lab
DEBU[0000] sending initial fire                          task=sync-lab
INFO[0000] waiting for next sync task...
INFO[0000] syncing task                                  source=registry.xxxxxx.com target="registry1.xxxxxx.com" task=sync-lab
INFO[0000] mapping                                       from=/sync/alpine to=/alpine-synctest
INFO[0000] refreshing credentials                        registry=registry.xxxxxx.com
INFO[0000] refreshing credentials                        registry="registry1.xxxxxx.com"
INFO[0001] syncing tag                                   tag=3.14
DEBU[0002] Getting image source signatures
DEBU[0003] Copying blob sha256:ce42088e28cfff1fd4a25fe4aa16b527c4ddb72ee25d6045588985e20add021b
DEBU[0003] Copying config sha256:cc924c6569616daebad97cf976a78b70a612caf562d91615d363b5478d8f2c2a
DEBU[0004] Writing manifest to image destination
DEBU[0004] Storing signatures
INFO[0004] waiting for next sync task...
DEBU[0060] task firing                                   task=sync-lab
INFO[0060] syncing task                                  source=registry.xxxxxx.com target="registry1.xxxxxx.com" task=sync-lab
INFO[0060] mapping                                       from=/sync/alpine to=/alpine-synctest
INFO[0060] refreshing credentials                        registry=registry.xxxxxx.com
INFO[0060] refreshing credentials                        registry="registry1.xxxxxx.com"
INFO[0061] syncing tag                                   tag=3.14
DEBU[0062] Getting image source signatures
DEBU[0065] Copying blob sha256:ce42088e28cfff1fd4a25fe4aa16b527c4ddb72ee25d60455
88985e20add021b
DEBU[0066] Copying config sha256:cc924c6569616daebad97cf976a78b70a612caf562d91615d363b5478d8f2c2a
DEBU[0067] Writing manifest to image destination
DEBU[0067] Storing signatures
INFO[0067] waiting for next sync task...
DEBU[0120] task firing                                   task=sync-lab
INFO[0120] syncing task                                  source=registry.xxxxxx.com target="registry1.xxxxxx.com" task=sync-lab
INFO[0120] mapping                                       from=/sync/alpine to=/alpine-synctest
INFO[0120] refreshing credentials                        registry=registry.xxxxxx.com
INFO[0120] refreshing credentials                        registry="registry1.xxxxxx.com"
INFO[0121] syncing tag                                   tag=3.14
DEBU[0122] Getting image source signatures
DEBU[0123] Copying blob sha256:ce42088e28cfff1fd4a25fe4aa16b527c4ddb72ee25d6045588985e20add021b
DEBU[0123] Copying config sha256:cc924c6569616daebad97cf976a78b70a612caf562d91615d363b5478d8f2c2a
DEBU[0124] Writing manifest to image destination
DEBU[0124] Storing signatures
INFO[0124] waiting for next sync task...
^C
INFO[0161] received signal, stopping ...                 signal=interrupt
DEBU[0161] stopping tasks
DEBU[0161] task exiting                                  task=sync-lab
DEBU[0161] task exited                                   task=sync-lab
INFO[0161] all done
DEBU[0161] exit main

AWS ECR example in documentation

Will be good in AWS ECR example to describe whole process how it should look, like:

how dregsy config should look
what additional commands i must do like aws login or something (or is dregsy doing it for me)
what more?

attempt to sync nexus docker registry

I'm trying to synchronize docker registry, but I'm getting fail

2020-07-27T16:10:11Z [INFO]
2020-07-27T16:10:41Z [INFO] task 'task1' fired too soon, skipping
2020-07-27T16:10:41Z [INFO] waiting for next sync task...
2020-07-27T16:10:41Z [INFO]
2020-07-27T16:11:41Z [INFO] syncing task 'task1': '10.10.10.10:5000' --> '10.20.20.20:5000'
2020-07-27T16:11:41Z [INFO] mapping '/dev/app' to '/dev/app'
2020-07-27T16:11:41Z [INFO] pulling source image '10.10.10.10:5000/dev/app'
64: Pulling from dev/app
Digest: sha256:033952a79c057676881dd67340e92dab53e6385564575a3d3f9d99a4cece3b30
65: Pulling from dev/app
Digest: sha256:d383efc0a584ca9424a1282028ae549272a4cdad38e26bf20709e8d834ff3010
72: Pulling from dev/app
Digest: sha256:e0ad897b05da6a4829af49ee4f71e10d06ed4710b35f3cb883a0380bdad33c50
74: Pulling from dev/app
Digest: sha256:53008ada6c3dfa96ecb96b6c51f43809791ea5cee83d8ef1fad3c8a24cc20b05
Status: Image is up to date for 10.10.10.10:5000/dev/app
2020-07-27T16:11:41Z [INFO] relevant tags
2020-07-27T16:11:41Z [ERROR] error listing all tags of source image '10.10.10.10:5000/dev/app': malformed image ref: invalid reference format
2020-07-27T16:11:41Z [ERROR] error pushing target image: An image does not exist locally with the tag: 10.20.20.20:5000/dev/app
2020-07-27T16:11:41Z [INFO] setting tags for target image '10.20.20.20:5000/dev/app'
2020-07-27T16:11:41Z [INFO] pushing target image '10.20.20.20:5000/dev/app'
The push refers to repository [10.20.20.20:5000/dev/app]
2020-07-27T16:11:41Z [INFO]
2020-07-27T16:11:41Z [INFO] waiting for next sync task...
2020-07-27T16:11:41Z [INFO]



# relay type, either 'skopeo' or 'docker'
relay: docker

docker:
  # Docker host to use as the relay
  dockerhost: unix:///var/run/docker.sock
  # Docker API version to use, defaults to 1.24
  api-version: 1.24

# list of sync tasks
tasks:

  - name: task1 # required

    # interval in seconds at which the task should be run; when omitted,
    # the task is only run once at start-up
    interval: 60

    # determines whether for this task, more verbose output should be
    # produced; defaults to false when omitted
    verbose: true

    # 'source' and 'target' are both required and describe the source and
    # target registries for this task:
    #  - 'registry' points to the server; required
    #  - 'auth' contains the base64 encoded credentials for the registry
    #    in JSON form {"username": "...", "password": "..."}
    #  - 'auth-refresh' specifies an interval for automatic retrieval of
    #    credentials; only for AWS ECR (see below)
    #  - 'skip-tls-verify' determines whether to skip TLS verification for the
    #    registry server (only for 'skopeo', see note below); defaults to false
    source:
      registry: 10.10.10.10:5000
      auth: YYYYWRtaW46TmNZMTMwcnI=
    #  skip-tls-verify: true
    target:
      registry: 10.20.20.20:5000
      auth: YYYYWRtaW46TmNZMTMwcnI=
    #  skip-tls-verify: true

    # 'mappings' is a list of 'from':'to' pairs that define mappings of image
    # paths in the source registry to paths in the destination; 'from' is
    # required, while 'to' can be dropped if the path should remain the same as
    # 'from'. Additionally, the tags being synced for a mapping can be limited
    # by providing a 'tags' list. When omitted, all image tags are synced.
    mappings:
      - from: dev/app

docker run --privileged --rm -v /var/run/docker.sock:/var/run/docker.sock dregsy:0.3.0

Use GCR without auth?

Hi, I'd like to synch some public repos from GCR. An example being k8s.gcr.io/autoscaling/vpa-admission-controller.

It seems like the current implementation of dregsy requires me to provide project credentials because the registry has gcr.io in the name. This isn't necessary and I would like to avoid it. Is there any way to do this that I am missing?

If not, I'm willing to make a PR to address this. Perhaps the code could proceed without GCR credentials and treat the lack thereof as a warning and not an error?

unexpected end of JSON input

Any idea where an error like the one above would be coming from? It seems like things are syncing, but I see that in the logs for every repository.

time="2022-02-24T00:21:26Z" level=info msg="syncing task" source=123456789012.dkr.ecr.us-east-1.amazonaws.com target="remotereg01.example.com:8083" task=sync-wonder
time="2022-02-24T00:21:26Z" level=info msg=mapping from="regex:syncbase/.*" to="regex:syncbase/(.+)/[^/]*$,$1/"
time="2022-02-24T00:21:26Z" level=info msg="refreshing credentials" registry=123456789012.dkr.ecr.us-east-1.amazonaws.com
time="2022-02-24T00:21:29Z" level=info msg="refreshing credentials" registry="remotereg01.example.com:8083"
time="2022-02-24T00:21:29Z" level=debug msg="retrieving repository list"
time="2022-02-24T00:21:29Z" level=debug msg="ECR retrieving image list"
time="2022-02-24T00:21:32Z" level=debug msg="caching repository list"
time="2022-02-24T00:21:32Z" level=error msg="unexpected end of JSON input"
time="2022-02-24T00:21:32Z" level=error msg="unexpected end of JSON input"
time="2022-02-24T00:21:32Z" level=info msg="syncing tag" tag=21.10.0
time="2022-02-24T00:21:32Z" level=debug msg="time=\"2022-02-24T00:21:32Z\" level=info msg=\"Tag presence check\" imagename=\"123456789012.dkr.ecr.us-east-1.amazonaws.com/syncbase/appdynamics/cluster-agent:21.10.0\" tagged=true"
time="2022-02-24T00:21:32Z" level=debug msg="time=\"2022-02-24T00:21:32Z\" level=info msg=\"Copying image ref 1/1\" from=\"docker://123456789012.dkr.ecr.us-east-1.amazonaws.com/syncbase/appdynamics/cluster-agent:21.10.0\" to=\"docker://remotereg01.example.com:8083/appdynamics/cluster-agent:21.10.0\""
time="2022-02-24T00:21:33Z" level=debug msg="Getting image source signatures"
time="2022-02-24T00:21:37Z" level=debug msg="Skipping: image already present at destination"
time="2022-02-24T00:21:37Z" level=debug msg="time=\"2022-02-24T00:21:37Z\" level=info msg=\"Synced 1 images from 1 sources\""
time="2022-02-24T00:21:37Z" level=error msg="unexpected end of JSON input"
time="2022-02-24T00:21:37Z" level=info msg="syncing tag" tag=0.6.11

Repo name duplication when using wildcards

I am seeing some very odd duplication of repo names now that I have wildcards working (I moved to using eco.io as my source repo). For example with the config below, I see it trying to do this sync:

time="2022-02-08T17:45:04Z" level=info msg=mapping from="regex:myrepo/.*" to="regex:myrepo/(.+),$1/"
time="2022-02-08T17:45:04Z" level=info msg="refreshing credentials" registry=123456789.dkr.ecr.us-east-1.amazonaws.com
time="2022-02-08T17:45:07Z" level=info msg="refreshing credentials" registry="registry1.example.com:8083"
time="2022-02-08T17:45:07Z" level=debug msg="retrieving repository list"
time="2022-02-08T17:45:07Z" level=debug msg="ECR retrieving image list"
time="2022-02-08T17:45:10Z" level=debug msg="caching repository list"
time="2022-02-08T17:45:10Z" level=info msg="syncing tag" tag=5.0.6-alpine
time="2022-02-08T17:45:10Z" level=debug msg="time=\"2022-02-08T17:45:10Z\" level=info msg=\"Tag presence check\" imagename=\"123456789.dkr.ecr.us-east-1.amazonaws.com/myrepo/baserepo/redis:5.0.6-alpine\" tagged=true"
time="2022-02-08T17:45:10Z" level=debug msg="time=\"2022-02-08T17:45:10Z\" level=info msg=\"Copying image ref 1/1\" from=\"docker://123456789.dkr.ecr.us-east-1.amazonaws.com/myrepo/baserepo/redis:5.0.6-alpine\" to=\"docker://registry1.example.com:8083/baserepo/redis/redis:5.0.6-alpine\""

notice the redis/redis on the last line. I would expect that to just be baserepo/redis:5.0.6-alpine.

My config:

docker:
  api-version: '1.24'
  dockerhost: unix:///var/run/docker.sock
lister:
  cacheDuration: 1h
  maxItems: 100
relay: skopeo
skopeo:
  binary: skopeo
  mode: sync
tasks:
  - name: sync-reg1
    interval: 1800
    source:
      registry: 123456789.dkr.ecr.us-east-1.amazonaws.com
      auth-refresh: 1h
    target:
      registry: registry1.example.com:8083
    mappings:
    - from: regex:myrepo/.*
      to: regex:myrepo/(.+),$1/
    verbose: true

I have tried the to: regex with and without the trailing slash, it doesn't seem to make a difference. From the docs, and my earlier testing when I wasn't using a regex for the from: config, I would think this should work. I'm going try try something crazy like this next to: regex:myrepo/(.+)/[^/]*$,$1. But I'm afraid that will end up syncing /baserepo:5.0.6-alpine in the example above.

Any ideas why it's doubling the repo name?

Add multiarch support

ARM64 architecture becomes very popular and AWS EKS/ECR already supports this. So, syncing AMD64+ARM64 is good idea.
https://docs.docker.com/docker-for-mac/multi-arch/

Skopeo is already supports this feature. So, it needs to bypass "--all" from Dregsy command line to Skopeo.

skopeo copy
options:
--all
If source-image refers to a list of images, instead of copying just the image which matches the current OS and architecture (subject to the use of the global --override-os, --override-arch and --override-variant options), attempt to copy all of the images in the list, and the list itself.

Can the source of mappings use a pattern ?

Suppose I want to synchronize all repositories that are in a sub path : can I describe my mappings with one rule using a pattern or must I enumerate all the mappings (one by repository).
Can I have

mappings:
  - from: library/*

instead of

mappings:
  - from: library/repo1
  - from: library/repo2
  ....
  - from: library/repo99

platform architecture destination tag labeling

hi, i was wondering if dregsy has the ability to label the destination tag based on different architecture, so for instance tag (v4.7, amd64), (v4.6,aarch) when specifying platform: all in the config.yaml.

Thanks.

add support for a blacklist tag array

Sometimes you want to sync everything BUT a set of tags, this would be a great addition and I would think fairly quick to implement for someone who knows Go, sadly this person isn't me :(

    mappings:
      - from: repository/image
        to: repository/image
        blacklist-tags: ['some_tag', 'another_tag']

Exit success on failure instead of exit failure

Currently when an image is not available the program aborts with access success. Could the exit code be changed to non zero?

Attach repository policies to repository created in ECR

Hello! I am working on syncing images from Dockerhub/quay.io to ECR. However, I would need to attach an ECR Repository Policy to the repositories created by Dregsy - as far as I understand dregsy can not do this yet.

I am not too sure how to add this to the configuration file. Since policies are per-repository, I could need to add a configuration element to the mapping section - also, there could be several, so it should be an array?

mappings:
  - from: test/image
    to: archive/test/image
    [policies|ecr_policies]:
    - /path/to/policy.json

Is this something you would be willing to accept as a Pull Request? I would love your input.
Many thanks!

Mirror organization / namespace

Could such a feature be added?

bump skopeo version

Support parallel tasks

I'm working on a solution to synchronize images to a set of remote registries, and having the ability to run tasks in parallel would be a huge help. Right now I'm building n different configs and running m instances of dregsy. It would be much cleaner to have a single config will all of my tasks defined and have dregsy handle managing the parallelism.

Something along the lines of:

relay: skopeo

skopeo:
  binary: skopeo
  mode: copy
  # Number of tasks to run in parallel
  parallel_tasks: 4

docker build fails

I've tried to build the docker image but it fails. Unfortunately I'm not into go and its tools.

docker build  -t dregsy .
Sending build context to Docker daemon  176.1kB

Step 1/11 : FROM golang:1.10 as builder
1.10: Pulling from library/golang
54f7e8ac135a: Pulling fs layer
d6341e30912f: Pulling fs layer
087a57faf949: Pulling fs layer
5d71636fb824: Pulling fs layer
f368dba6a331: Pulling fs layer
1ea0c245e5c5: Pulling fs layer
f81588ba3d2a: Pulling fs layer
5d71636fb824: Waiting
f368dba6a331: Waiting
1ea0c245e5c5: Waiting
f81588ba3d2a: Waiting
087a57faf949: Verifying Checksum
087a57faf949: Download complete
d6341e30912f: Verifying Checksum
d6341e30912f: Download complete
54f7e8ac135a: Verifying Checksum
54f7e8ac135a: Download complete
f368dba6a331: Verifying Checksum
f368dba6a331: Download complete
5d71636fb824: Verifying Checksum
5d71636fb824: Download complete
f81588ba3d2a: Verifying Checksum
f81588ba3d2a: Download complete
1ea0c245e5c5: Verifying Checksum
1ea0c245e5c5: Download complete
54f7e8ac135a: Pull complete
d6341e30912f: Pull complete
087a57faf949: Pull complete
5d71636fb824: Pull complete
f368dba6a331: Pull complete
1ea0c245e5c5: Pull complete
f81588ba3d2a: Pull complete
Digest: sha256:dc6610eb1389be884a520e4c8846739d1834128a6ab2cc0a937fb52e231904c0
Status: Downloaded newer image for golang:1.10
 ---> 0a19f4d16598
Step 2/11 : LABEL stage=intermediate
 ---> Running in 78cd1d931e35
Removing intermediate container 78cd1d931e35
 ---> 393d5664ee6c
Step 3/11 : COPY . $GOPATH/src/github.com/xelalexv/dregsy/
 ---> c1aab0ab6287
Step 4/11 : WORKDIR $GOPATH/src/github.com/xelalexv/dregsy/
 ---> Running in 27c9fe1dc4c1
Removing intermediate container 27c9fe1dc4c1
 ---> d0af0c5349b4
Step 5/11 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -a 	-tags netgo -installsuffix netgo -ldflags '-w' 	-o /go/bin/dregsy ./cmd/dregsy/
 ---> Running in 8bb6d29501d8
internal/pkg/sync/config.go:17:2: cannot find package "github.com/aws/aws-sdk-go/aws" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/aws/aws-sdk-go/aws (vendor tree)
	/usr/local/go/src/github.com/aws/aws-sdk-go/aws (from $GOROOT)
	/go/src/github.com/aws/aws-sdk-go/aws (from $GOPATH)
internal/pkg/sync/config.go:18:2: cannot find package "github.com/aws/aws-sdk-go/aws/awserr" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/aws/aws-sdk-go/aws/awserr (vendor tree)
	/usr/local/go/src/github.com/aws/aws-sdk-go/aws/awserr (from $GOROOT)
	/go/src/github.com/aws/aws-sdk-go/aws/awserr (from $GOPATH)
internal/pkg/sync/config.go:19:2: cannot find package "github.com/aws/aws-sdk-go/aws/session" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/aws/aws-sdk-go/aws/session (vendor tree)
	/usr/local/go/src/github.com/aws/aws-sdk-go/aws/session (from $GOROOT)
	/go/src/github.com/aws/aws-sdk-go/aws/session (from $GOPATH)
internal/pkg/sync/config.go:20:2: cannot find package "github.com/aws/aws-sdk-go/service/ecr" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/aws/aws-sdk-go/service/ecr (vendor tree)
	/usr/local/go/src/github.com/aws/aws-sdk-go/service/ecr (from $GOROOT)
	/go/src/github.com/aws/aws-sdk-go/service/ecr (from $GOPATH)
internal/pkg/relays/docker/docker.go:16:2: cannot find package "github.com/docker/docker/api/types" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/docker/docker/api/types (vendor tree)
	/usr/local/go/src/github.com/docker/docker/api/types (from $GOROOT)
	/go/src/github.com/docker/docker/api/types (from $GOPATH)
internal/pkg/relays/docker/docker.go:17:2: cannot find package "github.com/docker/docker/client" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/docker/docker/client (vendor tree)
	/usr/local/go/src/github.com/docker/docker/client (from $GOROOT)
	/go/src/github.com/docker/docker/client (from $GOPATH)
internal/pkg/relays/docker/docker.go:18:2: cannot find package "github.com/docker/docker/pkg/jsonmessage" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/github.com/docker/docker/pkg/jsonmessage (vendor tree)
	/usr/local/go/src/github.com/docker/docker/pkg/jsonmessage (from $GOROOT)
	/go/src/github.com/docker/docker/pkg/jsonmessage (from $GOPATH)
internal/pkg/log/log.go:13:2: cannot find package "golang.org/x/crypto/ssh/terminal" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/golang.org/x/crypto/ssh/terminal (vendor tree)
	/usr/local/go/src/golang.org/x/crypto/ssh/terminal (from $GOROOT)
	/go/src/golang.org/x/crypto/ssh/terminal (from $GOPATH)
internal/pkg/sync/config.go:14:2: cannot find package "gopkg.in/yaml.v2" in any of:
	/go/src/github.com/xelalexv/dregsy/vendor/gopkg.in/yaml.v2 (vendor tree)
	/usr/local/go/src/gopkg.in/yaml.v2 (from $GOROOT)
	/go/src/gopkg.in/yaml.v2 (from $GOPATH)
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -a 	-tags netgo -installsuffix netgo -ldflags '-w' 	-o /go/bin/dregsy ./cmd/dregsy/' returned a non-zero code: 1

Build failures on non-linux platforms

The current Makefile only works on linux, PR#61 is a patch to fix this on non-linux platforms (at least it works on both Linux and Mac now).

sync the whole registry

Is there a way to sync the full registry? I tried from: * and from: regex:* but this errors happen:

time="2021-04-07T23:20:18Z" level=info msg=mapping from="/*" to="/*"                                     
time="2021-04-07T23:20:18Z" level=error msg="error listing image tags for ref 'source/*': time=\"2021-04-07T23:20:18Z\" level=fatal msg=\"invalid reference format\"\n, exit status 1"                                                                                                  
time="2021-04-07T23:20:18Z" level=error msg="one or more tasks had errors, please see log for details"

time="2021-04-07T23:31:43Z" level=info msg=mapping from="/regex:*" to="/regex:*"                         
time="2021-04-07T23:31:43Z" level=error msg="error listing image tags for ref 'source/regex:*': time=\"2021-04-07T23:31:43Z\" level=fatal msg=\"invalid reference format\"\n,exit status 1"

relay: skopeo
tasks:
  - name: sync # required
    verbose: true
    source:
      registry: source
      skip-tls-verify: true
    target:
      registry: target
      skip-tls-verify: true
    mappings:
      - from: "*"

Retire docker relay?

The docker relay seems to be much less efficient than the skopeo one. Syncing large registries involve downloading all tags locally first - which might take quite some space. Since the relay implementations are quite different, it makes adding new features more challenging. Is there an advantage in using the docker relay vs the Skopeo one? How about sunsetting it?

No output from skopeo, even with increased logging

Hey, first of all thank you for this project!
We hit the docker rate limit today, but it took us a while to figure it out because there were no logs from Skopeo shown, even with increased logging. All we got was output like this:

time="2022-07-29T09:14:10Z" level=info msg=mapping from=/busybox to=/busybox
time="2022-07-29T09:14:10Z" level=info msg="refreshing credentials" registry=docker.io
time="2022-07-29T09:14:10Z" level=info msg="refreshing credentials" registry=12345678910.dkr.ecr.eu-west-1.amazonaws.com
time="2022-07-29T09:14:10Z" level=info msg="target already exists" ref=12345678910.dkr.ecr.eu-west-1.amazonaws.com/busybox
time="2022-07-29T09:14:10Z" level=info msg="syncing tag" platform=all tag=latest
time="2022-07-29T09:14:50Z" level=error msg="exit status 1"
time="2022-07-29T09:14:50Z" level=error msg="errors during sync"

Only when I modified these lines to os.Stdout & os.Stderr I was able to see the Skopeo output. It's too much to show the user everything, but maybe it's better to buffer the output and print it if there was an error? We felt a bit mocked when it said "something failed, but I won't tell you what" 😄

Support string filtering on top of semver tag filtering

Problem: I want to sync all node images with semver constraints >=12.19.x || >=14.15.x and only if the tag name contains alpine or -buster.
This is currently not possible with the tag filtering .

One could use the regex feature:

- 'regex: 12\.19\.[0-9]-(alpine|buster)'
- 'regex: 14\.15\.[0-9]-(alpine|buster)'

however for every new node minor version we would have to modify that list so that it stays up to date.
Is there any way to achieve the desired outcome with the current tag filtering?

GCP auth per target and source

Hi, i would like to use the tool to sync between two registries that i have on GCP, the issue is each of them sits in a different project.
An option to set the authentication configuration for each of them would allow me two authenticate separately to each project with the least privileges.

Webhook support

Hi @xelalexv, thanks for providing this tool which has a lot of helpers for ECR compared to "raw Skopeo" (mainly update auth tokens and creating a repo if does not exist).

At Padoa, we are using it to keep in sync repositories from 2 distinct cloud provider managed registry instances: Azure Container Registry to AWS ECR.
The best way we found currently is to send a webhook at each manifest push on the origin so that it triggers a sync of the pushed tag. We hence added a support for incoming webhook as well as support for multiple parallel sync.

It is still a draft (we also need to update doc) and work is in progress but tell us if it is something you would like to include in the original dregsy, and how we can contribute in this way if so.

The fork is available here: https://github.com/padoa/dregsy/

Thanks in advance for any answer :)

closing one-off task causes panic: close of nil channel

hi, i'm using your tool to sync a bunch of docker images from dockerhub to ecr.
It worked fine till a certain point where it get this panic: close of nil channel

The following is part of the log.

... a whole bunch of successful syncs ...
08:22:40  time="2020-12-11T07:22:40Z" level=debug msg="Copying config sha256:38983bba610e04c6bd00ef5c26b86ad5cbff75cb6b83431595d6fca744a6f135"
08:22:40  time="2020-12-11T07:22:40Z" level=debug msg="Writing manifest to image destination"
08:22:40  time="2020-12-11T07:22:40Z" level=debug msg="Storing signatures"
08:22:40  time="2020-12-11T07:22:40Z" level=debug msg="stopping tasks" func="github.com/xelalexv/dregsy/internal/pkg/sync.(*Sync).SyncFromConfig" file="/go/src/dregsy/internal/pkg/sync/sync.go:149"
08:22:40  panic: close of nil channel
08:22:40  
08:22:40  goroutine 1 [running]:
08:22:40  github.com/xelalexv/dregsy/internal/pkg/sync.(*Task).stopTicking(0xc0000bdf00)
08:22:40  	/go/src/dregsy/internal/pkg/sync/task.go:138 +0x40
08:22:40  github.com/xelalexv/dregsy/internal/pkg/sync.(*Sync).SyncFromConfig(0xc0003a02e0, 0xc0001d4ea0, 0x0, 0x0)
08:22:40  	/go/src/dregsy/internal/pkg/sync/sync.go:152 +0x59b
08:22:40  main.main()
08:22:40  	/go/src/dregsy/cmd/dregsy/main.go:113 +0x24f

This is my config file

relay: skopeo
tasks:
  - name: task1
    verbose: true
    source:
      registry: docker.io
    target:
      registry: 754813824532.dkr.ecr.eu-west-1.amazonaws.com
      auth-refresh: 10h
    mappings:
      - from: alpine
        tags: ['latest']
      - from: postman/newman
        tags: ['latest','5-alpine']
      - from: aequitas/http-api-resource
        tags: ['latest']
      - from: python
        tags: ['2.7.13-alpine','3-alpine']
      - from: node
        tags: ['10-alpine']
      - from: skyscrapers/apt-package-resource
        tags: ['latest']
      - from: maven
        tags: ['3.6.3-jdk-8','3.5.2-jdk-8','3.6.3-jdk-11']
      - from: openapitools/openapi-generator-cli
        tags: ['v4.3.1']
      - from: selenium/standalone-chrome
        tags: ['3.141.59-20200719']
      - from: mysql
        tags: ['5.7']
      - from: pjpinten/packer-g10k
        tags: ['1.2']
      - from: renovate/renovate
        tags: ['latest']

It is ran as a jenkins job, so i do the scheduling there. This is a one shot job.
Jenkins executes:

docker run --rm -v /opt/jenkins/workspace/ose-Concourse_sync-docker_master/config.yaml:/config.yaml -e LOG_METHODS=true -e LOG_LEVEL=trace -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** xelalex/dregsy

Running this on my local machine works fines so maybe there is some kind of weird race condition?
I'm not to familiar with go. But it seems that there are timers (threads) in use which might cause this.
For me this is not really blocking as the sync itself works fine but the command does fail.
This is not very clean and my team asks questions about it.

Error syncing entire namespace

I have set up a namespace in my Quay registry that I want to sync to a remote Nexus3 registry. Things work fine if I specify a specific repo in my source registry, but not if I specify a namespace or a regex into that namespace.

For example, this:

    mappings:
      - from: regex:sync/.*
        to: regex:sync/(.*),synctest-$1

give me this error:

ERRO[0000] error getting catalog page: GET https://registry.xxxxxx.com/v2/auth?scope=registry%3Acatalog%3A%2A&service=registry.xxxxxx.com: INVALID_REQUEST: Unable to decode repository and actions: registry:catalog:*; map[]

If I change the mapping to:

    mappings:
      - from: sync/alpine
        to: regex:sync/(.*),synctest-$1

It will sync the alpine repo correctly.

Any idea what might be going on? I am assuming this has something to do with the way Quay is responding to the wildcard request, but want to make sure before I go open a ticket with RedHat.

I am using a local build of dregsy built from the master branch in GitHub and skopeo 1.4.1.

invalid character '_' looking for beginning of value

I am trying to dregsy image & skopeo relay for copying dockerhub image to google artifact registry repository but seeing following error

time="2021-04-15T07:54:53Z" level=error msg="invalid character '_' looking for beginning of value"

following is config.yaml file

relay: skopeo

# relay config sections
skopeo:
  binary: skopeo
  certs-dir: /etc/skopeo/certs.d

# list of sync tasks
tasks:

  - name: task1 # required
    interval: 60
    verbose: true
    source:
      registry: registry.hub.docker.com
      auth: <dockerhub_auth>
      lister:
        type: dockerhub
    target:
      registry: <google_artifact_registry_URL>
      auth: none
      skip-tls-verify: false
    mappings:
      - from: regex:xelalex/dregsy.*
        to: test/upstream/dregsy
        tags: ['master', 'latest']

Any help much appreciated

xelalexv / dregsy Goto Github PK

dregsy's Issues

Description of image digests

skopeo & digests

dregsy configuration with digests

Recommend Projects

Recommend Topics

Recommend Org

`skopeo` & digests