opensuse/helm-mirror

Helm plugin used to mirror repositories

helm-mirror Plugin

Helm plugin used to mirror Chart repositories

Usage

Mirror Helm Charts from an index file into a local folder.

For example:

helm-mirror https://yourorg.com/charts /yourorg/charts

This will download the index file and the latest version of the charts into the folder indicated.

The index file is a yaml that contains a list of charts in this format. Example:

apiVersion: v1
entries:
  chart:
  - apiVersion: 1.0.0
    created: 2018-08-08T00:00:00.00000000Z
    description: A Helm chart for your application
    digest: 3aa68d6cb66c14c1fcffc6dc6d0ad8a65b90b90c10f9f04125dc6fcaf8ef1b20
    name: chart
    urls:
    - https://kubernetes-charts.yourorganization.com/chart-1.0.0.tgz
  chart2:
  - apiVersion: 1.0.0
    created: 2018-08-08T00:00:00.00000000Z
    description: A Helm chart for your application
    digest: 7ae62d60b61c14c1fcffc6dc670e72e62b91b91c10f9f04125dc67cef2ef0b21
    name: chart
    urls:
    - https://kubernetes-charts.yourorganization.com/chart2-1.0.0.tgz

This will download these charts:

• https://kubernetes-charts.yourorganization.com/chart-1.0.0.tgz

• https://kubernetes-charts.yourorganization.com/chart2-1.0.0.tgz

into your destination folder.

Usage:

  helm-mirror [Repo URL] [Destination Folder] [flags]
  helm-mirror [command]

Available Commands: help Help about any command inspect-images Extract all the images of the Helm Charts. version Show version of the helm-mirror plugin

Flags:

  -a, --all-versions                                   gets all the versions of the charts in the chart repository
      --ca-file string                                 verify certificates of HTTPS-enabled servers using this CA bundle
      --cert-file string                               identify HTTPS client using this SSL certificate file
      --chart-name string                              name of the chart that gets mirrored
      --chart-version string                           specific version of the chart that is going to be mirrored
  -h, --help                                           help for mirror
  -i, --ignore-errors                                  ignores errors while downloading or processing charts
      --key-file string                                identify HTTPS client using this SSL key file
      --new-root-url https://mirror.local.lan/charts   New root url of the chart repository (eg: https://mirror.local.lan/charts)
      --password string                                chart repository password
      --username string                                chart repository username
  -v, --verbose                                        verbose output

Getting all charts

helm-mirror https://yourorg.com/charts /yourorg/charts --all-charts

This will download all the charts and all the available versions of the charts.

Getting one specific chart

helm-mirror https://yourorg.com/charts /yourorg/charts --chart-name nginx

This will download the latest version of the chart nginx.

Getting one specific chart with specific version

helm-mirror https://yourorg.com/charts /yourorg/charts --chart-name nginx --chart-version 2.14.3

This will download the version 2.14.3 of the chart nginx.

Use helm-mirror [command] --help for more information about a command.

Commands

inspect-images

Extract all the container images listed in each Helm Chart or the Helm Charts in the folder provided. This command dumps the images on stdout by default, for more options check output flag. Example:

• helm-mirror inspect-images /tmp/helm

• helm-mirror inspect-images /tmp/helm/app.tgz

The [folder|tgzfile] has to be a full path.

Usage

mirror inspect-images [folder|tgzfile] [flags]

Flags

-h, --help help for inspect-images

-i, --ignore-errors ignores errors whiles processing charts. (Exit Code: 2)

-o, --output string choose an output for the list of images.(default "stdout")

• file: outputs all images to a file
• json: outputs all images to a file in JSON format
• skopeo: outputs all images to a file in YAML format to be used as source file with the 'skopeo sync' command. For more information refer skopeo(1).
• stdout: prints all images to standard output
• yaml: outputs all images to a file in YAML format

helm-mirror inspect-images /tmp/helm --output stdout
helm-mirror inspect-images /tmp/helm -o stdout
helm-mirror inspect-images /tmp/helm -o file=filename
helm-mirror inspect-images /tmp/helm -o json=filename.json
helm-mirror inspect-images /tmp/helm -o yaml=filename.yaml
helm-mirror inspect-images /tmp/helm -o skopeo=filename.yaml

Global Flags

-v, --verbose verbose output

version

Displays the current version of mirror.

Install

Using Helm plugin manager (> 2.3.x)

helm plugin install https://github.com/openSUSE/helm-mirror --version master

Test

Clone repository into your $GOPATH. You can also use go get:

go get github.com/openSUSE/helm-mirror

Prerequisites

• Have GO installed.

• We use Go Dep as dependency manager so you'll need that too.

Bootstrap

Get all dependencies running:

dep ensure

Running tests

To run test on this package simply run:

make test

Testing with Docker

make test.unit

Building

Be sure you have all prerequisites, then build the binary by simply running

make mirror

your binary will be stored under bin folder

helm-mirror's Issues

helm-mirror: 2021/05/27 12:50:18.305417 root.go:140: error: not a valid URL protocol: ``

Program does not work:

When running: ./helm-mirror mirror https://charts.rook.io/release charts/ --all-versions the program exits with:

helm-mirror: 2021/05/27 12:50:18.305417 root.go:140: error: not a valid URL protocol: ``
Error: error: not a valid URL protocol

How to use the downloaded index files?

Hi,

Newbie question here.
I was able to compile helm-mirror and download indexes from gitlab chart.
Would someone please tell me how I can install/deploy gitlab chart using those index files?
Reason why I'm trying the mirror, is because after switching to VPN.
I won't be able to download those files.

Thanks,
William

unsupported protocol scheme ""

helm-mirror 0.3.1 version, Linux x86_64

helm-mirror https://helm.myrepo.ru $PWD --new-root-url http://mirror.repo/repository/helm

get.go:90: WARNING: processing chart ... unsupported protocol scheme ""

Is this project still active?

Do not use the API for fetching the tgz

The install-binary.sh fails.

+ local url=https://api.github.com/repos/openSUSE/helm-mirror/releases/latest
+ type curl
++ curl -s https://api.github.com/repos/openSUSE/helm-mirror/releases/latest
++ grep linux
++ awk '/\"browser_download_url\":/{gsub( /[,\"]/,"", $2); print $2}'
+ DOWNLOAD_URL=
+ downloadFile
+ PLUGIN_TMP_FILE=/tmp/helm-mirror.tgz
+ echo 'Downloading '
Downloading 
+ type curl
+ curl -L '' -o /tmp/helm-mirror.tgz
curl: (3) <url> malformed
+ fail_trap
+ result=3
+ '[' 3 '!=' 0 ']'
+ echo 'Failed to install helm-mirror'

This fails because you send a request via the api.github.com. Depending on the organization, there might be that the free limit is already exhausted. In my case:

"API rate limit exceeded for 195.135.221.2. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)"

Read: https://developer.github.com/v3/#rate-limiting

Error: fork/exec /home/pablo/.helm/plugins/helm-mirror/bin/mirror: no such file or directory

I am using ubuntu:
Linux #33~18.04.1-Ubuntu SMP Tue Oct 1 10:20:39 UTC 2019 x86_64 GNU/Linux

I ran:

helm plugin install https://github.com/openSUSE/helm-mirror --version master

Then I tried downloading a package to mirror:

helm mirror "https://kubecost.github.io/cost-analyzer/" "$PWD/tempcharts" --chart-name cost-analyzer --chart-version 1.45.2

And I got back this error:

Error: fork/exec ~/.helm/plugins/helm-mirror/bin/mirror: no such file or directory

Checking folder "~/.helm/plugins/helm-mirror/bin/", I see there's a file "helm-mirror" not "mirror".

If I rename the file manually, the command does not raise any error but I still don't get the package. The only thing downloaded is the index.yaml file.

inspect-images: improve command line options

I find the combination --output and--file-name options a bit inconvenient from the user perspective. E.g. depending on the value of --output, one should or shouldn't specify a --file-name.

I'd suggest suppressing the --file-name option and embed in --output both the format and the optional output file name.

E.g. --output file --file-name filename should become --output file=filename

helm-mirror inspect-images does not detect all images of grafana charts

Hi,

I had an issue when using helm-mirro inspect-images on official grafana charts.
There is the same issue with SUSE Grafana chart.

Some images - like busybox - are not detected.

List repo

$ helm repo list                                                                                                                                                                                  
NAME    URL
stable  https://kubernetes-charts.storage.googleapis.com
local   http://127.0.0.1:8879/charts

Fetch Chart

$ helm fetch stable/grafana

Inspect with helm-mirror

$ helm-mirror inspect-images ${PWD}/grafana-5.0.8.tgz                                                                                                                                              
grafana/grafana:6.6.2
bats/bats:v1.1.0

Inspect with helm

$ helm inspect ./grafana-5.0.8.tgz | grep reposi -A2
  repository: grafana/grafana
  tag: 6.6.2
  pullPolicy: IfNotPresent
--
  repository: curlimages/curl
  tag: 7.68.0
  pullPolicy: IfNotPresent
--
    repository: busybox
    tag: "1.31.1"
    pullPolicy: IfNotPresent

helm plugin install not working .

Below error,
`

• PROJECT_NAME=helm-mirror
• PROJECT_GH=openSUSE/helm-mirror
  ++ helm home --debug=false
• : /root/.helm/plugins/helm-mirror
• type cygpath
• [[ '' == \1 ]]
• trap fail_trap EXIT
• set -e
• initArch
  ++ uname -m
• ARCH=x86_64
• case $ARCH in
• ARCH=amd64
• initOS
  ++ tr '[:upper:]' '[:lower:]'
  +++ uname
  ++ echo Linux
• OS=linux
• case "$OS" in
• verifySupported
• local supported=linux-amd64
• echo linux-amd64
• grep -q linux-amd64
• type curl
• getDownloadURL
  ++ git -C /root/.helm/plugins/helm-mirror describe --tags --exact-match
• local version=
• '[' -n '' ']'
• local url=https://github.com/repos/openSUSE/helm-mirror/releases/latest
• type curl
  ++ curl -s https://github.com/repos/openSUSE/helm-mirror/releases/latest
  ++ grep -o -E 'v.+"'
  ++ awk '{split($0,a,/"/); print a[1]}'
• version=
• DOWNLOAD_URL=https://codeload.github.com/openSUSE/helm-mirror/tar.gz/
• echo 'using download url https://codeload.github.com/openSUSE/helm-mirror/tar.gz/'
  using download url https://codeload.github.com/openSUSE/helm-mirror/tar.gz/
• downloadFile
• PLUGIN_TMP_FILE=/tmp/helm-mirror.tgz
• echo 'Downloading https://codeload.github.com/openSUSE/helm-mirror/tar.gz/'
  Downloading https://codeload.github.com/openSUSE/helm-mirror/tar.gz/
• type curl
• curl -L https://codeload.github.com/openSUSE/helm-mirror/tar.gz/ -o /tmp/helm-mirror.tgz
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  100 21 100 21 0 0 22 0 --:--:-- --:--:-- --:--:-- 22
• installFile
• HELM_TMP=/tmp/helm-mirror
• mkdir -p /tmp/helm-mirror
• tar xf /tmp/helm-mirror.tgz -C /tmp/helm-mirror --strip-components=1
  tar: This does not look like a tar archive

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

• fail_trap
• result=2
• '[' 2 '!=' 0 ']'
• echo 'Failed to install helm-mirror'
  Failed to install helm-mirror
• echo 'For support, go to https://github.com/openSUSE/helm-mirror.'
  For support, go to https://github.com/openSUSE/helm-mirror.
• exit 2
  `

There's no latest tag

Currently:

curl --silent https://github.com/openSUSE/helm-mirror/releases/latest
<html><body>You are being <a href="https://github.com/openSUSE/helm-mirror/releases">redirected</a>.</body></html>

It should be something like that (example is from minikube):

curl --silent https://github.com/kubernetes/minikube/releases/latest
<html><body>You are being <a href="https://github.com/kubernetes/minikube/releases/tag/v0.30.0">redirected</a>.</body></html>

I think this is because it is flagged as pre-release atm, but I am not sure how this works.

Problem with relative paths in index.yaml files

helm-mirror works well if the upstream index.yaml file holds only URLs.
If the index.yaml file holds relative paths ( for example jetstack ) helm-mirror will fail:

  - apiVersion: v1
    appVersion: v0.0.1-alpha.1
    created: "2021-01-25T12:19:54.966Z"
    description: A Helm chart for istio-csr
    digest: 630da21d66e68d4da910e771279548b96e848b8fcbfc66e98b5a953a023b1ff2
    home: https://github.com/jetstack/istio-csr
    maintainers:
    - email: [email protected]
      name: joshvanl
    name: cert-manager-istio-csr
    sources:
    - https://github.com/cert-manager/istio-csr
    type: application
    urls:
    - charts/cert-manager-istio-csr-v0.0.1-alpha.1.tgz

Error Message: Get charts/cert-manager-v0.15-alpha.3.tgz: unsupported protocol scheme ""

plugin install does not work for helm3

I am using helm3.

helm version
version.BuildInfo{Version:"v3.0", GitCommit:"", GitTreeState:"", GoVersion:"go1.13.11"}

Installation error for helm3 plugin installation

helm plugin install https://github.com/openSUSE/helm-mirror --version master
using download url https://github.com/openSUSE/helm-mirror/releases/download/v0.3.1/helm-mirror-linux.tgz
Downloading https://github.com/openSUSE/helm-mirror/releases/download/v0.3.1/helm-mirror-linux.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   634  100   634    0     0    794      0 --:--:-- --:--:-- --:--:--   793
100 6831k  100 6831k    0     0  3159k      0  0:00:02  0:00:02 --:--:--  9.8M
Preparing to install into The Kubernetes package manager

Common actions for Helm:

- helm search:    search for charts
- helm pull:      download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

Environment variables:

+------------------+-----------------------------------------------------------------------------+
| Name             | Description                                                                 |
+------------------+-----------------------------------------------------------------------------+
| $XDG_CACHE_HOME  | set an alternative location for storing cached files.                       |
| $XDG_CONFIG_HOME | set an alternative location for storing Helm configuration.                 |
| $XDG_DATA_HOME   | set an alternative location for storing Helm data.                          |
| $HELM_DRIVER     | set the backend storage driver. Values are: configmap, secret, memory       |
| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.                  |
| $KUBECONFIG      | set an alternative Kubernetes configuration file (default "~/.kube/config") |
+------------------+-----------------------------------------------------------------------------+

Helm stores configuration based on the XDG base directory specification, so

- cached files are stored in $XDG_CACHE_HOME/helm
- configuration is stored in $XDG_CONFIG_HOME/helm
- data is stored in $XDG_DATA_HOME/helm

By default, the default directories depend on the Operating System. The defaults are listed below:

+------------------+---------------------------+--------------------------------+-------------------------+
| Operating System | Cache Path                | Configuration Path             | Data Path               |
+------------------+---------------------------+--------------------------------+-------------------------+
| Linux            | $HOME/.cache/helm         | $HOME/.config/helm             | $HOME/.local/share/helm |
| macOS            | $HOME/Library/Caches/helm | $HOME/Library/Preferences/helm | $HOME/Library/helm      |
| Windows          | %TEMP%\helm               | %APPDATA%\helm                 | %APPDATA%\helm          |
+------------------+---------------------------+--------------------------------+-------------------------+

Usage:
  helm [command]

Available Commands:
  completion  Generate autocompletions script for the specified shell (bash or zsh)
  create      create a new chart with the given name
  dependency  manage a chart's dependencies
  env         Helm client environment information
  get         download extended information of a named release
  help        Help about any command
  history     fetch release history
  install     install a chart
  lint        examines a chart for possible issues
  list        list releases
  mirror      Mirror Helm Charts from a repository into a local folder.
  package     package a chart directory into a chart archive
  plugin      install, list, or uninstall Helm plugins
  pull        download a chart from a repository and (optionally) unpack it in local directory
  repo        add, list, remove, update, and index chart repositories
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  show        show information of a chart
  status      displays the status of the named release
  template    locally render templates
  test        run tests for a release
  uninstall   uninstall a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client version information

Flags:
      --add-dir-header                   If true, adds the file directory to the header
      --alsologtostderr                  log to standard error as well as files
      --debug                            enable verbose output
  -h, --help                             help for helm
      --kube-context string              name of the kubeconfig context to use
      --kubeconfig string                path to the kubeconfig file
      --log-backtrace-at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log-dir string                   If non-empty, write log files in this directory
      --log-file string                  If non-empty, use this log file
      --log-file-max-size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
  -n, --namespace string                 namespace scope for this request (default "default")
      --registry-config string           path to the registry config file (default "/home/sles/.config/helm/registry.json")
      --repository-cache string          path to the file containing cached repository indexes (default "/home/sles/.cache/helm/repository")
      --repository-config string         path to the file containing repository names and URLs (default "/home/sles/.config/helm/repositories.yaml")
      --skip-headers                     If true, avoid header prefixes in the log messages
      --skip-log-headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

Use "helm [command] --help" for more information about a command./plugins/helm-mirror
mkdir: cannot create directory ‘The Kubernetes package manager\n\nCommon actions for Helm:\n\n- helm search:    search for charts\n- helm pull:      download a chart to your local directory to view\n- helm install:   upload the chart to Kubernetes\n- helm list:      list releases of charts\n\nEnvironment variables:\n\n+------------------+-----------------------------------------------------------------------------+\n| Name             | Description                                                                 |\n+------------------+-----------------------------------------------------------------------------+\n| $XDG_CACHE_HOME  | set an alternative location for storing cached files.                       |\n| $XDG_CONFIG_HOME | set an alternative location for storing Helm configuration.                 |\n| $XDG_DATA_HOME   | set an alternative location for storing Helm data.                          |\n| $HELM_DRIVER     | set the backend storage driver. Values are: configmap, secret, memory       |\n| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.                  |\n| $KUBECONFIG      | set an alternative Kubernetes configuration file (default "~’: File name too long
Failed to install helm-mirror
For support, go to https://github.com/openSUSE/helm-mirror.
Error: plugin install hook for "mirror" exited with error

After the installation error

helm mirror
Error: fork/exec /home/sles/.local/share/helm/plugins/helm-mirror/bin/mirror: no such file or directory

When I tried to use plugin install with helm2, it worked as expected. But with helm3, I encountered the above error.

No prebuild binary for macos-amd64

Hey Ho,

I've tried to install this plugin on macOS 10.14.6: helm plugin install https://github.com/openSUSE/helm-mirror --version v0.2.4

This command produces the following:

No prebuild binary for macos-amd64.
Failed to install helm-mirror
For support, go to https://github.com/openSUSE/helm-mirror.
Error: plugin install hook for "mirror" exited with error

Helm Version:

Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

inspect images: add skopeo sync compatible output

Add a new output format for the inspect images command. This should print a yaml file compatible with what skopeo sync expects.

For example, it should provide something like that:

registry.suse.com:
  images:
     nginx-ingress-controller:
        - 0.15.0
     foobar:
        - tag1
        - tag2
        - tag3

Assuming that registry.suse.com/nginx-ingress-controller:0.15.0, registry.suse.com/foobar:tag1, registry.suse.com/foobar:tag2 and registry.suse.com/foobar:tag3 are mentioned inside of the helm charts.