jenkins-x/bucketrepo

An artifacts repository using cloud storage (a lightweight replacement for Nexus)

bucketrepo's Issues

Error when downloading the file: writing file to cache: mkdir : no such file or directory

with no tekton pipeline changes, my release started failing today with

about to run: helm repo add 0 https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts in dir charts/kafka-cluster
Error: looks like "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts" is not a valid chart repository or cannot be reached: Get "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts/index.yaml": dial tcp 10.141.120.2:443: connect: connection timed out
error: failed to add repository: failed to run 'helm repo add 0 https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts' command in directory 'charts/kafka-cluster', output: 'Error: looks like "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts" is not a valid chart repository or cannot be reached: Get "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts/index.yaml": dial tcp 10.143.120.2:443: connect: connection timed out'
�[31m
Pipeline failed on stage 'from-build-pack' : container 'step-build-helm-build'. The execution of the pipeline has stopped.�[0m

the linked url and the bucketrepo logs display this error

  Error when downloading the file: writing file to cache: mkdir : no such file or directory

bucketrepo-bucketrepo-7d89dccf64-2ttjx bucketrepo {"timestamp":"2022-08-04T20:04:28Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"Error when downloading the file: writing file to cache: mkdir : no such file or directory","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":98,"functionName":"(*FileController).GetFile"}}}

bucketrepo does not work well with GCS

Everytime my bucketrepo pod restarts, I get an empty index.yaml

I am seeing this error in my logs:

{"timestamp":"2020-02-06T14:10:51Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to update cache for /tmp/bucketrepo/charts/index.yaml: reading file from cloud storage: blob (key \"tmp/bucketrepo/charts/index.yaml\") (code=NotFound): storage: object doesn't exist","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":295,"functionName":"(*FileController).chartReindex"}}}

It appears that bucketrepo is using the "cache.BaseDir" prefix in the cloud path as well. Whereas while updating the chart it uses the path without "cache.BaseDir".

As a temporary fix, I created a gs://repos/tmp/bucketrepo/charts/index.yaml with the expected contents manually, which results in bucket repo happily working with it, and updating the gs://repos/charts/index.yaml.

I am taking a copy of gs://repos/charts/index.yaml and pasting it to gs://repos/tmp/bucketrepo/charts/index.yaml every few days for now.

Implement latest api call

In many cases, especially with helm scenario in JX preview environments, we need to update a depedency in requirements file with the latest chart version of an app since it does not implement it.

with chart museum we used to invoke:
curl http://jenkins-x-chartmuseum:8080/api/charts/<ChartName>/latest --silent | jq -r .version

its true that helm search repo/chart -ojson would do kind of the same but this acts on the client side i.e. helm client interaction with the cached index file. in our case its not enough because latest JX builders include helm 2.14 which does not have -o/--output flag implemented.

we need the same with bucketrepo so we can use it.

Error when upgrading version stream

I just tried upgrading version stream from v1.0.399 to v1.0.453

In that set of changes, bucketrepo has gone from 0.1.35 to 0.1.38

The pipeline failed, as well as running jx boot when patching bucketrepo.

Error from pipeline:

Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.38\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"**\",\"jenkins.io/version\":\"61\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"**\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"env\":{},\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.38\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"},"labels":{"chart":"bucketrepo-0.1.38","jenkins.io/version":"61"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"bucketrepo"}],"containers":[{"env":{},"image":"gcr.io/jenkinsxio/bucketrepo:0.1.38","name":"bucketrepo"}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "jenkins-x-bucketrepo", Namespace: "**"
Object: &{map["apiVersion":"apps/v1" "metadata":map["annotations":map["deployment.kubernetes.io/revision":"3" "jenkins.io/chart":"env" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.35\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"**\",\"jenkins.io/version\":\"60\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"**\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.35\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"] "name":"jenkins-x-bucketrepo" "namespace":"**" "resourceVersion":"31907518" "generation":'7' "creationTimestamp":"2020-03-11T03:25:01Z" "labels":map["chart":"bucketrepo-0.1.35" "draft":"draft-app" "jenkins.io/chart-release":"jenkins-x" "jenkins.io/namespace":"**" "jenkins.io/version":"60"] "selfLink":"/apis/apps/v1/namespaces/**/deployments/jenkins-x-bucketrepo" "uid":"6d57ec82-3ddf-4ddf-8352-23401d98411f"] "spec":map["replicas":'\x01' "selector":map["matchLabels":map["app":"jenkins-x-bucketrepo"]] "template":map["metadata":map["creationTimestamp":<nil> "labels":map["app":"jenkins-x-bucketrepo" "draft":"draft-app"] "annotations":map["checksum/config":"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c"]] "spec":map["terminationGracePeriodSeconds":'\n' "dnsPolicy":"ClusterFirst" "serviceAccountName":"jenkins-x-bucketrepo" "securityContext":map[] "restartPolicy":"Always" "containers":[map["readinessProbe":map["timeoutSeconds":'\x01' "periodSeconds":'\n' "successThreshold":'\x01' "failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"]] "terminationMessagePolicy":"File" "imagePullPolicy":"IfNotPresent" "name":"bucketrepo" "command":["/bucketrepo"] "ports":[map["protocol":"TCP" "containerPort":'\u1f90']] "volumeMounts":[map["name":"config" "readOnly":%!q(bool=true) "mountPath":"/config"]] "livenessProbe":map["httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "initialDelaySeconds":'<' "timeoutSeconds":'\x01' "periodSeconds":'\n' "successThreshold":'\x01' "failureThreshold":'\x03'] "terminationMessagePath":"/dev/termination-log" "image":"gcr.io/jenkinsxio/bucketrepo:0.1.35" "args":["-config-path=/config" "-log-level=info"] "resources":map["limits":map["cpu":"100m" "memory":"256Mi"] "requests":map["cpu":"80m" "memory":"128Mi"]]]] "serviceAccount":"jenkins-x-bucketrepo" "schedulerName":"default-scheduler" "volumes":[map["name":"config" "secret":map["secretName":"bucketrepo-config" "defaultMode":'\u01a4']]]]] "strategy":map["type":"RollingUpdate" "rollingUpdate":map["maxUnavailable":"25%" "maxSurge":"25%"]] "revisionHistoryLimit":'\n' "progressDeadlineSeconds":'\u0258'] "status":map["observedGeneration":'7' "replicas":'\x01' "updatedReplicas":'\x01' "readyReplicas":'\x01' "availableReplicas":'\x01' "conditions":[map["lastUpdateTime"
:"2020-03-19T18:17:53Z" "lastTransitionTime":"2020-03-11T03:25:01Z" "reason":"NewReplicaSetAvailable" "message":"ReplicaSet \"jenkins-x-bucketrepo-98667859f\" has successfully progressed." "type":"Progressing" "status":"True"] map["lastTransitionTime":"2020-05-12T15:16:14Z" "reason":"MinimumReplicasAvailable" "message":"Deployment has minimum availability." "type":"Available" "status":"True" "lastUpdateTime":"2020-05-12T15:16:14Z"]]] "kind":"Deployment"]}
for: "/tmp/helm-template-workdir-790200993/jenkins-x/output/namespaces/**/env/charts/bucketrepo/templates/part0-deployment.yaml": cannot restore slice from map'

Error from jx boot:

Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.38\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"jx\",\"jenkins.io/version\":\"1\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"jx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"env\":{},\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.38\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"},"labels":{"chart":"bucketrepo-0.1.38","jenkins.io/version":"1"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"bucketrepo"}],"containers":[{"env":{},"image":"gcr.io/jenkinsxio/bucketrepo:0.1.38","name":"bucketrepo"}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "jenkins-x-bucketrepo", Namespace: "jx"
Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["deployment.kubernetes.io/revision":"3" "jenkins.io/chart":"env" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.35\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"jx\",\"jenkins.io/version\":\"60\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"jx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.35\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"] "creationTimestamp":"2020-03-11T03:25:01Z" "generation":'7' "labels":map["chart":"bucketrepo-0.1.35" "draft":"draft-app" "jenkins.io/chart-release":"jenkins-x" "jenkins.io/namespace":"jx" "jenkins.io/version":"60"] "name":"jenkins-x-bucketrepo" "namespace":"jx" "resourceVersion":"31924380" "selfLink":"/apis/apps/v1/namespaces/jx/deployments/jenkins-x-bucketrepo" "uid":"6d57ec82-3ddf-4ddf-8352-23401d98411f"] "spec":map["progressDeadlineSeconds":'\u0258' "replicas":'\x01' "revisionHistoryLimit":'\n' "selector":map["matchLabels":map["app":"jenkins-x-bucketrepo"]] "strategy":map["rollingUpdate":map["maxSurge":"25%" "maxUnavailable":"25%"] "type":"RollingUpdate"] "template":map["metadata":map["annotations":map["checksum/config":"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c"] "creationTimestamp":<nil> "labels":map["app":"jenkins-x-bucketrepo" "draft":"draft-app"]] "spec":map["containers":[map["args":["-config-path=/config" "-log-level=info"] "command":["/bucketrepo"] "image":"gcr.io/jenkinsxio/bucketrepo:0.1.35" "imagePullPolicy":"IfNotPresent" "livenessProbe":map["failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "initialDelaySeconds":'<' "periodSeconds":'\n' "successThreshold":'\x01' "timeoutSeconds":'\x01'] "name":"bucketrepo" "ports":[map["containerPort":'\u1f90' "protocol":"TCP"]] "readinessProbe":map["failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "periodSeconds":'\n' "successThreshold":'\x01' "timeoutSeconds":'\x01'] "resources":map["limits":map["cpu":"100m" "memory":"256Mi"] "requests":map["cpu":"80m" "memory":"128Mi"]] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File" "volumeMounts":[map["mountPath":"/config" "name":"config" "readOnly":%!q(bool=true)]]]] "dnsPolicy":"ClusterFirst" "restartPolicy":"Always" "schedulerName":"default-scheduler" "securityContext":map[] "serviceAccount":"jenkins-x-bucketrepo" "serviceAccountName":"jenkins-x-bucketrepo" "terminationGracePeriodSeconds":'\n' "volumes":[map["name":"config" "secret":map["defaultMode":'\u01a4' "secretName":"bucketrepo-config"]]]]]] "status":map["availableReplicas":'\x01' "conditions":[map["lastTransitionTime":"2020-03-11T03:25:01Z" "lastUpdateTime":"2020-03-19T18:17:53Z" "message":"ReplicaSet \"jenkins-x-bucketrepo-98667859f\" has successfully progressed." "reason":"NewReplicaSetAvailable" "status":"True" "type":"Progressing"] map["lastTransitionTime":"2020-05-12T15:52:20Z" "lastUpdateTime":"2020-05-12T15:52:20Z" "message":"Deployment has minimum availability." "reason":"MinimumReplicasAvailable" "status":"True" "type":"Available"]] "observedGeneration":'7' "readyReplicas":'\x01' "replicas":'\x01' "updatedReplicas":'\x01']]}
for: "/var/folders/zc/g4gbg59j2cg_fq6vf75mbkyc0000gn/T/helm-template-workdir-569312494/jenkins-x/output/namespaces/jx/env/charts/bucketrepo/templates/part0-deployment.yaml": cannot restore slice from map'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step helm apply --boot --remote --name jenkins-x --provider-values-dir ../kubeProviders' command in directory 'env', output: ''

Bucketrepo with S3 gives error on cache operations

bucket repo is set with s3 (via JX).
charts are stored properly on the bucket but regardless the bucketrepo has errors on cache like the following:
{"timestamp":"2020-04-10T16:53:12Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to update cache for /tmp/bucketrepo/charts/index.yaml: reading file from cloud storage: blob (key \"tmp/bucketrepo/charts/index.yaml\") (code=NotFound): NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 7FB39D20F87ACD3B, host id: n5DXh6s30meT+d4lOiCAGu9mIpzK+BxxZJJbZVVDKSCA7sg0NxeU9W63mf0fexc9R3je5mQPxNA=","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":295,"functionName":"(*FileController).chartReindex"}}}

it seems that bucket repo treats /tmp/bucketrepo (default cache dir) as cloud storage (s3) and not local dir.

Instructions

Hi there, app sounds promising, but having trouble running this on docker.

Goal is to run this with my teamcity/maven infra and artifacts should end up in the bucket.

I specifically grabbed "gcr.io/jenkinsxio/bucketrepo:0.1.38", assigned a persistent volume "bucketrepo_config -> /config"

Added the config path parameter as "/config". On first launch there is config.yaml in the root. Just in case there's confusion about what "/" means, i copied it to /config also using the console shell. Restarted.

All i see in the logs is:

{"timestamp":"2020-05-11T16:28:28Z","message":"Reading config file","severity":"WARNING","context":{"data":{"error":{}}}}

Bucketrepo does not support apiVersion: v2 of helm charts

in v2, requirement are put directly in Chart.yaml (requirements.yaml is still supported) maybe we can add support for this.

KInldy provide more examples for self-hosted cluster

Great work but need working examples configurations for self-hosted K#S clusters:3

1. In case we are on k3s, how do we configure to use a storage bucket from GCS or S3? I see references to jx-requirements.yaml's storage section, but how does auth work in that case without the cluster itself running on GKE?
2. How do we use Minio? Lets say we configured minio backed by GCS, how does one authenticate with Minio?

Wrong error handling preventing clean downloads

After d0458f7 I get an error message like ERRO[0002] Error when serving the file from cache: reading file from cache: <nil> for every download when actually no real error happened. Since the error message is also injected at the end of the HTTP response e.g. Maven reports numerous checksum mismatch errors.

Why does it happen? The above mentioned commit d0458f7 changed the semantics with the error mapping: places that before returned nil in case of no error now always return some non-nil wrapped value.

This happens at least due to d0458f7#diff-f7ebf5bb6047d56d88d9243d98113c4fL134

All charts wiped out each release, PVC support needed

Each time bucketrepo pod recreates it wiped out all the charts and that break jx-git-operator functionality

Local Cache in Container Never Synced with S3 Bucket

I've deployed bucketrepo alongside an S3 Bucket, but no what config settings I try, the updateCloudStorage function never gets called and my bucket remains empty forever.

Config:

http:
  addr: ':8080'
  username: ''
  password: ''
storage:
  enabled: true
  bucket_url: 's3://my-cache-bucket'
cache:
  base_dir: '/tmp/bucketrepo'
repositories:
- url: https://myorg.jfrog.io/artifactory/maven
  header:
    Authorization: 'Bearer <token>'

When I request an artifact I get the following debug logs on the first try:

"GetFile, filename: /com/myorg/package1/1/package1-1.pom"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"
"Read file from repository: /com/myorg/package1/1/package1-1.pom"
"Trying to download from repository: https://myorg.jfrog.io/artifactory/maven"
"Write file to cache: /com/myorg/package1/1.187/package1-1.pom\n"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"

Subsequent requests generate the following:

"GetFile, filename: /com/myorg/package1/1/package1-1.pom"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"

There's no mention of the debug line "Write file to cloud storage: <file>" which would indicate that it's never trying to take any of the files it pulled from the repo's to the cloud-cache.

Am I misunderstanding something?

support s3 bucket encrypted with kms

Relevant google/go-cloud#2802

use a version of go-cloud that supports AWS IRSA

Currently bucketrepo does not seem to work with IRSA for AWS s3

environment variables in config.yaml

I have BUCKET_HOST, BUCKET_NAME, BUCKET_PORT and BUCKET_REGION environment variables set.
Would it be possible to use then in the config.yaml in .storage.bucket_url. For example like this "s3://${BUCKET_NAME}?endpoint=${BUCKET_HOST}:${BUCKET_PORT}&region=${BUCKET_REGION}&s3ForcePathStyle=true"

These variables are created automatically by my "ObjectBucketClaim" and bucket name is a unique automatic generated id, so it's hard to know before you deploy it.

for AWS s3 access we may need to set AWS_REGION

dockerhub image is not accessible

Trying to use the suggested command

docker run -v $(pwd)/config:/config -p 8080:8080 jenkinsxio/bucketrepo -config-path=/config

fails with

docker: Error response from daemon: pull access denied for jenkinsxio/bucketrepo, repository does not exist or may require 'docker login'.

Bucketrepo resets index.yaml content on s3 on startup

Interacting with bucketrepo for charts works. charts are being pushed, index.yaml is being updated (you can see both via helm search and via s3 file content).
if deleting the pod, after the new pod coming up the index.yaml is being reset with entries: {} field being empty - also helm search return no results.

Expected behaviour: bucketrepo picks up the index.yaml from s3 and continuous from the same state.

Does this project support hosting apt repos?

Thanks!

run jx3 in minicube visit bucketrepo的index.yaml return 404

when i see the log of bucketrepo，i get this warning:
reading file from cloud storage: no cloud storage available

Does bucketrepo support tar archives ?

I want to host tar and zip archives where the backend storage is AWS S3. I tried using nexus but it consumes a lot of resources.
As of now I have no plans of hosting all types of archives. Only looking at simple tar.gz and zip archives.
Does bucketrepo support them and can the repo be private so that only people with right credentials can access them ?

Latest release (0.1.41) doesn't work

Error: failed to start container "bucketrepo": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bucketrepo\": stat /bucketrepo: no such file or directory": unknown

From here:
https://github.com/jenkins-x/bucketrepo/blob/master/charts/bucketrepo/templates/deployment.yaml#L28

which makes sense cause it was removed here:
a2fb727

Newest release errors with previous cache dir settings

The newest release is not backwards compatible.

Setting the following settings was required for bucketrepo to not delete all of your charts each time it started:

config:
  cache:
    dir: ./

This now results in the following error:

➜  ~ k logs jenkins-x-bucketrepo-7bb5944448-bzg4n
{"timestamp":"2020-10-06T14:51:14Z","message":"Configuration: http:\n  address: :8080\n  https: false\n  certificate: \"\"\n  key: \"\"\n  username: xxx\n  password: xxx\n  chartpath: charts\nstorage:\n  enabled: true\n  bucketurl: s3://repository-20200702162217040100000006\n  timeout: 5m0s\n  prefix: \"\"\ncache:\n  basedir: ./\nrepositories:\n- url: https://repo.maven.org/maven2\n  timeout: 1m0s\n  header: {}\n- url: https://repo1.maven.org/maven2\n  timeout: 1m0s\n  header: {}\n- url: https://services.gradle.org/distributions/\n  timeout: 1m0s\n  header: {}\n- url: https://plugins.gradle.org/m2/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.jenkins-ci.org/public/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.jenkins-ci.org/releases/\n  timeout: 1m0s\n  header: {}\n- url: https://jitpack.io/\n  timeout: 1m0s\n  header: {}\n- url: https://registry.npmjs.org/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.spring.io/milestone/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.spring.io/release/\n  timeout: 1m0s\n  header: {}\n- url: http://uk.maven.org/maven2/\n  timeout: 1m0s\n  header: {}\n","severity":"INFO","context":{}}
{"timestamp":"2020-10-06T14:51:14Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to initialise controller: failed to create charts dir charts: mkdir charts: permission denied","severity":"CRITICAL","context":{"reportLocation":{"filePath":"internal/main.go","lineNumber":39,"functionName":"main"}}}