Giter Site home page Giter Site logo

Implement reindex operation about helm-s3 HOT 13 CLOSED

hypnoglow avatar hypnoglow commented on August 15, 2024
Implement reindex operation

from helm-s3.

Comments (13)

Stono avatar Stono commented on August 15, 2024

Hey,
Just adding my +1 to this, I was just about to raise an issue around concurrency issues we're having when multiple pipelines happen to index at the same time.

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

Hey folks, I rolled out reindex command in 0.5.0

Let me know if it works well or not.

from helm-s3.

markhu avatar markhu commented on August 15, 2024

Got this error:

$ helm s3 reindex redacted
Warning: reindex feature is in beta. If you experience any issues,
please provide your feedback here: https://github.com/hypnoglow/helm-s3/issues/22

2018/01/09 12:43:22 traverse the chart repository: load archive from s3 object: gzip: invalid header
Error: plugin "s3" exited with error

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

@markhu , seems like the file was processed is not a helm chart. Do you have any other files in your repository directory, besides index.yaml and chart files?

from helm-s3.

markhu avatar markhu commented on August 15, 2024

Um, you mean like a .DS_Store file?!? (I swear it wasn't me, it was my predecessor!) :)
But I also found sub-directories. Which is probably why I got these next errors:

2018/01/09 14:03:03 traverse the chart repository: load archive from s3 object: chart metadata (Chart.yaml) missing

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

Directories should be ignored. I guess the plugin should print some additional debug info like a file name.

Also, this is also an interesting case - should the plugin fail on such error, or just output the warning and keep working? Maybe we can even control the behavior with some cli flag, so a user can choose.

from helm-s3.

AndriiHryshaienko avatar AndriiHryshaienko commented on August 15, 2024

Hi, after reindex repo index.yaml is empty now:

apiVersion: v1
entries: {}
generated: 2018-03-09T14:59:17.746753+02:00

from helm-s3.

Evesy avatar Evesy commented on August 15, 2024

Hi,

FWIW we've observed some timeouts with the reindex command after ~8 minutes:

upload index to the repository: upload index to S3 bucket: RequestCanceled: request context canceled
caused by: context deadline exceeded
Error: plugin "s3" exited with error

This is in a repo with ~5000 charts, so we may have just reached a limit on how many charts can be indexed in a reasonable amount of time.

from helm-s3.

kfiles avatar kfiles commented on August 15, 2024

My experience is the same as @AndriiHryshaienko. No matter how many times I ran helm s3 reindex myrepo, I just had an empty index.yaml. I was forced to do an s3 cp of all my charts to a local dir, do helm repo index localdir and then fix the urls and upload the index.yaml back to s3. Ugh.

from helm-s3.

marcin-je avatar marcin-je commented on August 15, 2024

@AndriiHryshaienko I got same issue as @kfiles

Seems the issue happens when

helm s3 init s3://sample-bucket/charts
helm repo add myrepo s3://sample-bucket/charts

But not when:

helm s3 init s3://sample-bucket
helm repo add myrepo s3://sample-bucket

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

@Evesy

I think the simplest workaround here is to add --timeout flag, so you can specify how long are you willing to wait for reindexing your repo.

I'm adding --timeout support right away: #39

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

@AndriiHryshaienko @kfiles @marcin-je

This is definitely a bug. Thanks to @marcin-je pointed me out that it occurs when your repository resides within bucket directory, but not bucket root.

Fix: #40
Will be released soon in v0.6.1

from helm-s3.

hypnoglow avatar hypnoglow commented on August 15, 2024

https://github.com/hypnoglow/helm-s3/blob/master/CHANGELOG.md#changed

Moved helm s3 reindex command out of beta, as it seems there are no more issues related to it.

from helm-s3.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.