barolab/verdaccio-minio

A verdaccio plugin for storing data in Minio

Verdaccio - Minio

A verdaccio plugin for storing data in Minio.

Usage

You can use this plugin by installing it globally:

# Install the package globally
$ yarn global add verdaccio-minio

# Print the directory for your global packages
$ yarn global dir
/usr/local/share/.config/yarn/global

# Create a symbolic link to your package directory in the verdaccio plugin folder
$ ln -s /usr/local/share/.config/yarn/global/node_modules/verdaccio-minio /verdaccio/plugins/verdaccio-minio

Then you'll need to provide a configuration file for the minio storage :

# This points to the plugin folder above
plugins: /verdaccio/plugins

# This is mandatory, otherwise verdaccio won't boot
storage: /verdaccio/storage/data

# Here's the plugin configuration option
store:
  minio:
    # The HTTP port of your minio instance
    port: 9000

    # The endpoint on which verdaccio will access minio (without scheme)
    endPoint: minio.minio.svc.cluster.local

    # The minio access key
    accessKey: this-is-not-so-secret

    # The minio secret key
    secretKey: this-is-not-so-secret

    # Disable SSL if you're accessing minio directly through HTTP
    useSSL: false

    # The region used by your minio instance (optional, default to "us-east-1")
    region: eu-west-1

    # A bucket where verdaccio will store it's database & packages (optional, default to "verdaccio")
    bucket: 'npm'

    # Number of retry when a request to minio fails (optional, default to 10)
    retries: 3

    # Delay between retries (optional, default to 100)
    delay: 50
# The rest of the verdaccio configuration

Once this is done you can start your Verdaccio instance, and check minio to see that the bucket as been created automatically

Docker

A docker image is also available for easy deployments. With docker compose :

version: '3.7'

services:
  verdaccio:
    image: barolab/verdaccio
    ports:
      - 4873:4873
    volumes:
      - ./config.yaml:/verdaccio/conf/config.yaml
      - ./htpasswd:/verdaccio/storage/htpasswd
    depends_on:
      - minio
    environment:
      VERDACCIO_PROTOCOL: http
      VERDACCIO_PORT: 4873

  minio:
    image: minio/minio:RELEASE.2020-02-07T23-28-16Z
    command: server /data
    volumes:
      - minio:/data
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY: this-is-not-so-secret
      MINIO_SECRET_KEY: this-is-not-so-secret

volumes: minio:

You'll need to write your config file and your htpasswd for the above to work.

If you'd like to check K8S resources I highly recommend you look at the Minio and Verdaccio Helm charts.

Contributing

It's highly recommended that you install the git hooks in your workstation before committing anything to this repository. You can run yarn hooks to install them. There's some documentation for contributors that you should read first :

• Code of Conduct
• Contribution Guide

You'll need docker & yarn for a better development experience with this module. You can run yarn start to start a minio & verdaccio containers on ports 9000 and 4873. Then using the /example folder you can install dependencies using verdaccio as a proxy.

verdaccio-minio's Issues

Packages files (tgz) get written with 0 bytes on minio

Hello,

I tried to use this plugin, although I wouldn't know if this is a minio-js issue, this plugin's or a combination of both.

Tested with the docker compose example on the npmjs site at https://www.npmjs.com/package/verdaccio-minio

Also tested with a custom verdaccio image with npm install verdaccio-minio beforehand (using verdaccio 4.2), got same results (also tested with 4.3.5).

The package.json files are written OK, but the published tgz files are put on minio with 0 bytes

Also tried (somehow) with the docker-compose-4.3.yml example on this same repo, but verdaccio 4.3 doesn't even detect the plugin.

PS: I forgot to mention, used the same configuration example as shown in the README.

PS n°2: I tested proxying lodash from npmjs repo, the tgz file was created correctly, so I guess is a publish issue.

Fix plugin version

The version string in the MinioDatabase class is not in sync with the version number in the package.json.

verdaccio looks for a plugin named minio-storage, while the NPM package is minio

verdaccio-minio/src/index.ts

Line 37 in fa3df90

this.config = { ...config.store['minio-storage'] };

This causes Verdaccio to fail if config.yaml looks like this:

store:
   minio-storage:

Conversely, the plugin fails if config.yaml looks like this:

store:
   minio:

Update to verdaccio 4.3.4

Update dependencies

Search function failure

Describe the bug

I use verdaccio-minio as a reference when implementing verdaccio-redis-storage, and I find a bug of the search API.

verdaccio-minio/src/index.ts

Line 97 in 83912ae

private async stat(name: string, onPackage: Callback): Promise<void> {

  public search(onPackage: Callback, onEnd: Callback, validate: (name: string) => boolean): void {
    this.retry(() => this.db.search(validate))
      .then((results) => Promise.all(results.map((r) => this.stat(r, onPackage))))
      .then(() => onEnd(null))
      .catch((error) => onEnd(error));
  }

...

  private async stat(name: string, onPackage: Callback): Promise<void> {
    const stat = await this.retry(() => this.client.stat(name));
    onPackage(stat); <------
  }

The onPackage(stat) is incorrect. The API is onPackage(stat, callback) (see _searchEachPackage below). Because it has a callback, it needs promisify.

https://github.com/verdaccio/verdaccio/blob/2a2ba63476e6babe05d5d7a3ce54d506e0c6ac18/src/lib/local-storage.ts#L613

  public search(startKey: string, options: any): IReadTarball {
    const stream = new ReadTarball({ objectMode: true });

    this._searchEachPackage(
      (item: Package, cb: CallbackAction): void => {
        // @ts-ignore
        if (item.time > parseInt(startKey, 10)) {
          this.getPackageMetadata(item.name, (err: VerdaccioError, data: Package): void => {
            if (err) {
              return cb(err);
            }

            // @ts-ignore
            const time = new Date(item.time).toISOString();
            const result = prepareSearchPackage(data, time);
            if (_.isNil(result) === false) {
              stream.push(result);
            }
            cb(null);
          });
        } else {
          cb(null);
        }
      },
      function onEnd(err): void {
        if (err) {
          stream.emit('error', err);
          return;
        }
        stream.end();
      }
    );

    return stream;
  }

The unit test hides the issue because the mocked onPackage method is just a mock.

verdaccio-minio/src/index.test.ts

Line 75 in 83912ae

const onPackage = jest.fn();

To Fix

  private async stat(name: string, onPackage: Callback): Promise<void> {
    const stat = await this.retry(() => this.client.stat(name));
    return new Promise((resolve, reject) => {
      onPackage(stat, err => {
        if (err) reject(err);
        else resolve();
      });
    });
  }

Reference https://github.com/openupm/verdaccio-redis-storage/blob/611076f001bbaa91837bbc1f589ed5f7eb447e9a/src/plugin.ts#L82

Unable to initialize client

When using this plugin, I get the following error:

verdaccio    |  warn --- config file  - /verdaccio/conf/config.yaml
verdaccio    |  warn --- Plugin successfully loaded: verdaccio-minio-storage
verdaccio    |  debug-=- [Minio] Loading current secret
verdaccio    |  debug-=- [Minio] Loading database db.json from cache
verdaccio    |  debug-=- [Minio] Failed to ensure bucket npm exist, { Error: socket hang up
verdaccio    |     at createHangUpError (_http_client.js:323:15)
verdaccio    |     at Socket.socketOnEnd (_http_client.js:426:23)
verdaccio    |     at Socket.emit (events.js:203:15)
verdaccio    |     at endReadableNT (_stream_readable.js:1145:12)
verdaccio    |     at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }
verdaccio    |  error-=- There was an error initializing client: Error: Failed to ensure bucket npm exist: socket hang up
verdaccio    |     at Client.initialize (/opt/verdaccio/node_modules/verdaccio-minio/lib/client.js:84:13)
verdaccio    |     at process._tickCallback (internal/process/next_tick.js:68:7)
verdaccio    |  debug-=- [Minio] Failed to load database from remote storage, { Error: socket hang up
verdaccio    |     at createHangUpError (_http_client.js:323:15)
verdaccio    |     at Socket.socketOnEnd (_http_client.js:426:23)
verdaccio    |     at Socket.emit (events.js:203:15)
verdaccio    |     at endReadableNT (_stream_readable.js:1145:12)
verdaccio    |     at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }

As you can see I'm running this in docker-compose, but attempting to connect to an internally hosted minio cluster.

Relevant config:
verdaccio: 4.2.2
verdaccio-minio: 0.1.5
minio: RELEASE.2019-04-09T01-22-30Z

config.yaml:

plugins: /verdaccio/plugins
store:
  minio-storage:
    port: 443
    endPoint: minio-dev.backends.tech
    accessKey: access_key
    secretKey: secret_key
    useSSL: false
    region: us-east-1
    bucket: 'npm'
    retries: 3
    delay: 50

Looking at the minio logs during startup, I see no indication of authentication issues but I do see active connections from the docker host running verdaccio.

This plugin is not cluster-ready

I was reviewing the code and found out that the plugin is handling a cached db in memory.

This is a problem for clustering, it defeats the purpose of using an external storage like minio, is like using the local storage feature of verdaccio.

I was going to propose the removal of this cache, the aws-s3 plugin always reads and writes the db file.

https://github.com/verdaccio/monorepo/blob/master/plugins/aws-s3-storage/src/index.ts

I will submit a PR soon for this. If you don't beat me to it by checking the #12