opencomponents/storage-adapters

📦 Storage-adapters & utilities for the OC registry

Storage-adapters

Storage-adapters & utilties for the OC registry

Adapters

Type	Name	Version
S3	oc-s3-storage-adapter
Google Storage	oc-gs-storage-adapter
⚠️ WIP! Riak Storage	oc-riak-storage-adapter
Azure Blob Storage	oc-azure-storage-adapter

Other packages included in this repo

Name	Version
oc-storage-adapters-utils

storage-adapters's Issues

Add license files to each package

[DX-366] Wrong protocol for oc-client.min.js when requested in registry web interface

If the registry is communicating with the storage with plain http instead of https, the request for oc-client.min.js is still using the https protocol.

Of course, the storage should use a SSL/TLS encryption when exposed publicly, but in this case I'm running a registry locally. The registry runs on http://localhost:3333 and the storage on http://localhost:8080. When accessing the registry in the browser (http://localhost:3333/) there is a request in the browser for https://localhost:8080/bucket/components/oc-client/0.42.16/src/oc-client.min.js. Notice the https protocol. As the storage isn't exposed with https locally that yields an error.

The problem seems to be in /oc/src/registry/domain/repository.js, where the protocol https is hardcoded in getComponentPath, getStaticClientPathand and getStaticClientMapPath.

The configuration used for the registry looks like this:

let configuration = {
  verbosity: 5,
  baseUrl: 'http://localhost:3333',
  port: 3333,
  tempDir: './temp/',
  refreshInterval: 600,
  pollingInterval: 5,
  storage: {
    adapter: riak,
    options: {
      key: '...',
      secret: '...',
      bucket: 'foo',
      region: 'us-east-1',
      componentsDir: 'components',
      signatureVersion: 'v2', 
      sslEnabled: false,
      path: '//localhost:8080/foo/', 
      s3ForcePathStyle: true, 
      debug: true, 
      endpoint: {
        protocol: 'http',
        hostname: 'localhost',
        port: '8080'
      }  
    }    
  },
  env: { name: 'production' }
};

Fix setting server.js as public when publishing from Windows machines

Whenever we publish components from windows machines server.js files will be marked as public because of node treating '' and '/' as different characters (which they are :P ) and storage adapters checking for only one variant

File streams handling improvement for oc-azure-storage-adapter

Implementing workaround/hack for the private/public file permissions (described in #61) have had a side effect when putting file content when content is provided as a stream.

After we have written the stream to private container, the stream is at its end. I have not been able to figure out how to reset this stream to beginning for writes to public container, I've tried the following:

• use fs.read with empty buffer, read length of 0 and position set to 0 in an attempt to reset the stream
  • this had no effect on stream whatsoever
• pipe through azure write stream
  • as one could expect azure write stream is not pass-through, so any further reads would basically hang

In order to be able to write the actual content to public container after using up original stream with file content, I set up additional stream from azure private container to azure public container. Even though it puts ugly and unnecessary strain on the network I opted in to do this because:

• I couldn't solve original issue
• I did not want to load files into memory, that would put unnecessary memory pressure on registry and I think it could potentially block event loop

How to resolve this issue

Figure out a way to either:
a) Solve #61
b) Reset the stream to it's beginning so that it can be reused

Fix 'double-storage' workaround in oc-azure-blob-storage

Azure Blob Storage handles ACL in a different way than S3 - it does not allow setting permissions per file/blob, but rather per container (something containing multiple blobs).

To that extent we will need to create 2 containers in Azure, one for public files and one for private files.

The only problem with that approach is the fact that oc informs storage adapter if the file is public or private only when writing, but not when reading a file - so we would not know from which container to read the file. While we could extend oc to do that, in order to get something working and then iterate on it I've opted in for a workaround:

• When we ask azure-storage-adapter to read a file we always read it from private container
• When we write a private file we write it to private container
• When we write a public file we write it to both private and public container

This way we can ensure azure-storage-adapter will be compatible with oc storage-adapter api and we do not have to modify oc itself, however in the long run it would be best to modify oc-registry in such a way where we don't need this workaround.

Extract azure helper functions

Extract azure helper functions into separate file. This would allow us to test them separately from the adapter itself.

Add support for options when creating new S3 client

In order to support a non Amazon S3 compliant object storage, like Riak CS, Minion etc, it is necessary to be able to override the endpoint generated when creating a new client using new AWS.S3(); in the oc-s3-storage-adapter. The endpoint that is created by default follows the pattern https://{service}.{region}.amazonaws.com.

By specifying an options attribute in the S3 configuration of the registry configuration, the options can be used when creating a new client, ie new AWS.S3(conf.options);.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#constructor-property

oc registry not available

When I do the command
oc registry add https://opencomponents127.blob.core.windows.net
for regestering my azure blob storage, oc registery is not available.

Add minio support

Moving the issue from here opencomponents/oc#726

Add tests to the oc-storage-adapters-utils package

Add Google Cloud Platform support

Moving this issue here opencomponents/oc#682

Add Microsoft Azure Blob Storage support

Moving the issue from here opencomponents/oc#553

Add dependencies.io

Registry is not starting - components_list_get

Dear all,

I am not able to start the registry with s3 configured. Everthing looks fine and I can only see one issue here:

CommonPrefixes is undefined. If I debug the data object then I see that the s3 client had read the bucket.

the components.json file is currently empty because I try to upload the first component.

Does anybody has an idea?

Thanks in advance

Pierre

oc-gs-storage-adapter failing

There are occasional errors that aren't being logged due to opencomponents/oc#1022 the issue below looks to be an combination of a few errors all being logged at the same time. However what I think I can take away from this is that there is an auth issue. It looks like a new client is being created every time, which would call the auth api. Perhaps we can can create this once and re-use it.

https://github.com/opencomponents/storage-adapters/blob/master/packages/oc-gs-storage-adapter/index.js#L43

data: { error: 'internal_failure' },
--
connection: 'close',
'content-encoding': 'gzip',
'x-content-type-options': 'nosniff',
status: 503,
config:
data:
{ Error: internal_failure
at Generator.next (<anonymous>)
body:
at Gaxios.<anonymous> (/app/node_modules/gaxios/build/src/gaxios.js:72:27)
{ config:
url: 'https://www.googleapis.com/oauth2/v4/token',
{ 'cache-control': 'private',
server: 'scaffolding on HTTPServer2',
{ 'Content-Type': 'application/json',
status: 503,
headers: [Object],
paramsSerializer: [Function: paramsSerializer],
'x-xss-protection': '0' },
code: '503' } components/components.json

Dependencies.io troubleshooting

We're opening this issue to inform you about problems with your dependencies.io configuration or build failures.

We'll add a comment here when we see problems. You can close this issue when
you have things resolved -- we'll open it back up if we detect new problems.

Helpful links:

• dependencies.io/docs
• https://app.dependencies.io/

Note: To help keep your repo as clean as possible, this is the only issue
that we're going to open and comment on. Feel free to delete the comments
that are made as you resolve them, if you want to keep this issue short and tidy.