orangewise/s3-files

Stream selected files from an Amazon s3 bucket/folder.

Hi there, I'm Orangewise 👋

s3-files's Issues

Unable to download files from root directory of the bucket

When "" passed as folder parameter (root directory of the bucket), createKeyStream returns null.
"/" and "./" also doesn't work.

Provide a TypeScript declaration file

This will allow TypeScript users to inspect and build off the types defined.

Not able to access files

How can we provide AWS credentials.

I am getting below error while fetching files.
CredentialsError: Missing credentials in config

RangeError [ERR_INVALID_OPT_VALUE]: The value "2246554069" is invalid for option "size"

Using s3-files as a dependency of s3-zip.

I am attempting pipe a large collection of files that can include one large file with size ranging from 500MB to 5GB. On data transfer, when I attempt to pipe a file larger than ~2GB, I get the following error:

buffer.js:269
    throw err;
    ^

RangeError [ERR_INVALID_OPT_VALUE]: The value "2246554069" is invalid for option "size"
    at Function.allocUnsafe (buffer.js:291:3)
    at Function.concat (buffer.js:473:23)
    at bufferConcat (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\concat-stream\index.js:117:17)
    at ConcatStream.getBody (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\concat-stream\index.js:64:42)
    at ConcatStream.<anonymous> (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\concat-stream\index.js:37:51)
    at ConcatStream.emit (events.js:194:15)
    at ConcatStream.EventEmitter.emit (domain.js:441:20)
    at finishMaybe (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\readable-stream\lib\_stream_writable.js:624:14)
    at endWritable (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\readable-stream\lib\_stream_writable.js:643:3)
    at ConcatStream.Writable.end (D:\Development\speedwise-platform-app-web\node_modules\s3-files\node_modules\readable-stream\lib\_stream_writable.js:571:22)
    at PassThrough.onend (_stream_readable.js:629:10)
    at Object.onceWrapper (events.js:277:13)
    at PassThrough.emit (events.js:194:15)
    at PassThrough.EventEmitter.emit (domain.js:441:20)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

The code:

router.get('/:mongoId/version/:versionId/download', (req, res) => {

  const { mongoId, versionId } = req.params

  res.setTimeout(1000 * 60 * 25)
  getVersionFilesForZipFolder(mongoId, versionId)
    .then(({ ids, fileLocations }) => {
      downloadZipDatasetVersion(ids, fileLocations)
      .pipe(res)
    })
})

function downloadZipDatasetVersion(ids, filePaths) {

  const params = { s3: MY_S3_INSTANCE, bucket: MY_BUCKET, debug: true }
  const folder = 'MY_FOLDER/'

  return s3Zip.archive(params, folder, ids, filePaths)
}

ids is an array of file references for s3

[
  'file-name-in-s3-1',
  'file-name-in-s3-2',
  'file-name-in-s3-3',
  'file-name-in-s3-4',
]

and fileLocations is an array of objects that look as follows:

[
  { name: '/dir1/originalfilename1.txt' },
  { name: '/dir2/originalfilename2.txt' },
  { name: '/dir3/originalfilename3.txt' },
  { name: '/dir4/originalfilename4.txt' },
]

Empty keyStream

When calling this function in s3-zip:
var fileStream = s3Files.createFileStream(keyStream)
console.log('fileStream: ', fileStream)

I'm receiving an empty stream:
fileStream: Stream {
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
readable: true }

The keyStream being passed is:
StreamArray {
_readableState:
ReadableState {
objectMode: true,
highWaterMark: 16,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: null,
pipesCount: 0,
flowing: null,
ended: false,
endEmitted: false,
reading: false,
sync: true,
needReadable: false,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
defaultEncoding: 'utf8',
ranOut: false,
awaitDrain: 0,
readingMore: false,
decoder: null,
encoding: null },
readable: true,
domain: null,
_events: {},
_eventsCount: 0,
_maxListeners: undefined,
_i: 0,
_l: 2,
_list:
[ 'folder/file1.jpeg',
'folder/file2.jpeg' ] }

Trying to figure out where it's broken

endpoint go wrong.

I ran out to get this error

Hostname/IP doesn't match certificate's altnames: "Host: my-bucket-name.s3.s3-eu-west-1.amazonaws.com. is not in the cert's altnames: DNS:s3-eu-west-1.amazonaws.com, DNS:*.s3-eu-west-1.amazonaws.com, DNS:s3.eu-west-1.amazonaws.com, DNS:*.s3.eu-west-1.amazonaws.com, DNS:s3.dualstack.eu-west-1.amazonaws.com, DNS:*.s3.dualstack.eu-west-1.amazonaws.com, DNS:*.s3.amazonaws.com"

look like it always insert .s3. between my bucket name and region. so I tested by force set the endpoint to "my-bucket-name.s3-eu-west-1.amazonaws.com" and it work!

so my work around for now is set some option to override endpoint like this.

AWS.config.update({
region: opts.region,
endpoint: opts.region + '.amazonaws.com' //this line
});

An in-range update of aws-sdk is breaking the build 🚨

Version 2.7.3 of aws-sdk just got published.

Branch	Build failing 🚨
Dependency	aws-sdk
Current Version	2.7.2
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

As aws-sdk is a direct dependency of this project this is very likely breaking your project right now. If other packages depend on you it’s very likely also breaking them.
I recommend you give this issue a very high priority. I’m sure you can resolve this 💪

---

Status Details

• ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

• ✅ coverage/coveralls First build on greenkeeper/aws-sdk-2.7.3 at 100.0% Details

Release Notes

Release v2.7.3

See changelog for more information.

Commits

The new version differs by 1 commits .

• bf0896b Updates SDK to v2.7.3

See the full diff.

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

Make it possible to add an VersioId to the getObject call

it would be great to add Objects { Key, VersionId } in the file Array.

Example:
.createKeyStream(folder, [file1, { Key, VersionID }, file3, file4])

This would be awsome

Any plan to upgrade aws-sdk from v2 to v3?

Since v2 is deprecated: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html