ikhsanalatsary/multer-sharp-s3

Multer Sharp S3 is streaming multer storage engine permit to transform / resize the image and upload to AWS S3.

multer-sharp-s3's Issues

Enable Sharp ARM 64 support

Need to update the version of Sharp, as it doesn't include prebuilt libraries for ARM 64 until version 0.30.0.
Sharp v0.30.1 is the latest as of today.

Reference: lovell/sharp#2938

I could submit a PR if you'd like; should just be a package.json update for sharp to 0.30.1.

Thanks!

Fix sharp version in peerDependencies

The version in peerDependencies is 0.24.x, and the version in dependencies is 0.25.x. This is my fault - I forgot to fix this in #27

Thanks.

I don't believe the version in @types/sharp has been updated yet.

sharp-linux-x64.node: undefined symbol: vips_fail_on_get_type

node: symbol lookup error: /home/ranga/code/mykids/code/kidsapi/node_modules/multer-sharp-s3/node_modules/sharp/build/Release/sharp-linux-x64.node: undefined symbol: vips_fail_on_get_type

Update readme

• from prev release v0.2.1

Multiple files not working properly

Hi,

Thanks very much for creating this wonderful package.
I tried it with multiple sizes yesterday. It worked when the request has one file. When there are multiple files, it doesn't seem to work.

In router, I am using:

fileUploadResizeS3.fields([
	{ name: 'eventImage', maxCount: 1 },
	{ name: 'courseMap', maxCount: 1 }
]),

fileUploadResizeS3 is where I define multer-sharp-s3:

const fileUploadResizeS3 = multer({
	limits: 1500000, 
	storage: s3Storage({
		ACL: 'public-read', 
		s3: s3,
		Bucket: process.env.S3_BUCKET_NAME,
		metadata: (req, file, cb) => {
			const ext = MIME_TYPE_MAP[file.mimetype];
			cb(null, { fieldName: file.fieldname });
		},
		Key: (req, file, cb) => {
			const UUID = uuid();
			const ext = MIME_TYPE_MAP[file.mimetype];

			let S3Folder;
                          .........

			cb(null, S3Folder + '/' + UUID + '.jpg');
		},
		multiple: true,
		resize: [{ suffix: 'small', width: 300 }, { suffix: 'original' }],
		toFormat: { type: 'jpeg', options: { quality: 100 } }
	}),
	fileFilter: (req, file, cb) => {
		const isValid = !!MIME_TYPE_MAP[file.mimetype];
		let error = isValid ? null : new Error('Invalid MIME type!');
		cb(error, isValid);
	}
});

It did return 2 file arrays:

req.files =  [Object: null prototype] {
  eventImage: [
    {
      fieldname: 'eventImage',
      originalname: 'Bowser.jpg',
      encoding: '7bit',
      mimetype: 'image/jpeg',
      small: [Object],
      original: [Object]
    }
  ],
  courseMap: [
    {
      fieldname: 'courseMap',
      originalname: 'CourseMap.png',
      encoding: '7bit',
      mimetype: 'image/png',
      small: [Object],
      original: [Object]
    }
  ]
}

The problem is files inside of eventImage are actually coureMap files. CourseMap files are empty, 0 byte.

Please look into this issue and let me know if you need more information.

Thanks again!

Upload from URL to S3

Thank you for an amazing library. It helps to update user profile pictures very easily. Is there any way to make it work with google auth for new users?

Google sends profile picture URL like this "https://lh3.googleusercontent.com/a/AATXAJxSeXuBKi9jM9hYezWjz0ePryguXRuGYgRV_OC5=s96-c"

Problem is that sometimes this URL gets expired, so probably better to upload a profile picture to AWS. Is there any way to make it?

An in-range update of @types/node is breaking the build 🚨

The devDependency @types/node was updated from 11.13.7 to 11.13.8.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/node is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Keep .GIF/ .Webp animated

Hi guys,

I am having problem retaining animation when uploading images to s3. Could someone help me about this ?

Thank you.

issue

nodejs 10.15.1 not supporting

Not able to install in node version of 10.15.1 we can't change nodejs version for our product can can multer-s3-sharp support node js 10.15.1 version

Memory leak detected.

Hi.
It seems that when uploading an image with multiple resizes causes memory leaks under a small load. I've performed a load test using this library and it fails around 15 concurrent requests. Here's the error stack. Any help would be appreciated.

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit
at _addListener (events.js:247:17)
at Sharp.addListener (events.js:263:10)
at Sharp.Readable.on (_stream_readable.js:800:35)
at ManagedUpload.send (/Users/me/Documents/wip/project/node_modules/aws-sdk/lib/s3/managed_upload.js:185:11)
at /Users/me/Documents/wip/project/node_modules/aws-sdk/lib/util.js:798:25
at new WrappedPromise (/Users/me/Documents/wip/project/node_modules/async-listener/es6-wrapped-promise.js:13:18)
at ManagedUpload.promise (/Users/me/Documents/wip/project/node_modules/aws-sdk/lib/util.js:797:14)
at MergeMapSubscriber.sizes.pipe.operators_1.mergeMap [as project] (/Users/me/Documents/wip/project/node_modules/multer-sharp-s3/dist/main.js:113:52)
at MergeMapSubscriber._tryNext (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:69:27)
at MergeMapSubscriber._next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:59:18)
at MergeMapSubscriber.Subscriber.next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/Subscriber.js:66:18)
at MergeMapSubscriber.notifyNext (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:92:26)
at InnerSubscriber._next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/InnerSubscriber.js:28:21)
at InnerSubscriber.Subscriber.next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/Subscriber.js:66:18)
at /Users/me/Documents/wip/project/node_modules/rxjs/internal/util/subscribeToPromise.js:7:24
at propagateAslWrapper (/Users/me/Documents/wip/project/node_modules/async-listener/index.js:504:23)
at /Users/me/Documents/wip/project/node_modules/async-listener/glue.js:188:31
at /Users/me/Documents/wip/project/node_modules/async-listener/index.js:541:70
at /Users/me/Documents/wip/project/node_modules/async-listener/glue.js:188:31

Suffix on Resize problem

Thanks for amazing package, I have a little problem about suffix, i want to resize image to multiple size, this is my code :

the result what i got on S3 like this :

the problem is no type on my uploaded file, I've try add static extension on suffix like

{suffix : "1920px.jpg"}

it's work !, but i need extension with dynamic responsibility

I've to try use file.originamname for Key but the output something like :

suffix print after the extension

my expectation for output is "<filename>-<suffix>.<extension>" , How do I got this output ?

Thanks !

Auto Rotating the Image

Hello I am using multer-sharp-s3 for upload and resize images on s3

but some how it's auto rotating some images

Can anyone help on this

`const upload = multer({
fileFilter: (req, file, cb) => {
console.log(file);

if (file.mimetype == "application/octet-stream"){
    file.mimetype = 'image/jpeg';
}
if (file.mimetype == "image/png" || file.mimetype == "image/jpg" || file.mimetype == "image/jpeg") {
  cb(null, true);
} else {
  req.file_error = "INVALID";
  cb(null,false);
  //cb('Only .png, .jpg and .jpeg format allowed!', false);
  //return cb(new Error('Only .png, .jpg and .jpeg format allowed!'));

}

},
storage: s3Storage({
s3: s3,
Bucket: process.env.S3_BUCKET_NAME,
Key: (req, file, cb) => {
let originalname = file.originalname;
var extension = originalname.substr(originalname.lastIndexOf('.') + 1);
crypto.pseudoRandomBytes(8, (err, raw) => {
cb(err, err ? undefined : "v2/poster/" + raw.toString("hex"))+'.'+extension;
});
},
ACL: process.env.S3_ACL,
multiple: true,
resize: [{ suffix: "landingDeck", width: 800,height: 600, fit: "fill" }],
//resize: [{suffix: 'landingDeck' }],
//cacheControl: "max-age=31536000",

}),
});
`

Base64 Image string with multer-sharp-s3

Is it possible to accept Base64 Image from users and upload it to s3?

Setup GitHub action

• replacing current CI / CD
• auto publish to npm registry

Problem with Sharp version on Apple M1 chips

There's a problem with Sharp version you're using on M1 Macbooks. It just cannot be installed.
Can you update Sharp to the latest version?

https://sharp.pixelplumbing.com/install#apple-m1

Node.js multer-sharp-s3 working locally but error on Heroku: Error: Input buffer contains unsupported image format

I was able to run this locally with following code, but when I pushed it to Heroku, it gives below error.

Error: Input buffer contains unsupported image format

Code:

const aws = require('aws-sdk')
const { v4: uuidv4 } = require('uuid');

aws.config.update({
secretAccessKey: process.env.AWS_ACCESS_KEY,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
region: process.env.AWS_LOCATION,
})

const s3 = new aws.S3()

const s3Storage = require('multer-sharp-s3')
const MIME_TYPE_MAP = {
'image/png': 'png',
'image/jpeg': 'jpg',
'image/jpg': 'jpg',
}

const storage = s3Storage({
s3: s3,
Bucket: process.env.AWS_BUCKET_NAME,
Key: function(req, file, cb) {
console.log(file)
const isValid = MIME_TYPE_MAP[file.mimetype]
let error = new Error('Invalid mime type')
if (isValid) {
error = null
}
cb(error,${process.env.AWS_BUCKET_NAME}/places/${uuidv4()})
},
ACL: 'public-read',
multiple: true,
resize: {
width: 400,
height: 400,
},
})

const upload = multer({ storage })

multer-sharp-s3 not running with docker node alpine

Docker restart only after node require multer-sharp-s3

Multer Sharp S3 stop nodejs docker without logging any error

Docker Restarting (139) 37 seconds ago

Stackoverflow question: https://stackoverflow.com/q/61280152/6786941

Not able to upload BMP format

When I'm trying to upload BMP file to S3, I'm facing the following error below.

Error: Input buffer contains unsupported image format

It seems like the multer package does not support BMP format. Do you have any ideas?

{rotate: true} breaks my app

Hello, maybe I'm not using the rotate how I should be but in the previous version it worked just fine. Since 0.2.0 whenever I upload an image this happens. I took the liberty of going into the code and as I saw the part that is commented in transformer.js was responsible for things working. Maybe I'm totally off-point but I said to give you my observations since they may be of use to you.

Cheers

An in-range update of @types/jest is breaking the build 🚨

The devDependency @types/jest was updated from 24.0.11 to 24.0.12.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@types/jest is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Metadata and Content-Disposition

Will this library support Metadata and Content-Disposition?

I've tried adding Content-Disposition like this:

var upload = multer({
    storage: multerS3({
        s3: s3,
        Bucket: s3Bucket,
        ContentType: multerS3.AUTO_CONTENT_TYPE,
        ContentDisposition: 'attachment',
        Key: (req, file, cb) => {
            cb(null, file.originalname);
        }
    }),
    fileFilter: mediaFilter
});

but in my S3 bucket, the metadata Content-Disposition does not get added.

S3 Upload Property has Content-Disposition as one of its properties.

Looking forward to this.

How to resize a image dynamically by passing a dynamic width

I am trying to set the resizing width from the database values
I have successfully developed the function with the Static Width. but doing it dynamic from the database is not possible

Can please guide about making the width value dynamic from the database using Helper function.

let partsImageResolution = Helper.getResolution();
const uploadS4 = multer({
    storage: s4Storage({
         s3: s3,
        ACL: 'public-read',
        Bucket: 'mxcdn/parts',
        contentType: s4Storage.AUTO_CONTENT_TYPE,
        Key: (req, file, cb) => {
            cb(null, Date.now() + '-' + file.originalname);
        },
        resize: partsImageResolution
    })
}).any();

When I Pass directory Attribute it will not upload in that directory

My Code:
const storage = s3Storage({
// Key: (req, file, cb) => {
// crypto.pseudoRandomBytes(16, (err, raw) => {
// cb(err, err ? undefined : raw.toString('hex'))
// })
// },
s3,
Bucket: bucketName,
acl: acl,
metadata: metaData,
multiple: true,
resize: [
{ suffix: 'xlg', directory: 'product/extralarge', width: 1200, height: 1200 },
{ suffix: 'lg', directory: 'product/large', width: 800, height: 800 },
{ suffix: 'md', directory: 'product/medium', width: 500, height: 500 },
{ suffix: 'sm', directory: 'product/small', width: 300, height: 300 },
{ suffix: 'xs', directory: 'product/xtrasmall', width: 100 },
{ suffix: 'original', directory: 'product/original' }
],
});

It just stores as
bucket/filename-xs.jpg
not as bucket/product/xs/filename.jpg

My response:
{
"images": {
"fieldname": "file",
"originalname": "1(1).jpg",
"encoding": "7bit",
"mimetype": "image/jpeg",
"xs": {
"ACL": "public-read",
"ETag": ""2f8701ed2c65e6ae4ec68705c3bbbba2"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-xs",
"key": "6d90e0c93ccd5a31234f25c58dba4915-xs",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-xs",
"Bucket": "lumier32",
"directory": "xtrasmall",
"width": 100,
"height": 100,
"premultiplied": false,
"size": 3375,
"ContentType": "jpeg"
},
"sm": {
"ACL": "public-read",
"ETag": ""003a633ae136ad767df1b661431418c2"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-sm",
"key": "6d90e0c93ccd5a31234f25c58dba4915-sm",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-sm",
"Bucket": "lumier32",
"directory": "small",
"width": 300,
"height": 300,
"premultiplied": false,
"size": 17346,
"ContentType": "jpeg"
},
"md": {
"ACL": "public-read",
"ETag": ""2e8103cf1bec4e46012c30332138636e"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-md",
"key": "6d90e0c93ccd5a31234f25c58dba4915-md",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-md",
"Bucket": "lumier32",
"directory": "medium",
"width": 500,
"height": 500,
"premultiplied": false,
"size": 35310,
"ContentType": "jpeg"
},
"original": {
"ACL": "public-read",
"ETag": ""5055954117ccfe24edcea2ddb1ceef48"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-original",
"key": "6d90e0c93ccd5a31234f25c58dba4915-original",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-original",
"Bucket": "lumier32",
"directory": "original",
"width": 384,
"height": 384,
"premultiplied": false,
"size": 26063,
"ContentType": "jpeg"
},
"lg": {
"ACL": "public-read",
"ETag": ""30a0a0a80c7f6a76ace203bde3d337cd"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-lg",
"key": "6d90e0c93ccd5a31234f25c58dba4915-lg",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-lg",
"Bucket": "lumier32",
"directory": "large",
"width": 800,
"height": 800,
"premultiplied": false,
"size": 66100,
"ContentType": "jpeg"
},
"xlg": {
"ACL": "public-read",
"ETag": ""dd67b9a9d2c503772b6b84ba78789e25"",
"Location": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-xlg",
"key": "6d90e0c93ccd5a31234f25c58dba4915-xlg",
"Key": "6d90e0c93ccd5a31234f25c58dba4915-xlg",
"Bucket": "lumier32",
"directory": "extralarge",
"width": 1200,
"height": 1200,
"premultiplied": false,
"size": 113850,
"ContentType": "jpeg"
}
},
"imageUrl": "Bucket/6d90e0c93ccd5a31234f25c58dba4915-xs",
"message": "Successfully uploaded!"
}

[dependencies] Update sharp version

sharp version needs to be updated in order to support Node v12. Version in your dependencies causes compilation failure because Node v12 removed deprecated v8::Handle.

Unable to add metadata

Hi,

Thank you for making this library. It is super useful and I would like to use it for my app in production. I was wondering if it is possible to add metadata and content type to an image when uploading. As of now, I don't see them as options in your README.

I was able to set contentType by using multer-s3 library, but it would be nice if multer-sharp-s3 could have this feature too to avoid having to install both libs :)

Additionally, I tried adding metadata to an image but it doesn't seem to work. Here's the code I tried.

const express = require("express");
const aws = require("aws-sdk");
const multer = require("multer");
const multerS3 = require("multer-s3");
const s3Storage = require("multer-sharp-s3");

...

const storage = s3Storage({
  s3,
  Bucket: "bucket-name",
  contentType: multerS3.AUTO_CONTENT_TYPE,
  metadata: (req, file, cb) => {
    cb(null, { ...req.body });
  },
  Key: (req, file, cb) => {
    const currentTime = getCurrentTime();
    fileName = `${req.body.category}-${currentTime}-${file.originalname}`;
    cb(null, fileName);
  },
  resize: {
    width: 400,
    height: 400
  },
  max: true
});
const upload = multer({ storage });

...

Image is uploaded to S3 without added metadata. I tried capitalizing the metadata key to match your other keys' convention, but it gave me this error: InvalidParameterType: Expected params.Metadata to be a map.

Not able to upload multiple images

Hi,

This lib is very usefull, it would be good to have support to multiple uploads at once.

router.post('/update', isAuthorized, upload.array('images', 20), update);

const upload = multer({
storage: s3Storage({
s3,
ACL: 'public-read',
Bucket: config.awsBucket,
multiple: true,
crop: 'attention',
resize: [
{ suffix: 'xlg', width: 1200 },
{ suffix: 'lg', width: 800, height: 600 },
{ suffix: 'md', width: 500, height: 500 },
{ suffix: 'sm', width: 265, height: 190 },
{ suffix: 'xs', width: 100 },
],
Key(req, file, cb) {
cb(null, ${Buffer.from(file.originalname).toString('base64')}_${Date.now().toString()});
},
}),
});

Right now if I use this code to upload several images, the first one is uploaded successfully but the rest all are stored with 0 bites.

Is this library still maintained?

Is this library still maintained? I love using it but have not seen much action on current issues. Great work on the implementation though.

An in-range update of aws-sdk is breaking the build 🚨

The devDependency aws-sdk was updated from 2.442.0 to 2.443.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

aws-sdk is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build could not complete due to an error (Details).

---

Release Notes for Release v2.443.0

See changelog for more information.

Commits

The new version differs by 1 commits.

• 7e91a1a Updates SDK to v2.443.0

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Rewrite Lib

Improve parallel programming using worker_threads or using worker_pool. I don't know it is possible or not in multer lib