Large File Handling (eg 2.5 MB file) Slower When InMemory is True

[expressjs]

Large File Handling (eg 2.5 MB file) Slower When InMemory is True

[joelsouza]

I'm noticing multer is often very slower using memoryStorage. I running this app on Heroku (2 dynos 2.5GB RAM each) and can't reproduce this problem local.

I'm uploading/processing binary files < 10mb

Node: 10.13.0
Express: 4.16.4
Multer: 1.4.1

Procfile

web: node --optimize_for_size --max_old_space_size=960 --gc_interval=100 ./bin/www

From the Heroku logs:

14 Dec 2018 22:10:40.888111...app web.2 - multer: 546.008ms
14 Dec 2018 22:10:41.273111...app web.2 - multer: 799.298ms
14 Dec 2018 22:11:02.265113...app web.2 - multer: 14022.644ms
14 Dec 2018 22:11:20.837113...app web.1 - multer: 15048.781ms
14 Dec 2018 22:11:27.108113...app web.2 - multer: 15581.935ms
14 Dec 2018 22:11:50.460112...app web.2 - multer: 9385.592ms
14 Dec 2018 22:12:10.792112...app web.1 - multer: 1873.741ms
14 Dec 2018 22:12:17.253112...app web.1 - multer: 6949.123ms
14 Dec 2018 22:12:18.849111...app web.2 - multer: 459.384ms
14 Dec 2018 22:12:19.396111...app web.1 - multer: 762.943ms
14 Dec 2018 22:12:48.720112...app web.2 - multer: 8931.458ms
14 Dec 2018 22:12:48.927113...app web.2 - multer: 33099.362ms
14 Dec 2018 22:13:12.387112...app web.1 - multer: 6995.448ms
14 Dec 2018 22:13:52.009112...app web.2 - multer: 1641.400ms
14 Dec 2018 22:13:56.123111...app web.1 - multer: 448.621ms
14 Dec 2018 22:13:56.851111...app web.2 - multer: 746.916ms
14 Dec 2018 22:13:58.788112...app web.2 - multer: 7218.541ms
14 Dec 2018 22:14:22.591111...app web.2 - multer: 402.261ms
14 Dec 2018 22:14:22.840111...app web.2 - multer: 688.741ms
14 Dec 2018 22:14:23.346112...app web.1 - multer: 5194.058ms
14 Dec 2018 22:14:26.019111...app web.2 - multer: 534.709ms
14 Dec 2018 22:15:13.401112...app web.2 - multer: 3775.944ms
14 Dec 2018 22:15:17.796111...app web.2 - multer: 397.492ms
14 Dec 2018 22:15:18.204111...app web.2 - multer: 752.125ms
14 Dec 2018 22:15:41.958111...app web.1 - multer: 544.359ms
14 Dec 2018 22:15:42.480111...app web.2 - multer: 836.787ms
14 Dec 2018 22:15:47.693112...app web.1 - multer: 1727.417ms
14 Dec 2018 22:15:52.398111...app web.1 - multer: 791.530ms
14 Dec 2018 22:15:53.531112...app web.1 - multer: 6623.814ms
14 Dec 2018 22:15:56.511112...app web.1 - multer: 4379.295ms
14 Dec 2018 22:16:30.912111...app web.2 - multer: 585.118ms
14 Dec 2018 22:17:10.618112...app web.2 - multer: 1406.447ms
14 Dec 2018 22:17:13.976111...app web.2 - multer: 117.054ms
14 Dec 2018 22:17:16.091112...app web.2 - multer: 1649.331ms

My route is somthing like this:

const router = require('express').Router()
const multer = require('multer')
const upload = multer({ storage: multer.memoryStorage() })
const fileUpload = upload.single('file')

router.post('/file', (req, res) => {

  console.time('multer')
  fileUpload(req, res, async (err) => {
    console.timeEnd('multer')

    if (err)  console.error(err)
    	
    // 
    // upload file to AWS S3.
    // 

    return res.json({ })
  })
})

module.exports = router

Is there any way to search for the root of the problem?

[IsaiahJTurner]

Buffer is null for me. Is that normal?

[jpfluger]

If inMemory is true and the buffer is null, this is not normal. If the default setting is used (where inMemory is false), then yes it is normal. Maybe it's the latter?

[LinusU]

I'm not sure that this is an issue anymore with never node. From the blog:

UPDATE: None of this should be necessary, as FileReadStream in the latest node uses buffers by default. However, it appears that either I'm doing something wrong or the docs are out of date, as it doesn't work that way on node HEAD.

We are using buffers internally in 1.0.0.

Please reopen if you feel that this is still an issue.