Giter Site home page Giter Site logo

async-busboy's Introduction

Promise Based Multipart Form Parser

NPM version build status Test coverage npm download

The typical use case for this library is when handling forms that contain file upload field(s) mixed with other inputs. Parsing logic relies on busboy. Designed for use with Koa2 and Async/Await.

Examples

Async/Await (using temp files)

import asyncBusboy from 'async-busboy';

// Koa 2 middleware
async function someFunction(ctx, next) {
  const { files, fields } = await asyncBusboy(ctx.req);

  // Make some validation on the fields before upload to S3
  if (checkFiles(fields)) {
    files.map(uploadFilesToS3);
  } else {
    return 'error';
  }
}

Async/Await (using custom onFile handler, i.e. no temp files)

import asyncBusboy from 'async-busboy';

// Koa 2 middleware
async function someFunction(ctx, next) {
  const { fields } = await asyncBusboy(ctx.req, {
    onFile: function (fieldname, file, filename, encoding, mimetype) {
      uploadFilesToS3(file);
    },
  });

  // Do validation, but files are already uploading...
  if (!checkFiles(fields)) {
    return 'error';
  }
}

ES5 with promise (using temp files)

var asyncBusboy = require('async-busboy');

function someFunction(someHTTPRequest) {
  asyncBusboy(someHTTPRequest).then(function (formData) {
    // do something with formData.files
    // do someting with formData.fields
  });
}

Async API using temp files

The request streams are first written to temporary files using os.tmpdir(). File read streams associated with the temporary files are returned from the call to async-busboy. When the consumer has drained the file read streams, the files will be automatically removed, otherwise the host OS should take care of the cleaning process.

Async API using custom onFile handler

If a custom onFile handler is specified in the options to async-busboy it will only resolve an object containing fields, but instead no temporary files needs to be created since the file stream is directly passed to the application. Note that all file streams need to be consumed for async-busboy to resolve due to the implementation of busboy. If you don't care about a received file stream, simply call stream.resume() to discard the content.

Working with nested inputs and objects

Make sure to serialize objects before sending them as formData. i.e:

// Given an object that represent the form data:
{
  field1: 'value',
  objectField: {
    key: 'anotherValue',
  },
  arrayField: ['a', 'b'],
  //...
}

Should be sent as:

// -> field1[value]
// -> objectField[key][anotherKey]
// -> arrayField[0]['a']
// -> arrayField[1]['b']
// .....

Here is a function that can take care of this process

const serializeFormData = (obj, formDataObj, namespace = null) => {
  var formDataObj = formDataObj || {};
  var formKey;
  for (var property in obj) {
    if (obj.hasOwnProperty(property)) {
      if (namespace) {
        formKey = namespace + '[' + property + ']';
      } else {
        formKey = property;
      }

      var value = obj[property];
      if (
        typeof value === 'object' &&
        !(value instanceof File) &&
        !(value instanceof Date)
      ) {
        serializeFormData(value, formDataObj, formKey);
      } else if (value instanceof Date) {
        formDataObj[formKey] = value.toISOString();
      } else {
        formDataObj[formKey] = value;
      }
    }
  }
  return formDataObj;
};

// -->

Try it on your local

If you want to run some test locally, clone this repo, then run: node examples/index.js From there you can use something like Postman to send POST request to localhost:8080. Note: When using Postman make sure to not send a Content-Type header, if it's filed by default, just delete it. (This is to let the boudary header be generated automatically)

Use cases:

  • Form sending only octet-stream (files)

  • Form sending file octet-stream (files) and input fields. a. File and fields are processed has they arrive. Their order do not matter. b. Fields must be processed (for example validated) before processing the files.

async-busboy's People

Contributors

amit777 avatar bodasia avatar chadkirby avatar christensson avatar dependabot[bot] avatar dominhhai avatar fireyy avatar hyralm avatar jd-robbs avatar loborobo avatar m4nuc avatar octet-stream avatar twentyrogersc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

async-busboy's Issues

can i get the file's size ?

hi, i has a question

for example:

{fields, files} = await asyncBusboy(ctx.req)

i can get the mime, path, filename and others information of files[0] but the file's size.

how to get size? is must to use the fs.stat(path) or read to buffer ?

Only picture file may be upload successfully

I test the lib in koa2, I found only picture file may be upload successfully, the .txt .execl files was unable to upload successfully! Is there options to set? Such as file type, upload path and so on.
BTW, I have no function named checkFiles and uploadFilesToS3, so I'd commend the lines for those codes.
Thank, please.

using asyncBusboy in two middleware

I have midllware for validate fields formData

const checkAndSendErrorValidateMiddleware = (schema, transformError) => async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)
   
  const filesFields = files.reduce((acc, fileReadStream) => ({ ...acc, [fileReadStream.fieldname]: fileReadStream }), {})
    
  const resultValidate = validate({ ...fields, ...filesFields }, schema)
  const errors = transformError(resultValidate.errors)
  if (resultValidate.errors.length !== 0) {
    ctx.body = JSON.stringify({
      errors,
      data: null,
    })
 }
 await next()
}

and controler midlleware

const controler = async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)

  const instanceModel = await db.findById(fields.id)
 // e.t.c
})

when I connect them in a chain, then in controler fields, files empty. Why?

route.post('/url', checkAndSendErrorValidateMiddleware(validateScheme), controler)

Replace unsupported busboy version with fastify fork

Hi,

we forked busboy and fixed two critical bugs in the package, which could cause the node-process to crash or to hang. We also improved the performance and added some new features. It does not have breaking changes so it is a drop-in-replacement for busboy. It has a code coverage of about 95%.

https://github.com/fastify/busboy/blob/master/CHANGELOG.md
https://github.com/fastify/busboy
https://www.npmjs.com/package/@fastify/busboy

for tracking reasons:
fastify/busboy#68

Is there a way to expose uploading progress?

Hi i just tried out this tool. It's awesome. Is there a way or an example you can give for tracking the uploading progress? Is this something that can be done on the client side right away or do i have to implement some sort of custom logic on the server?

Can I do "file.pip(stream)" like the way co-busboy does?

Hi,

After the server receive the formdata and have been successfully parsed by async-busboy you built. I need to save the file to some other directory. How can I make it?

The reason why I didn't use co-busboy is that it act like a middleware. But I expect the saving operations can be processed in a specific route.

Here's the co-busboy example:

var parse = require('co-busboy')

app.use(function* (next) {
  // the body isn't multipart, so busboy can't parse it
  if (!this.request.is('multipart/*')) return yield next

  var parts = parse(this)
  var part
  while (part = yield parts) {
    if (part.length) {
      // arrays are busboy fields
      console.log('key: ' + part[0])
      console.log('value: ' + part[1])
    } else {
      // otherwise, it's a stream
      part.pipe(fs.createWriteStream('some file.txt'))
    }
  }
  console.log('and we are done parsing the form!')
})

What I expect:

const router = require('koa-router)()
router.post('/api/upload', async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)
  files.map(file => {
    // saving file
  }
  ...
}

(By printing out the file variable in the map function above and the part variable in the while loop. I notice that file is kind of like a ReadStream object while part is a FileStream object. )

Sorry about asking a question that is not fully related to your repo. I've tried some other parser and methods but it seems like it still cannot work properly. Could you give me some advice about how to solve this problem?

Much appreciated

Processing unindexed array fields cause a whole process to crash

A common use case is to use array fields with no specified indexes, e.g.:

<input type="checkbox" name="fruits[]" value="apple">
<input type="checkbox" name="fruits[]" value="orange">
<input type="checkbox" name="fruits[]" value="banana">

However, upon receiving more than one such field, e.g. fruits[]=apple and fruits[]=banana, the whole Node process crashes with this stack trace:

RangeError: Maximum call stack size exceeded
    at Function.keys (<anonymous>)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:193:22)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)

There are 2 problems here. I propose the following solutions:

  1. There should be no process crash, in any circumstances. Upon any error, even internal, the exception should be thrown. It seams that the underlying busboy module poorly handles errors inside registered event handlers.
    Suggested solution: wrap any internal field and file processing function with a try..catch, which would reject the main promise.
  2. This use case is very common and should be handled properly. And making your own parser can be difficult and sometimes even unsafe.
    I think, in general, TJ’s qs module could (and maybe even should) be used here for robust field parsing and aggregating. Off the top of my head:
const qs = require('qs')
const fieldList = [
	{name: 'fruits[]', value: 'apple'},
	{name: 'fruits[]', value: 'banana'},
]
const fields = qs.parse(fieldList
	.map(({name, value}) => encodeURIComponent(name) + '=' + encodeURIComponent(value))
	.join('&'))

Using form name arrays without defined key name causes a TypeError.

I found an issue when trying to use array values without a defined key name. This can be easily replicated.

I am unsure if this is an issue in the wrapper or the original code, though the error occurs in the reconcile function, which is not present in the synchronous version; hence why I am sharing the issue here.

<input type="password" name="password[]"> <!-- Should be treated as password[0] -->
<input type="password" name="password[]"> <!-- Should be treated as password[1] -->

The actual error occurring:

TypeError: Cannot create property 'undefined' on string ''
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:208:24)
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:206:12)
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:206:12)
    at onField (Path\To\Project\node_modules\async-busboy\index.js:84:5)
    at emitMany (events.js:127:13)
    at Busboy.emit (events.js:204:7)
    at Busboy.emit (Path\To\Project\node_modules\busboy\lib\main.js:38:33)
    at UrlEncoded.end (Path\To\Project\node_modules\busboy\lib\types\urlencoded.js:205:14)
    at Busboy.emit (Path\To\Project\node_modules\busboy\lib\main.js:31:36)
    at finishMaybe (_stream_writable.js:510:14)
    at endWritable (_stream_writable.js:520:3)
    at Busboy.Writable.end (_stream_writable.js:485:5)
    at IncomingMessage.onend (_stream_readable.js:511:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

Putting the following works around the issue, however creates problems when dynamically creating elements based on user interaction:

<input type="password" name="password[0]">
<input type="password" name="password[1]">

Documentation needs an update

Documentation is wrong with current file

(filePromises, fieldname, file, filename, encoding, mimetype)

filePromises is the first argument, not the file object

Sometime asyncBusboy never responding

Sometimes I found that asyncBusboy not responding because promise never resolved or rejected.
I traced inside asyncBusBoy source code and I found that sometimes request's 'close' event handler called
before busboy's 'finish' handler ( before onEnd called).
I modified source code like "request.on('close', onEnd)" instead of "request.on('close', cleanup)" and it works perfectly.
I'm afraid I'm doing wrong and make some side effects. Replacing handler to onEnd instead of cleanup is safe?

can't get files

apiroute.post("/upload",async function (ctx,next) {
    asyncBusboy(ctx.req).then(function (formData) {
        console.log(formData);
    })
});
console  ->  { fields: { modName: '23333', mod: 'undefined' }, files: [] }

this files is empty
use koa-router and async-busboy

whit demo

 var bd = new FormData();
                bd.append("modName","23333");
                bd.append("mod",sj.file);
                    $.ajax({
                        url: '/api/upload',
                        type:"post",
                        cache:false,
                        data:bd,
                        processData: false,
                        contentType:false,
                        success:function () {
                            console.log("OK");
                        }
                    });
            }

small file can not be parsed?

I find a bug.

A small file which maybe 9k can not be pushed into 'files'.I debug it and perhaps it returned before the stream was pushed into the files.


I get what's wrong.

image

the busboy closed before the writeStream open.

so if i code like this

function onFile(files, fieldname, file, filename, encoding, mimetype) {
  const tmpName = file.tmpName = new Date().getTime()  + fieldname  + filename;
  const saveTo = path.join(os.tmpDir(), path.basename(tmpName));
  const writeStream = file.pipe(fs.createWriteStream(saveTo));
    const readStream = fs.createReadStream(saveTo);
    readStream.fieldname = fieldname
    readStream.filename = filename
    readStream.transferEncoding = readStream.encoding = encoding
    readStream.mimeType = readStream.mime = mimetype;
    files.push(readStream);
}

then it work.

is it a bug or there is anyother reason why you code such as?

formData.files as on object

I think it would be nicer to have files as an object of keys -> ReadStreams so it would be easier to validate the existence certain files

ENOENT: no such file or directory in `os.tmpDir()`

Only sometimes when sending files via multipart/form-data and uploading them to S3 I am getting the following error:

Uncaught Error: ENOENT: no such file or directory, open '/var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/filename.jpg'
      at Error (native)

When I log files variable, it always lists all files as ReadStreams with path set to /var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/filename.jpg, however. Plus, when I do ls -la /var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/ on a os.tmpDir() dir afterwards, desired files exist inside:

const { files, fields } = await asyncBusboy(ctx.req)

Wondering whether async-busboy causing the issue when creating ReadStreams or something with Promises I use for uploading files to S3…
The issue happens only sometimes. Most of the times, uploading is works successfully.

node 14

It seems that node 14 has some issue with this lib, the following code hangs:

const { files, fields } = await asyncBusboy(context.req, config);

node 13 is fine.
Thanks.

Can't read the file content

I am using async-busboy to read a .txt file uploaded from multipart form in postman. However, it can't read the file content so the buffer in the returned "files" object is always empty, and the length of the file is zero.

My code:
router.post( '/attachment',async (ctx, next) => {
    try {
        const {files, fields} = await asyncBusboy(ctx.req);
        ctx.code = 200;
        ctx.body = {
            code: 200,
            data: files,
        };
    } catch (err) {
        log.error(err);
        ctx.throw(500, 'Internal Server Error');
    }
  },
);

The returned file object:
[
    {
        "_readableState": {
            "objectMode": false,
            "highWaterMark": 65536,
            "buffer": {
                "head": null,
                "tail": null,
                "length": 0
            },
            "length": 0,
            "pipes": null,
            "pipesCount": 0,
            "flowing": null,
            "ended": false,
            "endEmitted": false,
            "reading": false,
             "sync": true,
            "needReadable": false,
            "emittedReadable": false,
            "readableListening": false,
            "resumeScheduled": false,
            "destroyed": false,
            "defaultEncoding": "utf8",
            "awaitDrain": 0,
            "readingMore": false,
            "decoder": null,
            "encoding": null
        },
        "readable": true,
        "domain": null,
        "_events": {},
        "_eventsCount": 1,
        "path": "/tmp/0d812ea775d47-test1.txt",
        "fd": null,
        "flags": "r",
        "mode": 438,
        "start": 0,
        "end": null,
        "autoClose": true,
        "pos": 0,
        "bytesRead": 0,
        "fieldname": "",
        "filename": "test1.txt",
        "encoding": "7bit",
        "transferEncoding": "7bit",
        "mime": "text/plain",
        "mimeType": "text/plain"
    }
    ]
}

504 Gateway timeout for specific files (.csv / .xlsx)

Send the files with csv or xlsx in formdata, But async busboy just keep on waiting for the response and ending up with 504 gateway timeout. But if I use .xls file extension , It successfully parsing that incoming file. Could not find what's the issue for other file formats like .csv or .xlsx.

Publish a new version to npm

I was trying to debug and fix a bug described in #42. It took me a while to realize, it is already fixed in master branch, just not released to latest version (1.1.0).

Could you please release new version from master?

writing unfinished while get files

var {files, fields} = await parseFile(ctx.req, {autoFields: true});
fs.readFileSync(files[0].path); // <buffer > (a empty buffer, because writing is unfinished)

is that current?

Now I have to code like this:

return new Promise((resolve, reject) => {
  let index = 0;
  let interval = setInterval(() => {
    let data = fs.readFileSync(file.path);
    if (data) {
      resolve(parse(data));
      clearInterval(interval);
    } else if (index > 10) {
      reject(new RestError(400, 'FILE_LOAD_ERR', 'no file loaded'));
      clearInterval(interval);
    }
    index++;
  }, 100);
});

and it returned current buffer.

How can you do onFile with async?

Hi

So I have a question I want to do an async/await pattern with the onFile custom handler.

I tried several ways and I dont' think I found one that works out.

I know this may be very simple, but I couldn't figure it out myself.

Throw on('limit') ?

Would it be possible to implement the file.on('limit') event? I would really like to have an error thrown like shown below or exposing the listeners or having same options for callbacks, not sure what would be the best practice here...

function onFile( //...
//...
  file.on('limit', () => {
    const err = new Error('Reach file size limit');
    err.code = 'Request_file_size_limit';
    err.status = 413;
    throw err;
  })

How to "drain the request stream"

Quoting the doc: "When the consumer stream drained the request stream, files will be automatically removed, otherwise the host OS should take care of the cleaning process."

What would be the best practice for reading the files e.g. to write them in a data base. I'm currently using fs.readFileSync(files[0].path) and deleting the file with fs.unlink() function.

Edit: Extending the question, because it's probably related:
To get the file size one must use fs.stat(PATH) but that's only possible when you have the path already. But how to skip a file upload that exceeds a certain size?

Note file caching in the readme

I understand that file caching is inevitable in a promise-based form parser, since to be able to have an overview of the whole form in one go you also need to, well, read and process the whole body in one go. This is obvious if you stop and think about it for a second, but, being in a rush to just get stuff done, this has escaped my attention, until a slightly later "duh" moment. A quick note, warning people about the difference in nature between async-busboy and co-busboy / busboy, somewhere high up in the readme, could save some time for somebody else in a hurry.

Just in case, this matters, because if you're interested in doing any processing/spying on a file (e.g. checking its magic bytes or calculating a hash), piping it through your transforms while it's downloading is more performant than saving it on the disk and then reading from there.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.