vekexasia/mysqldumpsplit

A fast and memory efficient mysqldump (per table) file splitter written in Node.js

mysqldumpsplit

mysqldumpsplit. is a sysadmin utility that splits a single .sql file into several .sql files (one per table)

The Module is written in ES6 transpiled using the wonderful Babel library.

You can leverage the cli command or import it into your node project. Please take a look at ./src/cli.js on how to use it within your node project.

Requirements

• node.js - v0.12 or newer!

Installation

On node project

npm install mysqldumpsplit

As cli command

npm install mysqldumpsplit -g

Performances

Since the library processes the .sql file using node streams, it can handle large files without fully loading it in memory.

Using a pretty decent Laptop I was able to split a 180 tables, 16GB .sql file in under 2 minutes using less than 50MB of RAM.

Cli usage

The cli command has only one required param: the .sql file path. It will split the file within the same folder.

If you wish to customize the location of the splitted files, provide the extra -o argument as described below:

  Usage: mysqldumpsplit [options] <dumpfile>

  Options:

    -h, --help             output usage information
    -V, --version          output the version number
    -o, --output [folder]  The output folder, by default it will be the same folder of the dump

How it works

Under the hood this node module uses my streamsplit node library to search for the string Table structure for table within the dump file. It then splits the original file using the locations of such token. Hence if you have a mysqldump that does not contain such string this won't work. If you're using a popular backup script please file a bug so that I can eventually add other popular tokens :)

mysqldumpsplit's Issues

Error: "Promise is not defined"

Hello,

I have got the following error:

xxx@xxx /var/www/xxxxxx (release) $ mysqldumpsplit database.sql
Starting :)
[ReferenceError: Promise is not defined]

cheers,
Peter

Error, if chunk size > 128kb

hello ,

If the chunk size exceeds the maximum string length of V8, then the following error will occur:

Error: toString failed
    at Buffer.toString (buffer.js:377:11)

this line will fail:

 var matches = /for table `(.*?)`/.exec(chunk.toString('utf8'));

add maximum size

Please add an option, to split large tables into multiple files

Error

Getting this error when I run it on a 3gb file.

Starting :)
{ Error: EINVAL: invalid argument, read errno: -4071, code: 'EINVAL', syscall: 'read' }

File was produced with
mysqldump --user xxx --password=yyy --host zzz mydatabase --compress=TRUE --add-drop-table --add-drop-trigger --add-locks -r mybackup.sql

Output filenames do not contain database name

It would be good it the filename of the output files also contained the database name. Now they only contain the dump file name and the table name.

If the dump file is named dump.sql and it contains two databases d1 and d2 both containing the a table with the same name my_table it will be a collision.

Current output
dump-my_table.sql

Suggested output
dump-d1-my_table.sql
dump-d2-my_table.sql

receiving { Error: EINVAL: invalid argument, read errno: -4071, code: 'EINVAL', syscall: 'read' }

Hi,

Just found your project and it would really solve a problem I'm having with being able to split a large dump file. Are you still supporting this branch ??

I'm currently getting the following error: { Error: EINVAL: invalid argument, read errno: -4071, code: 'EINVAL', syscall: 'read' } when trying to run the project. I do see that some libs are out of data, but currently can't take on updating the code myself.