Giter Site home page Giter Site logo

node-sql-ddl-sync's Introduction

NodeJS SQL DDL Synchronization

Build Status

Install

npm install sql-ddl-sync

Dialects

  • MySQL
  • PostgreSQL
  • SQLite

About

This module is part of ORM. It's used synchronize model tables in supported dialects. Sorry there is no API documentation for now but there are a couple of tests you can read and find out how to use it if you want.

Example

Install orm & the required driver (eg: mysql). Create a file with the contents below and change insert your database credentials. Run once and you'll see table ddl_sync_test appear in your database. Then make some changes to it (add/drop/change columns) and run the code again. Your table should always return to the same structure.

var orm   = require("orm");
var mysql = require("mysql");
var Sync  = require("sql-ddl-sync").Sync;

orm.connect("mysql://username:password@localhost/database", function (err, db) {
	if (err) throw err;
	var driver = db.driver;

	var sync = new Sync({
		dialect : "mysql",
		driver  : driver,
		debug   : function (text) {
			console.log("> %s", text);
		}
	});

	sync.defineCollection("ddl_sync_test", {
    id     : { type: "serial", key: true, serial: true },
    name   : { type: "text", required: true },
    age    : { type: "integer" },
    male   : { type: "boolean" },
    born   : { type: "date", time: true },
    born2  : { type: "date" },
    int2   : { type: "integer", size: 2 },
    int4   : { type: "integer", size: 4 },
    int8   : { type: "integer", size: 8 },
    float4 : { type: "number",  size: 4 },
    float8 : { type: "number",  size: 8 },
    photo  : { type: "binary" }
  });

	sync.sync(function (err) {
		if (err) {
			console.log("> Sync Error");
			console.log(err);
		} else {
			console.log("> Sync Done");
		}
		process.exit(0);
	});
});

PostgreSQL UUID

{ type: 'uuid', defaultExpression: 'uuid_generate_v4()' }

Test

To test, first make sure you have development dependencies installed. Go to the root folder and do:

npm install

Then, just run the tests.

npm test

If you have a supported database server and want to test against it, first install the module:

# if you have a mysql server
npm install mysql
# if you have a postgresql server
npm install pg

And then run:

node test/run-db --uri 'mysql://username:password@localhost/database'

node-sql-ddl-sync's People

Contributors

aeppert avatar dresende avatar dxg avatar harish2704 avatar killroy42 avatar locke avatar nicholasf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-sql-ddl-sync's Issues

TypeError: cannot call method 'query' of undefined

I'm trying to use your library, but I can't seem to get it working. I've got the following code in a file named db_sync.js and I've been running it with node db_sync.

As far as I can tell all of the correct modules are installed, but I feel like I'm missing something.

My code:

var orm    = require("orm");
var mysql  = require("mysql");
var Sync   = require("sql-ddl-sync").Sync;

var mysqlUrl = "mysql://username:password@host/database"; //this actually points to my database on ClearDB.

orm.connect(mysqlUrl, function (err, db) {
  if (err) throw err;

  var driver = db.driver;

  var sync = new Sync({
    dialect: "mysql",
    driver: driver,
    debug: function (text) {
      console.log("> %s", text);
    }
  });

  sync.defineCollection("person", {
    firstName: String,
    lastName: String,
    age: Number
  });


  sync.sync(function (err) {
    if (err) {
      console.log("> Sync Error");
      console.log(err);
    } else {
      console.log("> Sync Done");
    }
    process.exit(0);
  });
});

This is the error I get:

TypeError: Cannot call method 'query' of undefined
    at Object.exports.hasCollection (/Users/calekennedy/Projects/node-orm2-example/node_modules/sql-ddl-sync/lib/Dialects/mysql.js:10:5)
    at Sync.processCollection (/Users/calekennedy/Projects/node-orm2-example/node_modules/sql-ddl-sync/lib/Sync.js:16:11)
    at processNext (/Users/calekennedy/Projects/node-orm2-example/node_modules/sql-ddl-sync/lib/Sync.js:372:5)
    at Object.Sync.sync (/Users/calekennedy/Projects/node-orm2-example/node_modules/sql-ddl-sync/lib/Sync.js:383:11)
    at /Users/calekennedy/Projects/node-orm2-example/db_sync.js:36:8
    at Handshake._callback (/Users/calekennedy/Projects/node-orm2-example/node_modules/orm/lib/ORM.js:129:13)
    at Handshake.Sequence.end (/Users/calekennedy/Projects/node-orm2-example/node_modules/mysql/lib/protocol/sequences/Sequence.js:78:24)
    at Handshake.Sequence.OkPacket (/Users/calekennedy/Projects/node-orm2-example/node_modules/mysql/lib/protocol/sequences/Sequence.js:87:8)
    at Protocol._parsePacket (/Users/calekennedy/Projects/node-orm2-example/node_modules/mysql/lib/protocol/Protocol.js:202:24)
    at Parser.write (/Users/calekennedy/Projects/node-orm2-example/node_modules/mysql/lib/protocol/Parser.js:62:12)

Add PostgreSQL support

Hi @dxg , I added you because it seems you use more postgresql than me. I'm going to add this, just wanted to tell you about this project in case you want to help :)

Bug with indexes and mapsTo

If attributes are set as unique and they have a mapsTo property, the mapsTo is ignored and obviously MySQL gives an error that key doesn't exist.

Adding the following on line 229 in the Sync.js fixes it
k = collection.properties[k].mapsTo;

processKeys issue in sqlite dialect

I opened an issue on node-orm2 (which consumes this module) but it never received any attention.

There seems to be a line that keeps a custom key type from being used. If I set the id as a string it does not make the column a primary key.

In my local environment i removed this check and it seemed to pass the tests. I can't tell what the need is for this line, can someone tell me what the purpose of it is, and why it needs to be here?

If there is no reason I can submit a pull request to remove it.

General question

Hi Dresende,

Nice work here. One thing I don't understand is how to apply data transformations when a DDL schema changes (to support production data changes).

For example, take the name column in the README's example.

name   : { type : "text", required: true }

What if a requirement meant that this should split into 'first_name' and 'last_name' and these two new columns should use the data in name? So the name column would be split into tokens and the last token becomes last_name and the first token becomes the first_name. Let's say this code is in a helper module that exports nameTransformer().

first_name   : { type : "text", required: true }
last_name   : { type : "text", required: true }

How would we manage to call nameTransformer() in the lifecycle of dropping the name column and creating the two new columns?

Cheers,
Nicholas

sqlite dialect: getCollectionProperties error

/[...]/migrate-orm2/node_modules/sql-ddl-sync/lib/Dialects/sqlite.js:58
					column.defaultValue = m[0];
					                       ^
TypeError: Cannot read property '0' of null
    at Statement.<anonymous> (/[...]/migrate-orm2/node_modules/sql-ddl-sync/lib/Dialects/sqlite.js:58:29)

replacing the line with column.defaultValue = dCol.dflt_value; seems to work.

Create adds autoincrement fields

HI,
I'm probably doing something simple wrong but i have a model (postgres) definition:

db.define('tbl_subscribers', {
subnumber:Number,
subfirstname: String,
sublastname: String,
subtype_id: Number,
subemail: String,
// etc.
},
{
id: 'subnumber',
});

subnumber is the id field, defined as
subnumber integer NOT NULL DEFAULT nextval(('subid_seq'::text)::regclass),

Every time i try to create sending a req.body such as

{ subemail: '[email protected]',
subfirstname: 'JOHN',
suburb: { suburb: 'DOMVILLE', id: 9683, state: 'QLD', postcode: 4357 },
suburb_id: 4357,
sublastname: 'DOE',
program_id: 1,
subtype_id: 2 }

I get the error:
[error: null value in column "subnumber" violates not-null constraint]

sql dump:
INSERT INTO "tbl_subscribers" ("subemail", "subfirstname", "suburb_id", "sublastname", "program_id", "subtype_id", "subnumber") VALUES ('[email protected]', 'JOHN', 4357, 'DOE', 1, 2, NULL)

So how do I remove subnumber from the sql query?

Thanks, D

sync.js in node-sql-ddl-sync

At file: node-sql-ddl-sync / lib / Sync.js text is
function Sync(options) {
....
var Dialect = require("./Dialects/" + driver.dialect);
...}

An error occurs when I use driver for sqlite: driver.dialect is undefined.
So, i think that "driver.dialect" should be "options.dialect".
Is it right?

Bug when trying to add unique index from node-orm2

Using node-orm2 and this model definition:

db.define('user', {
    email:          { type: 'text', size: 40, required: true, unique: 'email' }
});

After calling db.syncPromise() I get this error:

{ Error: ER_NO_SUCH_TABLE: Table 'shop.undefined' doesn't exist
    at Query.Sequence._packetToError (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
    at Query.ErrorPacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
    at Protocol._parsePacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:279:23)
    at Parser.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:39:16)
    at Socket.<anonymous> (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:103:28)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:548:20)
    --------------------
    at Protocol._enqueue (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:145:48)
    at PoolConnection.query (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:208:25)
    at /home/bob/git/study/node-shop/server/node_modules/orm/lib/Drivers/DML/mysql.js:231:9
    at Handshake.onConnect (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Pool.js:64:7)
    at Handshake.Sequence.end (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
    at Handshake.Sequence.OkPacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:97:8)
    at Protocol._parsePacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:279:23)
    at Parser.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:39:16)
    at Socket.<anonymous> (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:103:28)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:548:20)
  cause: 
   { Error: ER_NO_SUCH_TABLE: Table 'shop.undefined' doesn't exist
       at Query.Sequence._packetToError (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:52:14)
       at Query.ErrorPacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
       at Protocol._parsePacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:279:23)
       at Parser.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Parser.js:76:12)
       at Protocol.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:39:16)
       at Socket.<anonymous> (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:103:28)
       at emitOne (events.js:96:13)
       at Socket.emit (events.js:188:7)
       at readableAddChunk (_stream_readable.js:176:18)
       at Socket.Readable.push (_stream_readable.js:134:10)
       at TCP.onread (net.js:548:20)
       --------------------
       at Protocol._enqueue (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:145:48)
       at PoolConnection.query (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:208:25)
       at /home/bob/git/study/node-shop/server/node_modules/orm/lib/Drivers/DML/mysql.js:231:9
       at Handshake.onConnect (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Pool.js:64:7)
       at Handshake.Sequence.end (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:88:24)
       at Handshake.Sequence.OkPacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/sequences/Sequence.js:97:8)
       at Protocol._parsePacket (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:279:23)
       at Parser.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Parser.js:76:12)
       at Protocol.write (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/protocol/Protocol.js:39:16)
       at Socket.<anonymous> (/home/bob/git/study/node-shop/server/node_modules/mysql/lib/Connection.js:103:28)
       at emitOne (events.js:96:13)
       at Socket.emit (events.js:188:7)
       at readableAddChunk (_stream_readable.js:176:18)
       at Socket.Readable.push (_stream_readable.js:134:10)
       at TCP.onread (net.js:548:20)
     code: 'ER_NO_SUCH_TABLE',
     errno: 1146,
     sqlMessage: 'Table \'shop.undefined\' doesn\'t exist',
     sqlState: '42S02',
     index: 0,
     sql: 'DROP INDEX `user` ON `undefined`',
     model: 'user' },
  isOperational: true,
  code: 'ER_NO_SUCH_TABLE',
  errno: 1146,
  sqlMessage: 'Table \'shop.undefined\' doesn\'t exist',
  sqlState: '42S02',
  index: 0,
  sql: 'DROP INDEX `user` ON `undefined`',
  model: 'user' }

Deeper investigation have led me through node-orm2 sources into this package and file https://github.com/dresende/node-sql-ddl-sync/blob/master/lib/Dialects/mysql.js#L294, to the function convertIndexRows. It gets an sql query result as the only parameter and tries to convert it into hash containing info about indexes existing on a table. But as I noticed, it deals with wrong column names in result.

Literally, it queries SELECT index_name, column_name, non_unique FROM information_schema.statistics WHERE table_schema = ? AND table_name = ? with params ["shop","user"] (aka [driver.config.database, name]). Typing the query by hand into mysql promt given this output:

mysql> SELECT index_name, column_name, non_unique FROM information_schema.statistics WHERE table_schema = 'shop' AND table_name = 'user';
+------------+-------------+------------+
| INDEX_NAME | COLUMN_NAME | NON_UNIQUE |
+------------+-------------+------------+
| email      | email       | 0          |
| PRIMARY    | id          | 0          |
+------------+-------------+------------+
2 rows in set (0.00 sec)

As you can see, here are column names in upper case, while in the function convertIndexRows you are indexing them in lower case like rows[i].index_name. Correcting every occurrence of index_name, column_name and non_unique inside this function to their capitalized version solved my issue. Is this a typo or I configured something wrong on my machine?

As a result of original version of function, I got this:

{ undefined: { columns: [ undefined ], unique: false } }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.