Giter Site home page Giter Site logo

plv8x's Introduction

plv8x

Build Status

plv8x helps you manage functions and packages in plv8, postgresql's javascript procedural language support.

Quick start with docker

Using the docker-based postgresql with plv8js enabled:

% docker run -p 5433:5432 -d --name postgres clkao/postgres-plv8:9.4

% createdb -U postgres -h 127.0.0.1 -p 5433 test
% export PLV8XDB=postgres://[email protected]:5433/test

% plv8x --list
plv8x: 392.25 kB

# import the qs package from npm
% npm i qs; plv8x -i qs; plv8x --list
plv8x: 392.25 kB
qs: 9.37 kB

# this is now evaluated inside postgresql
% plv8x -e 'require("qs").parse("foo=bar&baz=1")'
{ foo: 'bar', baz: '1' }

# .. which  is actually equivalent to:
% psql $PLV8DB -c 'select |> $$ require("qs").parse("foo=bar&baz=1") $$'
        ?column?
-------------------------
 {"foo":"bar","baz":"1"}
(1 row)


Install plv8js

Note: Requires postgresql 9.0 or later.

postgresql PGDG apt respository now ships plv8js extension:

wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql-9.2-plv8

Or you can install with pgxnclient:

sudo easy_install pgxnclient
sudo pgxn install plv8

Install LiveScript (pre-requisite)

% npm i -g LiveScript

Install plv8x

% git clone git://github.com/clkao/plv8x.git; cd plv8x
% npm i -g .

Quick start

Enable plv8x for your database:

% createdb test
% plv8x -d test -l
plv8x: 491425 bytes

We support synonymous PLV8XDB and PLV8XCONN environment variables, so there's no need to type -d over and over again on the command line:

% export PLV8XDB=test

To connect with ident (local Unix user) authentication, specify the path to the socket directory with -d:

% plv8x -d /var/run/postgresql -l
plv8x: 491425 bytes

Now create some test data with json columns: (example table from Postgres 9.3 feature highlight: JSON operators)

% psql test
test=# CREATE TABLE aa (a int, b json);
CREATE TABLE
test=# INSERT INTO aa VALUES (1, '{"f1":1,"f2":true,"f3":"Hi I''m \"Daisy\""}');
INSERT 0 1
test=# INSERT INTO aa VALUES (2, '{"f1":{"f11":11,"f12":12},"f2":2}');
INSERT 0 1
test=# INSERT INTO aa VALUES (3, '{"f1":[1,"Robert \"M\"",true],"f2":[2,"Kevin \"K\"",false]}');
INSERT 0 1

Instead of b->'f1', we use b~>'this.f1', which means bind b as this and evaluate the right hand side (this.f1):

test=# SELECT b~>'this.f1' AS f1, b~>'this.f3' AS f3 FROM aa WHERE a = 1;
 f1 |         f3
----+--------------------
 1  | "Hi I'm \"Daisy\""

If you like coffee, @ works too:

test=# SELECT b~>'@f1' AS f1, b~>'@f3' AS f3 FROM aa WHERE a = 1;
 f1 |         f3
----+--------------------
 1  | "Hi I'm \"Daisy\""

For multiple keys, you can of course do b~>'@f1'~>'@f12', but single expression will do:

test=# SELECT b~>'@f1'~>'@f12' AS f12_long, b~>'@f1.f12' AS f12 FROM aa WHERE a = 2;
 f12_long | f12
----------+-----
 12       | 12

Ditto for arrays:

postgres=# SELECT b~>'@f1[0]' as f1_0 FROM aa WHERE a = 3;
f1_0
------
1

Unary ~> for just evaluating the expression:

test=# SELECT ~>'[1 to 10]' AS f1
           f1
------------------------
 [1,2,3,4,5,6,7,8,9,10]

~> is actually a shorthand for |> '~>...'. Using raw |> for plain old javascript:

test=# SELECT '{"foo": [1,2,3]}'::json |> 'function() { return this.foo[1] }';
 ?column?
----------
 2

Expression works too:

test=# SELECT '{"foo": [1,2,3]}'::json |> 'return this.foo[1]';
 ?column?
----------
 2

CoffeeScript:

test=# SELECT '{"foo": [1,2,3]}'::json |> '@foo[1]';
 ?column?
----------
 2

<| is |> reversed:

test=# SELECT '@foo.1 * 5' <| '{"foo": [1,2,3]}'::json
 ?column?
----------
 10

|> as unary operator:

test=# SELECT |> '~> plv8x.require "LiveScript" .compile "-> \Hello" {+bare}';
               ?column?
--------------------------------------
 "(function(){\n  return Hello;\n});"

Importing nodejs modules and creating user functions

Let's try reusing some existing npm modules:

% npm i -g qs
% plv8x -i qs # same as: plv8x -i qs:/path/to/qs/package.json
% psql test

# parse a query string
test=# SELECT |>'require("qs").parse("foo=bar&baz=1")' AS qs;
           qs
-------------------------
 {"foo":"bar","baz":"1"}

# actually use the parsed query string as json
test=# SELECT qs~>'@foo' AS foo FROM  (SELECT ~>'require("qs").parse("foo=bar&baz=1")' AS qs) a;
  foo
-------
 "bar"

# create a user function from qs so we don't have to require it:
% plv8x -f 'plv8x.json parse_qs(text)=qs:parse'
ok plv8x.json parse_qs(text)
# Now parse_qs is a postgresql function:
test=# SELECT parse_qs('foo=bar&baz=1') AS qs;
           qs
-------------------------
 {"foo":"bar","baz":"1"}

Calling conventions for user functions

We support both synchronous and async functions, as well as bare functions defined in module.exports.

By default, the first two arguments to an async (back-call) function is taken to be error and result respectively:

% plv8x -f 'fn(text):text=pkg:'           # out = pkg(x)
% plv8x -f 'fn(text):text=pkg:method'     # out = pkg.method(in)
% plv8x -f 'fn(text):text=pkg:<-'         # pkg(x, cb(err, out))
% plv8x -f 'fn(text):text=pkg:<-method'   # pkg.method(x, cb(err, out))

Using an underscore, one can specify exactly which async callback parameter to expect from the lifted function:

% plv8x -f 'fn(text):text=pkg:<-'         # pkg(x, cb(err, out))
% plv8x -f 'fn(text):text=pkg:_<-'        # pkg(x, cb(out))
% plv8x -f 'fn(text):text=pkg:,_<-'       # pkg(x, cb(_0, out))
% plv8x -f 'fn(text):text=pkg:,,_<-'      # pkg(x, cb(_0, _1, out))

License

MIT

plv8x's People

Contributors

audreyt avatar clkao avatar poga avatar shirkey avatar solidsnack avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

plv8x's Issues

test coverage

figure out a way to also collect coverage from code run in plv8 context

Auto-inject based on function annotation

For this module:

double.$plv8x = '(numeric):numeric'
export function double => (it * 2)

Upon import, one should be able to automatically derive a SQL-level injection based on the signature.

prelude versioning

we can store version in plv8x inside the plv8 context and use it to decide if we want to bootstrap and import prelude

Add support for JSONB.

pl/v8 can't seem to properly handle JSONB.

Test case

Having the following function:

CREATE OR REPLACE FUNCTION jsonb_test(data anyelement)
  RETURNS jsonb AS
    $BODY$
       return JSON.stringify(data);
    $BODY$
LANGUAGE plv8 STABLE STRICT;

When running:

SELECT jsonb_test('{"a":1, "b":{"c": 2}}'::json);
result: {"a": 1, "b": {"c": 2}}

will result in {"a": 1, "b": {"c": 2}} which is the expected result.

When running

SELECT jsonb_test('{"a":1, "b":{"c": 2}}'::jsonb);

we get "{\"a\": 1, \"b\": {\"c\": 2}}", which is not ideal.

Support for postgreSQL 10 and 11

For Postgresql 10 and 11 and plv8 2.3.9 (latest at the moment of writing), running plv8 -i after successfully running a plv8 --list returns:

2018-12-16 15:53:04.155 UTC [98] ERROR:  function "array_to_json" already exists with same argument types
2018-12-16 15:53:04.155 UTC [98] STATEMENT:
        SET client_min_messages TO WARNING;
        DO $PLV8X_EOF$ BEGIN

        DROP FUNCTION IF EXISTS array_to_json (__0 anyarray) CASCADE;
        EXCEPTION WHEN OTHERS THEN END; $PLV8X_EOF$;

        CREATE FUNCTION array_to_json (__0 anyarray) RETURNS plv8x.json AS $PLV8X__BODY__$
        ;
        return JSON.stringify((eval((function () {return (function(it){
          return it;
        }) })()))(__0));
        $PLV8X__BODY__$ LANGUAGE plv8 IMMUTABLE STRICT;
/plv8x/lib/index.js:29
        throw err;
        ^

Error: ERROR:  function "array_to_json" already exists with same argument types

    at module.exports.Client._readError (/plv8x/node_modules/pg-native/index.js:80:13)
    at module.exports.Client._read (/plv8x/node_modules/pg-native/index.js:121:19)
    at emitNone (events.js:106:13)
    at PQ.emit (events.js:208:7)

Under the hood

Hi,

Congratulations for this work. I need to understand what is going on under the hood? Could you provide information how this is working? Where is js files stored, how they are included in our functions etc?

Best Regards,

problem with long UTF8 char (such as 𨑨 U+28468 / UTF8:F0 A8 91 A8)

No idea of what's going wrong.
on a db named "twblg" encoded in UTF8.
data from moedict-data-twblg imported with

$ xzcat dump.sql.xz | psql --db twblg

I got the following behaviour with some long UTF8 char

This works:

$ plv8x -d twblg -c "SELECT * FROM entries WHERE 詞目 = '𨑨' ;"
[ { '主編號': '23001',
    '屬性代號': '2',
    '詞目': '𨑨',
    '音讀': 'tshit',
    '文白': '替',
    '部首': '辵',
    '部首序': '162-04-08',
    '方言差對應': '' } ]

This does not:

$ plv8x -d twblg -E "plv8.execute 'SELECT * FROM entries WHERE 詞目=\'𨑨\''"
[]

but this does works with ‘好‘:

$ plv8x -d twblg -E "plv8.execute 'SELECT * FROM entries WHERE 詞目=\'好\''"
[ { '主編號': '2282',
    '屬性代號': '1',
    '詞目': '好',
    '音讀': 'hó',
    '文白': '白',
    '部首': '女',
    '部首序': '038-03-06',
    '方言差對應': '[方]043' },
  { '主編號': '2283',
    '屬性代號': '1',
    '詞目': '好',
    '音讀': 'hònn',
    '文白': '文',
    '部首': '女',
    '部首序': '038-03-06',
    '方言差對應': '' } ]

(SHOW SERVER_ENCOGING returns UTF8)
(Debian wheezy)

support coffee-script

onejs-bundled coffee-script dist can't be loaded, as it requires fs during compilation.

Not able to import `qs` to run examples

I've tried both the OneJS (https://github.com/clkao/plv8x/tree/master) and the
Browserify (https://github.com/clkao/plv8x/tree/browserify) branches.

The OneJS branch results in:

postgres=# SELECT ~>'require("qs").parse("foo=bar&baz=1")';
WARNING:  failed to load module qs: Error: Cannot find module "qs" @[module: lib/index package: plv8x]
ERROR:  TypeError: Cannot call method 'parse' of undefined
DETAIL:  undefined() LINE 1:   return require("qs").parse("foo=bar&baz=1");

The Browserify branch results in:

postgres=# SELECT ~>'require("qs").parse("foo=bar&baz=1")';
WARNING:  failed to load module LiveScript:
ERROR:  TypeError: Cannot call method 'compile' of undefined
DETAIL:  boot() LINE 82:   return require('LiveScript').compile(expression, {

Yet the install appears to have succeeded:

postgres=# SELECT name FROM plv8x.code;
 name
-------
 qs
 plv8x

postgres@...:~$ plv8x -d /var/run/postgresql -l
qs: 12615 bytes
plv8x: 899907 bytes

This on a default install of Postgres 9.3 on Ubuntu 14.04 with Node v0.10.29
and NPM 1.4.14 (both of which were install via packages).

sourcemap support

now that we switch to browserify, we should be able to make use of sourcemap when reporting error.

plv8x -i qs:/path/to/qs/package.json - Does not work

The command in your example does not work as expected.
I'm running this:

git clone git://github.com/clkao/plv8x.git
cd plv8x
sudo npm i -g .
npm install LiveScript
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -l
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -i LiveScript:node_modules/LiveScript/package.json
npm install qs
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -i qs:node_modules/qs/package.json
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -l

It causes #23 to happen.

I did not want to install these packages globally with the -g flag, so I just ran npm install qs and pointed to the package.js.

What does work is this:

git clone git://github.com/clkao/plv8x.git
cd plv8x
sudo npm i -g .
sudo npm i -g LiveScript
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -l
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -i LiveScript
sudo npm i -g qs
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -i qs
plv8x -d postgresql://admin:[email protected]:5432/testplv8x -l

But that requires a global install of the packages I want to load.

WARNING: failed to load module qs: SyntaxError: Unexpected token }

I'm having an issue similar to #18 and #22.

$ plv8x -d testplv8x -l
plv8x: 392.33 kB
qs: 467 B
LiveScript: 467 B

Running the test queries:

testplv8x=# SELECT |> 'require("qs").parse("foo=bar&baz=1")';
WARNING:  failed to load module qs: SyntaxError: Unexpected token }
WARNING:  SyntaxError: Unexpected token }
    at plv8x.require (boot:27:23)
    at eval (eval at <anonymous> (json_eval:5:25), <anonymous>:1:20)
    at json_eval:5:48
    at null.<anonymous> (json_eval:6:8)
ERROR:  TypeError: Cannot call method 'parse' of undefined
DETAIL:  undefined() LINE 0: (function(){return require("qs").parse("foo=bar&baz=1")})
testplv8x=# SELECT qs~>'@foo' AS foo FROM  (SELECT ~>'require("qs").parse("foo=bar&baz=1")' AS qs) a;
WARNING:  failed to load module LiveScript: SyntaxError: Unexpected token }
WARNING:  SyntaxError: Unexpected token }
    at plv8x.require (boot:27:23)
    at compileLivescript (boot:83:10)
    at cls (boot:51:38)
    at Object.xpressionToBody (boot:57:12)
    at json_eval_ls:5:25
    at null.<anonymous> (json_eval_ls:6
ERROR:  TypeError: Cannot call method 'compile' of undefined
DETAIL:  boot() LINE 82:   return require('LiveScript').compile(expression, {

My Postgres and plv8 versions:

testplv8x=# SELECT version();
                                               version
------------------------------------------------------------------------------------------------------
 PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
(1 row)

testplv8x=# DO $$ plv8.elog(WARNING, 'plv8.version = ', plv8.version); $$ LANGUAGE plv8;
WARNING:  plv8.version =  1.4.0
DO

Using the latest plv8x:

~/plv8x$ git log
commit 3c19d57adfa5050c27715699d2369d2c441c817d
Author: Chia-liang Kao <[email protected]>
Date:   Thu Feb 19 01:30:27 2015 +0800

    quick start with docker-based plv8js-enabled pg

commit 0f5ee2f982029bb2c0c0a27fe38d67ad7d8481fb
Author: Chia-liang Kao <[email protected]>
Date:   Wed Feb 18 23:55:41 2015 +0800

    modernize ls-based bin

Unable to require most modules

I'm having an issue that looks very similar to #18. Any pointers on how to resolve?

# plv8x -d 'postgres://jason:<password>@localhost:5432/platinum33' -l

microtime: 9414 bytes
json-stringify-safe: 5041 bytes
qs: 16192 bytes
plv8x: 1038096 bytes

SELECT |> 'require("qs").parse("foo=bar&baz=1")' AS qs;

WARNING: failed to load module fs: Error: Cannot find module "./lib/" @[module: index package: qs]
WARNING: failed to load module qs: Error: Cannot find module "./lib/" @[module: index package: qs]

ERROR: TypeError: Cannot call method 'parse' of undefined
DETAIL: undefined() LINE 0: (function(){return require("qs").parse("foo=bar&baz=1")})
********** Error **********

ERROR: TypeError: Cannot call method 'parse' of undefined
SQL state: XX000
Detail: undefined() LINE 0: (function(){return require("qs").parse("foo=bar&baz=1")})

SELECT |> 'require("util").log("test")';

WARNING: failed to load module fs: Error: Cannot find module "fs" @[module: lib/index package: plv8x]
WARNING: failed to load module util: Error: Cannot find module "util" @[module: lib/index package: plv8x]

ERROR: TypeError: Cannot call method 'log' of undefined
DETAIL: undefined() LINE 0: (function(){return require("util").log("test")})
********** Error **********

ERROR: TypeError: Cannot call method 'log' of undefined
SQL state: XX000
Detail: undefined() LINE 0: (function(){return require("util").log("test")})

SELECT |> 'require("microtime").log("now")';

WARNING: failed to load module fs: Error: Cannot find module "fs" @[module: lib/index package: plv8x]
ERROR: TypeError: Cannot call method 'log' of undefined
DETAIL: undefined() LINE 0: (function(){return require("microtime").log("now")})
********** Error **********

ERROR: TypeError: Cannot call method 'log' of undefined
SQL state: XX000
Detail: undefined() LINE 0: (function(){return require("microtime").log("now")})

However, this one seems to work for some reason. I'm not sure if maybe it's because I have the syntax wrong fro this one.

SELECT |> 'require("json-stringify-safe")({"monkey":{"bot":"cat"}})'

"{"monkey":{"bot":"cat"}}"

missing cmd.js for global installation

sudo npm i -g .

gets

npm ERR! Error: ENOENT, chmod '/usr/lib/node_modules/plv8x/bin/cmd.js'
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[email protected]>

npm ERR! System Linux 3.11.0-12-generic
npm ERR! command "/usr/bin/node" "/usr/bin/npm" "i" "-g" "."
npm ERR! cwd /home/caasi/Documents/g0v/plv8x
npm ERR! node -v v0.10.21
npm ERR! npm -v 1.3.11
npm ERR! path /usr/lib/node_modules/plv8x/bin/cmd.js
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /home/caasi/Documents/g0v/plv8x/npm-debug.log
npm ERR! not ok code 0

but

npm i
sudo npm i -g .

works

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.