nmb10/ambryfdw

This FDW (Foreign Data Wrapper) can be used to access data stored in the ambry partitions packed with msgpack.

DEPRECATED: Merged into ambry_sources . Also it is broken because format of the packed partition files changed.

Ambry partition msgpack files Foreign Data Wrapper (multicorn)

This FDW can be used to access data stored in the ambry partitions packed with msgpack. Each column defined in the table will be mapped, in order, against columns in the ambry partition file.

Installation.

Install multicorn:

wget https://github.com/Kozea/Multicorn/archive/v1.2.3.zip
unzip v1.2.3.zip
cd Multicorn-1.2.3
make && sudo make install

Install ambryfdw:

pip install ambryfdw

Run tests.

python setup.py test

Options

filename (required)

The full path to the gzipped partition file containing the data in msgpack format. This file must be readable to the postgres user.

Usage

Supposing you want to parse the following msgpack partition file, located in /tmp/test.msg::

Year,Make,Model,Length
1997,Ford,E350,2.34
2000,Mercury,Cougar,2.38

You can declare the following table:

    CREATE SERVER partition_srv foreign data wrapper multicorn options (
        wrapper 'ambryfdw.PartitionMsgpackForeignDataWrapper'
    );

    CREATE FOREIGN TABLE partition_test (
           year numeric,
           make character varying,
           model character varying,
           length numeric
    ) SERVER partition_srv OPTIONS (
           filename '/tmp/test.msg');

Now you can query ambry partition

    SELECT * FROM partition_test;

     year |  make   | model  | length
    ------+---------+--------+--------
     1997 | Ford    | E350   |   2.34
     2000 | Mercury | Cougar |   2.38
    (2 lines)

ambryfdw's Issues

Ambry changed partition file format. Update wrapper.

See CivicSpleen/ambry_sources#1 for details.

ambryfdw should check packed partitions format and fail if format does not match.