gcp/bigquery support

[fippo]

gcp/bigquery support

[fippo]

note: https://stackoverflow.com/questions/50470044/syntax-error-expected-in-bigquery -- no varchar, no bigint, no real. easy to search-replace in features-v2.sql

[fippo]

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json -- this seems like the best way to import data, quite similar to how kinesis is writing to s3 and that gets loaded into redshift

[fippo]

so the UI for that is a bit weird... one can load into existing tables by creating a new table.
It looks like there is an issue with booean fields. redshift is happy to take 0/1 values but bigquery insists on boolean values. That will require making sure all features flagged as boolean return actual boolean values...

[fippo]

ah no... extract.js converts booleans:

    if (feature === false) feature = 0;
    if (feature === true) feature = 1;

I wonder if that is legacy or actually required for redshift. If it is then this should be moved to database.js

[fippo]

I think the best way to load it is to buffer 100 entries (or 60 seconds) or so as a line-json file and then use load() https://cloud.google.com/bigquery/docs/loading-data-local?hl=en#loading_data_from_a_local_data_source