use presto?

[fippo]

use presto?

[ggarber]

I think what we have now is more a pipeline where data passes through different stages than different databases to query/join together. Maybe I missunderstood prestodb. How do you think we could use it?

BTW, I'm starting to use Amazon Data Pipelines, where you define/schedule the transformations (now Dynamodb->Redshift but I'm trying to export it to a .csv in S3 too).

[fippo]

what I found useful was the ability to pull in the raw data (in our case: the log from s3), process it (extract features) and then get the result as something sql-ish.

not sure if presto really does that.

[ggarber]

That is what you can do with with Amazon Data Pipelines and with Amazon Lambda. I started using Lambda for Snoop but I'm not using it now.

Amazon Lambda: receives an event every time there is a new S3 entry and executes a script that can process it and store it wherever you want.

Amazon Data Pipeline: you schedule a process with N Data Nodes (S3, Dynamo, Redshift...) and Activities between them executing any processing you want with MApReduce like operations.