haleemur/dbt-athena-dandandan

dbt adapter for Athena

dbt-athena-dandandan's Issues

Feature: Materialization for `INSERT INTO`

An athena limitation is that at most 100 (partitions + buckets) are supported by CTAS. An extension to this work is to incorporate INSERT INTO ... with CTAS so greater numbers of partitions can be supported.

This could probably be done in a 4-stage operation:

1. get the list of all partitions-buckets combinations (may require the user to specify a query to get partitions, e.g. select distinct field_list from from-clause where where-clause)
2. create batches of 100 partitions / buckets
3. for first batch issue a CTAS
4. for all batches following, issue an INSERT INTO

First, we should explore the possible usecases for INSERT INTO in the dbt context