Comments (1)
Adding a bit more context.
The issue occur also for simple models like:
{{ config(
materialized='incremental',
incremental_strategy='merge',
unique_key='user_id',
table_type='iceberg',
force_batch='true'
) }}
select 1 as user_id, 'a' as user_name
The issue seems to be in get_partition_batches and create_table_as_with_partitions
Here are some possible solutions:
- if the underlying dataset is empty - create_table_as_with_partitions can create an empty table from SQL - no hurt/no pain - it's possible to check if __tmp_not_partitioned is empty and if it's the case, we just create an empty relation as tmp table.
- if the user try to use force_batch with not partitioned tables -> we have 2 options:
- all works normally and force_batch it's just ignored, e.g. pass partitioning(bool) to safe_create_table_as
- we raise an error and ask the user to remove force_batch from not partitioned tables
- if the user works with partitioned tables by the data is not more than 100 partitions partition batches everything should still work. - must be tested
@svdimchenko FYI.
from dbt-athena.
Related Issues (20)
- Implement support for `dbt-metricflow` HOT 1
- Config order for Iceberg tables HOT 4
- model contract enforce is not done at compile time HOT 8
- Add issue templates
- seeds with columns starting with underscore fail HOT 2
- Unable to compile test models in Athena HOT 1
- Add parameter `detailed_table_type` for `list_relations_without_caching` function
- Support column level statistics HOT 4
- Add default_lf_inherited_tags
- [Communication] Install Slack on the repository to enable release subscription HOT 4
- Prune old table version for incremental models. HOT 1
- Athena partitions limit fix (#360) fails with partitions defined as non-Athena functions HOT 4
- Log the query id HOT 2
- Post hook when too many partitions failed HOT 3
- Allow dbt-athena to grant permissions via lakeformation named resource method
- `external_location` ignored when `table_type='hive'` HOT 5
- Support multi-engine views in Athena
- Fine-Grain Permissions make re-creating tables difficult. HOT 9
- (Potential) bug with incremental iceberg tables HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-athena.