Comments (5)
You're right, this is an "internal" method, and there's no guarantee that materializations will always work this way (creating a macro named materialization_<name>
behind the scenes).
Given that dbt Labs maintains this adapter, I feel more comfortable with it. In fact, I think it will be a good check if we ever change this behavior. It isn't exactly documented, so it isn't exactly contracted in v1—but we can fairly assume that there are users who have observed this capability, and now projects which rely on it.
from dbt-snowflake.
@ErikDataDesigner Thanks for opening! It looks like query tag configs aren't used on snapshots in Snowflake today.
The query tag config is turned on/off in Snowflake materializations via two macros: set_query_tag()
(start) and unset_query_tag()
(end). E.g. in the Snowflake table
materialization:
https://github.com/dbt-labs/dbt/blob/454168204c90464de1753ced824f5e3417e6203a/plugins/snowflake/dbt/include/snowflake/macros/materializations/table.sql#L3
https://github.com/dbt-labs/dbt/blob/454168204c90464de1753ced824f5e3417e6203a/plugins/snowflake/dbt/include/snowflake/macros/materializations/table.sql#L30
It looks like those macros aren't included in the seed
materialization or the snapshot
materializations, because in both cases Snowflake uses the default ("global") version.
The short-term answer here would be to copy-paste the materialization code from the global project into the Snowflake plugin, and add those macro calls at the top and bottom. I... don't really like that! The alternative would be to find a hook into the python code, either within the run/seed/snapshot task or the connection logic, where we can string these up by default. I'll do a little more digging, but I'm not so hopeful of my chances.
In the meantime, it is possible to set a default query_tag for all dbt-related queries in the connection profile, which will include queries related to snapshots.
from dbt-snowflake.
How about wrapping the standard behaviour of the materialisation? Like:
{% materialization snapshot, adapter='snowflake' %}
{% set original_query_tag = set_query_tag() %}
{% set relations = materialization_snapshot_default() %}
{% do unset_query_tag(original_query_tag) %}
{{ return(relations) }}
{% endmaterialization %}
And
{% materialization seed, adapter='snowflake' %}
{% set original_query_tag = set_query_tag() %}
{% set relations = materialization_seed_default() %}
{% do unset_query_tag(original_query_tag) %}
{{ return(relations) }}
{% endmaterialization %}
This do work just fine as a project-scoped materialisation.
I would gladly open a PR for this. But I'm struggling to get my head around the Contributions Guide for dbt adapters. The one linked it this repo seems to be out of date (but that might be worth discussed in a separate issue ;) )
from dbt-snowflake.
@anthu Glad to hear that works locally. It's a pretty elegant way to support query tags for seeds + snapshots on Snowflake, without requiring a ton of duplicated code. I'd be open to a PR for it.
We do need new contributing guides for the adapter repos. For the time being, you can get started by:
- forking/cloning this repo
- in a virtualenv:
pip install -e . -r dev_requirements.txt
(will install latest changes fromdbt-core
as well) - making changes to the code inside
dbt/
- running dbt from that virtualenv
from dbt-snowflake.
Thanks for the prompt reply.
I got it to run in the meanwhile - the tricky part is definitely setting up the Snowflake connection for the test to run - it simply refused to pick them up:
this worked python3 -m pytest -m profile_snowflake
instead simply running pytest.
I will make it ready later - what are your thought on this being more or less an "internal" method?
from dbt-snowflake.
Related Issues (20)
- [Feature] add functional test to cover new cover new custom config for tests.
- [Feature] metadata-based freshness checks for external tables
- [Bug] list_relations_without_caching & proxy & no-populate-cache
- [Bug] `not_null` constraint only has effect when specified at the column level HOT 1
- [Bug] Replace underscores with hyphens in account identifiers
- [Feature] Replace underscores with hyphens in account ids HOT 1
- [Bug] Seed file already exists when I run dbt-build with full refresh model HOT 4
- [Bug] `quoting` config not working with 1.8 HOT 3
- [Feature] Support JWT Authentication
- [Feature] Improve key-pair auth performance HOT 6
- [Tech Debt] Fix DT testing that should be automated
- [Feature] Refresh mode for dynamic tables HOT 4
- [Feature] transient dynamic tables
- [Bug][SPIKE] Investigate flaky test
- [Feature] dynamic tables: support macros in "snowflake_warehouse" config value HOT 4
- [Bug] macro `snowflake__get_columns_in_relation` does not handle VECTOR data types correctly
- [Bug] Use `show ... starts with` instead of `show ... like` in `_show_object_metadata`
- Full refresh on Dynamic table HOT 2
- [Feature] Create relations as case-sensitive by default
- [Feature] Optionally refresh dynamic tables when not rebuilding HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-snowflake.