Comments (3)
@dwreeves Since you first posted this, we made some improvements in Cosmos, including using symbolic links.
Since Cosmos 1.2.4, the DbtLocalBaseOperator
creates symbolic links, as opposed to copying the full directory:
#660
It was based on the improvement we did on the LoadMode.DBT_LS
, released in Cosmos 1.1:
#488
Initially, we attempted to use flags/environment variables, but we noticed they did not work for a few versions of dbt that we currently support. PR #488 tries to give a bit more context on why we adopted symlinks. Do you think this approach addresses your original concerns?
from astronomer-cosmos.
This is a great writeup! I like this idea a lot. A couple notes/thoughts on my end:
You're right that the intent behind copying the directory was because of the read-only issue. The motivation behind the callback
is somewhat different, though - it's intended to support dbt docs generation. Today, that's the only use case I know of.
With dbt >= 1.5.2, we should definitely just create temp dirs for the target and logs directories. I think it's acceptable to use dbt version
for this check, despite the fact that it may add 1-2s to the execution time to run the command.
It may also be worth considering putting this behavior under a flag. If someone has an extremely large dbt project that they put in a write directory, and they're fine with writing to that directory, we should let them disable the cloning functionality. If we do this, and a user disables the cloning, we shouldn't need to check for their dbt version - which eliminates the 1-2 extra seconds.
Given that the main use case for the callback
function is to deal with the artifacts after running a dbt model, we may want to just modify the callback
function to take two/three arguments (path to logs, path to target, maybe also path to project dirs) instead of the one argument today (path to project dir). We're also working towards a 1.0 release right now, where it'll be (more) acceptable to make breaking changes. This should alleviate concerns with any changes we need to make to the callback
function.
from astronomer-cosmos.
Oh, that's a pretty clever solution for compatibility with older dbt versions. Excellent work!
My guess is that symlinks would solve the issue described in this ticket.
from astronomer-cosmos.
Related Issues (20)
- Decouple `LoadMode.AUTOMATIC` from `load()` method in `DbtGraph`
- Cosmos tasks randomly getting marked as zombie tasks HOT 5
- Fix hard to read AirflowException error logs from dbt command HOT 2
- bug please help me this is my project (ubuntu) HOT 6
- Permission issue with Cosmos cache in some restricted environments HOT 5
- [Bug]: Trial Issue using Bug report template HOT 1
- This a new feature request using Feature request Issue template HOT 1
- Link to dbt docs from task HOT 1
- [Cosmos CI] 2.7 unit tests taking a long time to run HOT 1
- Remove dependency on Pydantic
- Improve how we authorise non-commiters to run Cosmos integration tests
- Review dbt in Airflow Survey results
- Support deep linking in the DBT docs plugin HOT 5
- [Bug]: BigQuery profile target 'dev' invalid: 'schema' is a required property HOT 2
- [Docs]: Update outdated Data-Aware Scheduling docs
- Allow users to retrieve a model/seed/snapshot Dataset URI via a function
- Airflow using cosmos can't locate the dbt_project.yml file HOT 2
- Using EMR transient cluster with Cosmos HOT 1
- AthenaAccessKeyProfileMapping does not work as expected locally HOT 3
- [Bug] cosmos 1.4.1 does not create virtualenv when using `ExecutionMode.VIRTUALENV` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from astronomer-cosmos.