Slimming down tmp_dir to just the --target-dir

[dwreeves]

Slimming down tmp_dir to just the --target-dir

[tatiana]

@dwreeves Since you first posted this, we made some improvements in Cosmos, including using symbolic links.

Since Cosmos 1.2.4, the DbtLocalBaseOperator creates symbolic links, as opposed to copying the full directory:
#660

It was based on the improvement we did on the LoadMode.DBT_LS, released in Cosmos 1.1:
#488

Initially, we attempted to use flags/environment variables, but we noticed they did not work for a few versions of dbt that we currently support. PR #488 tries to give a bit more context on why we adopted symlinks. Do you think this approach addresses your original concerns?

[jlaneve]

This is a great writeup! I like this idea a lot. A couple notes/thoughts on my end:

You're right that the intent behind copying the directory was because of the read-only issue. The motivation behind the callback is somewhat different, though - it's intended to support dbt docs generation. Today, that's the only use case I know of.

With dbt >= 1.5.2, we should definitely just create temp dirs for the target and logs directories. I think it's acceptable to use dbt version for this check, despite the fact that it may add 1-2s to the execution time to run the command.

It may also be worth considering putting this behavior under a flag. If someone has an extremely large dbt project that they put in a write directory, and they're fine with writing to that directory, we should let them disable the cloning functionality. If we do this, and a user disables the cloning, we shouldn't need to check for their dbt version - which eliminates the 1-2 extra seconds.

Given that the main use case for the callback function is to deal with the artifacts after running a dbt model, we may want to just modify the callback function to take two/three arguments (path to logs, path to target, maybe also path to project dirs) instead of the one argument today (path to project dir). We're also working towards a 1.0 release right now, where it'll be (more) acceptable to make breaking changes. This should alleviate concerns with any changes we need to make to the callback function.

[dwreeves]

Oh, that's a pretty clever solution for compatibility with older dbt versions. Excellent work!

My guess is that symlinks would solve the issue described in this ticket.