Comments (5)
I updated the PR, I tested with SSO user, it works now, let me know if it works for you.
There was indeed an issue with the aws_profile_name set as blank string instead of None ...
"aws_profile_name": os.getenv("DBT_TEST_ATHENA_AWS_PROFILE_NAME") or None
with an empty env variable works for me.
from dbt-athena.
DBT_TEST_ATHENA_AWS_PROFILE_NAME
is optional : indeed if it is not passed, the value will be None and then it is the same as not putting the value, leading to AthenaConnection having default profile.
from dbt-athena.
DBT_TEST_ATHENA_AWS_PROFILE_NAME
is optional : indeed if it is not passed, the value will be None and then it is the same as not putting the value, leading to AthenaConnection having default profile.
But I don't use any profile because I export AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, AWS_SESSION_TOKEN
directly in the terminal. This works well with dbt run
or dbt compile
.
It works if I comment out `"aws_profile_name": os.getenv("DBT_TEST_ATHENA_AWS_PROFILE_NAME"),
It will be needed if make test
is executed by a job that assumes an IAM Role instead of having the credentials of IAM User.
from dbt-athena.
@mattiamatrix I've your same setup, I'm assuming a role using sso. I don't think that we need to change anything here, as in the cases where we use AWS_ env (doesn't matter how they are set), we can just ignore to setup the DBT_TEST_ATHENA_AWS_PROFILE_NAME.
from dbt-athena.
mmh, @nicor88 can you run make test
locally? I can't with the current setup. Let me know if you can reproduce.
My .env
looks like this make test
fails.
DBT_TEST_ATHENA_S3_STAGING_DIR=s3://athena-output-results-eu-west-2
DBT_TEST_ATHENA_REGION_NAME=eu-west-2
DBT_TEST_ATHENA_DATABASE=awsdatacatalog
DBT_TEST_ATHENA_SCHEMA=athena_dbt_tests
DBT_TEST_ATHENA_WORK_GROUND=primary
# DBT_TEST_ATHENA_AWS_PROFILE_NAME=
I believe this is because
@pytest.fixture(scope="class")
def dbt_profile_target():
return {
"type": "athena",
"s3_staging_dir": os.getenv("DBT_TEST_ATHENA_S3_STAGING_DIR"),
"schema": os.getenv("DBT_TEST_ATHENA_SCHEMA"),
"database": os.getenv("DBT_TEST_ATHENA_DATABASE"),
"region_name": os.getenv("DBT_TEST_ATHENA_REGION_NAME"),
"threads": 1,
"num_retries": 0,
"work_group": os.getenv("DBT_TEST_ATHENA_WORK_GROUND"),
"aws_profile_name": os.getenv("DBT_TEST_ATHENA_AWS_PROFILE_NAME"),
}
if DBT_TEST_ATHENA_AWS_PROFILE_NAME
is not defined, it returns
"aws_profile_name": None
But, this makes AthenaCredentials
or boto3
look for a default
profile that doesn't exist in my setup.
I can make it works with
@pytest.fixture(scope="class")
def dbt_profile_target():
return {
"type": "athena",
"s3_staging_dir": os.getenv("DBT_TEST_ATHENA_S3_STAGING_DIR"),
"schema": os.getenv("DBT_TEST_ATHENA_SCHEMA"),
"database": os.getenv("DBT_TEST_ATHENA_DATABASE"),
"region_name": os.getenv("DBT_TEST_ATHENA_REGION_NAME"),
"threads": 1,
"num_retries": 0,
"work_group": os.getenv("DBT_TEST_ATHENA_WORK_GROUND"),
# "aws_profile_name": os.getenv("DBT_TEST_ATHENA_AWS_PROFILE_NAME"),
}
from dbt-athena.
Related Issues (20)
- model contract enforce is not done at compile time HOT 8
- Add issue templates
- seeds with columns starting with underscore fail HOT 2
- Unable to compile test models in Athena HOT 1
- Add parameter `detailed_table_type` for `list_relations_without_caching` function
- Support column level statistics HOT 4
- Add default_lf_inherited_tags
- When using `force_batch=true` with incremental models, it will fail if there is no data to write HOT 1
- [Communication] Install Slack on the repository to enable release subscription HOT 4
- Prune old table version for incremental models. HOT 1
- Athena partitions limit fix (#360) fails with partitions defined as non-Athena functions HOT 4
- Log the query id HOT 2
- Post hook when too many partitions failed HOT 3
- Allow dbt-athena to grant permissions via lakeformation named resource method
- `external_location` ignored when `table_type='hive'` HOT 5
- Support multi-engine views in Athena
- Fine-Grain Permissions make re-creating tables difficult. HOT 9
- (Potential) bug with incremental iceberg tables HOT 6
- [Feature] set external_location for iceberg table HOT 1
- [Feature] Introduce ha flag for table materialisation and iceberg tables to allow users to control final location
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-athena.