aws-containers / airflow-on-fargate Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT No Attribution
License: MIT No Attribution
Hi - even with managed airflow available on AWS, I still see a need for this repo. I have needs to customize around airflow that I can't do with managed airflow. I would like to start tagging releases in the repo with upgraded airflow versions. I would also like to do some other upgrades like pscopg2 instead of pygresql. Let me know if I can be a maintainer, I am happy to submit PRs without being a maintainer, but would want tagging as we advance this repo so people can always use older versions of the repo as they need to. (older airflow versions as an example)
Our team was trying to setup Airflow through the following steps via CDK:
https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/
However, during the cdk deploy
step, and after answering "yes" to the prompt, we are encountering the following error:
#5 [2/7] RUN apt-get update && apt-get install -y python3-pip libcurl4-gnutls-dev librtmp-dev python3-dev libpq-dev
#5 sha256:787088182aea3ef729295dade2fab650429db33d33b8265ed2abda11ad49abff
#5 0.327 + apt-get update
#5 0.568 Get:1 http://security.debian.org/debian-security buster/updates InRelease [34.8 kB]
#5 0.580 Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
#5 0.633 Get:3 http://deb.debian.org/debian buster-updates InRelease [56.6 kB]
#5 0.715 Get:4 http://repo.mysql.com/apt/debian stretch InRelease [21.6 kB]
#5 0.769 Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [508 kB]
#5 0.916 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7909 kB]
#5 1.216 Get:7 http://deb.debian.org/debian buster-updates/main amd64 Packages [8788 B]
#5 1.227 Err:4 http://repo.mysql.com/apt/debian stretch InRelease
#5 1.227 The following signatures were invalid: EXPKEYSIG 8C718D3B5072E1F5 MySQL Release Engineering <[email protected]>
#5 2.403 Reading package lists...
#5 3.271 W: GPG error: http://repo.mysql.com/apt/debian stretch InRelease: The following signatures were invalid: EXPKEYSIG 8C718D3B5072E1F5 MySQL Release Engineering <[email protected]>
#5 3.271 E: The repository 'http://repo.mysql.com/apt/debian stretch InRelease' is not signed.
#5 ERROR: executor failed running [/bin/bash -o pipefail -e -u -x -c apt-get update && apt-get install -y python3-pip libcurl4-gnutls-dev librtmp-dev python3-dev libpq-dev]: exit code: 100
------
executor failed running [/bin/bash -o pipefail -e -u -x -c apt-get update && apt-get install -y python3-pip libcurl4-gnutls-dev librtmp-dev python3-dev libpq-dev]: exit code: 100
โ FarFlow failed: Error: Failed to publish one or more assets. See the error messages above for more information.
at publishAssets (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:391:74837)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async deployStack (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:393:390)
at async Object.deployStack2 [as deployStack] (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:400:149977)
at async /Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:400:135508
Do you have any insight as to how to solve this problem? Thanks!
This CDK code will try to create new VPC. It also should support existing VPC.
Hello,
It looks like you allow Celery to create SQS queue outside of CloudFormation stack. As for me it doesn't look good since you lose control towards the resource. Why don't you create the queue with CloudFormation and pass the url of the queue to Celery? I didn't find the information how to do that (like which parameters to use). Maybe you could show an example how to do that?
Thanks.
It's cool to run the workload on Serverless Fargate, but why not use Aurora Serverless to replace the RDS? so the total solution would be serverless, is that a good idea?
Are there any python forks for this? And on AWS CDK 2.x as well as Airflow:latest.
Thanks - Sandeep
I am fairly new to AWS and Airflow, I managed to deploy airflow using this repo and I was wondering how I sync my dags from s3 or github to the running containers?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.