hussein-awala/airflow-server

Docker configuration for airflow server with Localexecutor

Hi there, I'm Hussein

About Myself: 👨🏼‍💻 I'm a Senior Data Engineer within the Ads Network team at Voodoo in Paris, and an Apache committer and PMC member at Apache Airflow 💡 I design, develop and maintain data platforms, especially the modern lakehouse architectures and the stream processing applications 🔬 I am responsible for serving and improving the performance of the ML models and the feature store 🔒 I ensure the security of user data on the data platform, in compliance with the regulations in force (GDPR, e-privacy) 🤝🏻 I contribute to different popular open-source projects (Airflow, Iceberg, Hudi, ...) and my own open-source projects (spark-on-k8s, airflow-duckdb, and async-batcher) ⚡ Fun fact: I am always seeking new opportunities to learn. Also, I love to cook, swim and watch movies or TV series.

Projects I am working on currently: Improving the performance of a real-time bidding system by implementing a batching mechanism to reduce the resource consumption and the latency, and by optimizing the feature store structure and data freshness Designing and developing a new lakehouse architecture using spark, iceberg, airflow, dbt and S3, to store the company data in a modern and cheap data store respecting the GDPR

Some frameworks I am working with currently 💻:

Some of my GitHub Stats:

Connect with me:

airflow-server's Issues

How airflowinit service communicates with the other services?

Since all services are completely separated containers, how does what airflow-init service (container) does affects the other two (webservice and scheduler) if they're separated, creating files each one on its own file system and etc?

Airflowinit creates the user and password, rigth, but how does it makes a difference on the others? There's no clear communication between then, their folders or files. Even their volumes are pointing to different folders.

How airflowinit makes a difference since it seems to be creating an user in a container that never executes airflow inside it? (this is done on the others by using "command: airflow webserver, scheduler etc")

So, how the other containers can know that there's a default user created if they're isolated from airflowinit's container and their folders aren't linked by volumes?