lemay-ai/airflow_gpu

airflow_gpu's Issues

version of docker-compose and depends_on issue

Hello there,
I try to follow your setting up, and I get some problems with docker-compose file, actually in the alias:

  &airflow-common
  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.3}
  # build: 
  #   context: ./
  #   dockerfile: ./airflow/Dockerfile
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__PARALLELISM: 8
    AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 1
    AIRFLOW__SCHEDULER__PARSING_PROCESSES: 1
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY}
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-docker==5.0.3}
  volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./airflow/logs:/opt/airflow/logs
    - ./airflow/plugins:/opt/airflow/plugins
    - /var/run/docker.sock:/var/run/docker.sock
    - ./requirements.txt:/opt/airflow/requirements.txt
    - ./scripts/:/opt/airflow/scripts
    - ./core:/opt/airflow/core
    - ./data/:/opt/airflow/data
    - ./configs:/opt/airflow/configs
    - ./save_model/:/opt/airflow/save_model
    # - $PWD/data:/data   
    # TODO: Add native MySQL or MySQL container
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    redis:
        condition: service_healthy
    postgres:
        condition: service_healthy

• At the first, I think the version should be '3.3' instead off '3'. The next, the depends_on get some errors like :

You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
services.airflow-scheduler.depends_on contains an invalid type, it should be a string
services.airflow-worker.depends_on contains an invalid type, it should be a string
services.airflow-webserver.depends_on contains an invalid type, it should be a string
services.airflow-scheduler.depends_on contains an invalid type, it should be a string
services.airflow-webserver.depends_on contains an invalid type, it should be a string
services.airflow-worker.depends_on contains an invalid type, it should be a string
services.airflow-init.depends_on contains an invalid type, it should be a string
services.airflow-init.depends_on contains an invalid type, it should be a string

I hope you will check it again and upgrade the version for compose or have some suitable solutions for this