Investigate options to apply MLEM models within Airflow DAGS

[iterative]

Investigate options to apply MLEM models within Airflow DAGS

[mike0sv]

For now we are building docker images only for serving, so server option is required and run command is mlem serve.
That can be changed, but for that we first need to understand what it means to build docker for batch scoring

[aguschin]

For the reference, there is Airflow extension that adds support for DVC operations: https://github.com/covid-genomics/airflow-dvc

[aguschin]

UPD: we can take a look at other orchestration tools (dagster, prefect, etc.)

[aguschin]

Posting here a short discussion with @mike0sv:

• mlem apply model hdfs://… should work already thanks to fsspec - need to check this though
• if map-reduce saved many files that should be merged, the above command may not work. We may implement special Reader in MLEM that will receive many artifacts, will merge them in a single dataframe before apply (or will output each file as a batch)
• to apply the models in spark, we can create a UDF for MLEM models
• we can think about supporting sparkml in MLEM
• we can implement export to spark UDF model that will start spark and run the data through it when you do mlem apply spark-udf-model … (could have a wow-effect)
• with hadoop we can implement Readers as well, that will start spark and read data in batches

To my mind, we need to get the working demo as simple as possible first. If we can work with mlem apply model hdfs://… - nice, this can be used. If not, downloading data as csv and using mlem apply model data.csv can work as well. @mnrozhkov mentioned they have a project with batch scoring in GitLab, we can start with using MLEM there.

[aguschin]

One issue I found: when you want to build a docker image for Batch Scoring scenario, you absolutely need to specify --server option. Why? Let's either specify fastapi as default so people won't need to confuse themselves about this, or allow to build without any server to skip installing extra dependencies like FastAPI people may not need.

UPD: it was an issue with my local MLEM installation. We don't need to specify --server. By default it's FASTAPI. Maybe good to give an option to skip installing it anyway, but this is not a priority now.

[aguschin]

For reference, a product similar to MLEM that does export to Airflow Pipelines. Need to take a deeper look at it https://docs.lineapy.org/en/latest/guide/build_pipelines/pipeline_basics.html