mehdi-elion/mlflow-postgres-minio

Automatic setup and deploy a MLflow server

MLflow Server Automatic Setup

Short description

---

Automatic setup and deploy a MLflow server. This includes:

• A MLflow server
• A Postgres database to store MLflow metadata, such as:
  • experiment data;
  • run data;
  • model registry data.
• Minio, an open-source object storage system, to store and retrieve large files, such as:
  • model artifacts;
  • data sets.

Pre-requisites

---

Ubuntu with the following installed:

• Miniconda3
• Docker
• Docker Compose

You could use WSL2 on a Windows machine, as an alternative to an Ubuntu machine.

Setup your machine (local or remote server)

---

1. Install pyenv

Pyenv is used with MLflow to manage different Python versions and packages in isolated environments.

• Remove previous installations (optional):

rm -rf ~/.pyenv

• Install any necessary package:

sudo apt-get update -y
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

• Automatic install pyenv:

curl https://pyenv.run | bash

• Edit the ~/.bashrc file to recognize pyenv

sudo nano ~/.bashrc

• And copy/paste the following line into the end of the file:

# Config for PyEnv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

• Save/close (Crtl-X > y > Enter), and then refresh the ~/.bashrc file:

source ~/.bashrc

2. Create an MLflow environment:

• Create and activate your MLflow experiment environment:

conda create -n mlflow_env python=3.10
conda activate mlflow_env
pip install pandas scikit-learn mlflow[extras]

• Add these to the environment; edit to your own preferred secrets:

export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123
export MLFLOW_S3_ENDPOINT_URL=http://localhost:9000

3. Launch Docker Containers

• Clone this repo and navigate inside:

git clone https://github.com/pandego/mlflow-postgres-minio.git
cd ./mlflow-postgres-minio

• Edit default.env to your own preferred secrets:
• Launch the docker-compose command to build and start all containers needed for the MLflow service:

docker-compose --env-file default.env up -d --build

• Give it a few minutes and once docker-compose is finished, check the containers health:

docker ps -a

• You should see something like this:

• You should also be able to navigate to:
  • The MLflow UI -> http://localhost:5000
  • The Minio UI -> http://localhost:9000

That's it! 🥳 You can now start using MLflow!

Run a MLflow Pipeline

---

1. Train and register a ML model

• Simply run the example provided in this repo:

python ./wine_quality_example/wine_quality_example.py

2. Serve the previously trained model

2.1. Use a local conda environment with mlflow

• Serve the model by running the following command, replacing the <run_id> for your own:

mlflow models serve -m s3://mlflow/1/<run_id>/artifacts/model -h 127.0.0.1 -p 1234 --timeout 0 

• Let it run, it should look like this:

2.2. Build a containerized API

Mlflow also allows you ti build a dockerized API based on a model stored in one of your runs. The following command allows you to perform this opertation:

mlflow models build-docker \
    -m <full_path> \ 
    -n adorable-mouse \ 
    --enable-mlserver

You just have to replace <full_path> by the full path to your model artifact as provided in the MLFlow web UI.

2.3. Generate a Dockerfile

In case you just want to generate a docker file for later use, use the following command:

mlflow models generate-dockerfile \ 
    -m <full_path> \ 
    -d ./likeable-hound \ 
    --enable-mlserver

3. Send a request to test the served model

• In a different terminal, send a request to test the served model:

curl -X POST -H "Content-Type: application/json" --data '{"dataframe_split": {"data": [[7.4,0.7,0,1.9,0.076,11,34,0.9978,3.51,0.56,9.4]], "columns": ["fixed acidity","volatile acidity","citric acid","residual sugar","chlorides","free sulfur dioxide","total sulfur dioxide","density","pH","sulphates","alcohol"]}}' http://127.0.0.1:1234/invocations

• The output should be the something like the following:

$ {"predictions": [5.576883967129616]}

🎊 Et voilà! 🎊

---

References:

• https://mlflow.org/docs/latest/quickstart.html
• https://github.com/mlflow/mlflow-example
• https://ghcr.io/mlflow/mlflow
• https://github.com/Gorcenski/ds-workbench