phitoduck/mlops-club

A reference implementation of an end to end, open-source MLOps platform.

📣 Welcome to the MLOps Club

This repo is for our group project, trying to build an end-to-end MLOps platform.

Project Philosophies

• This will be a fully-functional, opinionated MLOps platform. It should be considered a reference implementation that can be forked and adapted to a different set of opinions.
• It should be useful the average hobbyist, i.e. cheap to run, doesn't need a whole team to maintain, etc.
• BUT, this project should develop our careers, so the platform should have a straightforward way to scale to small-to-medium-sized enterprise needs.
• To reduce complexity, we'll start off as a monorepo, and split off into separate repos if the monorepo becomes challenging.

Project Kickoff video

15 minutes if you watch on 2x speed 🔥

Contributing

The setup has only been tested for Linux and MacOS, not Windows :(. If you're on windows, you'll have the best contributor experience using the Windows Subsystem for Linux 2 (WSL2).

How do I run this project locally?

# install "just"; it's like "make", but less frustrating
brew install just

# install the project's python packages and pre-commit
just install

# start all of the metaflow services in docker-compose (don't forget to start docker first 😃)
# http://localhost:9001          -> minIO UI (simulates S3)
# http://localhost:3000          -> the Metaflow UI
# http://localhost:8080/api/doc  -> swagger API docs (currently broken when we run locally, not sure why 🤔)
just run-local-metaflow

# run a metaflow DAG (flow) against your local metaflow instance!
just run-sample-flow

With any luck, you'll see this lovely output:

📌 Go look at the Justfile and docker-compose.mlops-club.yml to see how it all works :)

How do I add code?

We use pull requests. Create new branches based on trunk for experimentation, then open a PR for it. You don't have to wait until you want to merge code to open a PR. For this project, the main purpose of doing PRs is to share knowledge and get early feedback on your ideas.

Passing the pre-commit checks isn't a huge deal. They are mostly for your own benefit to prevent you committing things to the repo that you don't want. You can always override pre-commit by running

git commit -m "I really want to commit this large file" --no-verify

Notes on commits

Ask Eric or Ryan if you need any help with these.

DON'T COMMIT...

• credentials. Feel free to put them in a .env file, but make sure it's gitignored!
• large files (large CSV, ML model weights, C binaries, video, etc.) use git LFS rather than committing it directly.
• unformatted code

The pre-commit hooks setup for this repo when you ran just install will remind you about these each time you commit :)

mlops-club's Issues

running ui/service/minio locally - won't return dag etc..

@phitoduck ... came across your repo .. was interested in it for its local integration of minio with metaflow. However, in running some basic flows, I'm hitting some obstacles ... when calling a python helloworld.py run, I'm passing in the various METAFLOW parameters you list here.

The only parameter where I don't have a value for is METAFLOW_SERVICE_AUTH_KEY ... where does this come from? I tried creating an access key in minio, but this isn't working for me.

export METAFLOW_SERVICE_AUTH_KEY=iigKSNbmWb4To3dWBgmE17KyHcFyxEAn5NlU5aR8

Thanks for your input.

which code fails in justfile

What is the code command in that first line of just? Should we add setup dependencies to the readme? Should we update the justfile to install them?