cml_dvc_case's People
Forkers
nsudhanva davesteps tibraga ankitmanchandaa nico-gui yashvip mvshmakov campos537 daavoo shivaholicvikas237 vs-sakthi davisjoshuar ngrigoriew g-delong tlm629 dhuntdegould mfizz rishikreddy2408 luvo95 eb-art sathviknapa choisuren0605 erfard timolivermaier natluqwerty howard-haowen cameronraysmith aayuprk15 pmelin shiwangi23 jack-sim san5167 testreporg ctufts disla-steeldevelop edwenger asad-ismail tranphong1991 moguraion theoh-io jfcalder marixko rafaeldasilva chrismaral partekucml_dvc_case's Issues
train a model on large dataset with gitlab-ci.yaml
Hi,
First of all, thank you for the nice tools developed by you. I am trying to create a training ML workflow with gitlab, CML and DVC with MinIO storage as my remote backup where I have my training dataset stored. my .gitlab-ci.yaml
looks like this:
stages:
- cml_run
cml:
stage: cml_run
image: dvcorg/cml:0-dvc2-base1-gpu
script:
- echo 'Hi from CML' >> report.md
- apt-get update && apt-get install -y python3-opencv
- pip3 install -r requirements.txt
- dvc remote add -d minio_data s3://bucket/dataset/
- dvc remote modify minio_data endpointurl http://<MINIOSERVER_IP_ADDRESS>:9000
- dvc remote modify minio_data use_ssl False
- export AWS_ACCESS_KEY_ID="xxxxxxx"
- export AWS_SECRET_ACCESS_KEY="xxxxxxx"
- dvc pull -r minio_data
- python main.py
- cml-send-comment report.md --repo=https://<my_gitlab_repo_url>
My setup is configured as following:
- A gitlab self-hosted runner listening for job (works: Ubuntu 20.04, 2 x RTX 3070 GPUs, ).
- An S3 MinIO storage server configured as DVC remote local backup (works with my credentials).
- A training script (works).
My workflow is working and I am able to train my model on the runner and queue jobs but I have the following issues with it (maybe there is a better way to do this, hence I am here asking for directions):
- For each training job, the entire dataset is pulled from the remote and then the model is trained. This is really slow. It is my requirement to keep using dvc for data versioning but is there a way to bypass the dataset pull
dvc pull -r minio_data
everytime and use the same data between different training jobs? (maybe mount volumes to the docker container?) - For MinIO authentication, I do not want to put my credentials as in AWS_SECRET_ACCESS_KEY in the
.gitlab-ci.yaml
file, in case more than one person want to use this workflow to queue their training jobs in a collaborative environment. What other options do I have? - Is there a way to configure a local container registry cache for the runner (and this worflow) where I can put all the necessary docker images and use them instead of adding dependencies to the workflow like I am doing and let docker handle it?
Any feedback or suggestions would be appreciated. Thank you.
Is there a possibilty to use the dvc in cml with GDrive
As a remote dir I use GDrive.
When I pull the data to my personal computer it requires manual authentication conducted by clicking the provided link.
When I try to use a similar action as you do I get error cause remote computer on which github tries to run action can't conduct an authentification in this way.
Is there a possibility to authenticate gdrive in a smillar way as you do withn S3 bucket - with some kind of token?
Cannot publish plot images
Hello,
I am trying to reproduce the test case but I am unable to include plot images in my report it throws this error
"""
{"code":"ERR_INVALID_URL","input":"\r\n<title>400 Bad Request</title>\r\n\r\n
"
"""
Without images the report looks good, here is my cml.yaml file
name: train-my-model
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- uses: iterative/setup-cml@v1
- uses: iterative/setup-dvc@v1
- uses: actions/setup-python@v2
with:
python-version: '3.x'
- name: cml
env:
repo_token: ${{ secrets.REPO_SECRET }}
run: |
pip install -r requirements.txt
# Pull dataset with DVC
#dvc pull data
# Reproduce pipeline if any changes detected in dependencies
#dvc repro
# Use DVC metrics diff to compare metrics to master
git fetch --prune --unshallow
dvc metrics diff master >> report.md
# Add figure to report
dvc plots diff --target loss.csv --show-vega master > vega.json
vl2png vega.json -s 1.3 > vega.png
echo '![](./vega.png)' >> report.md
cml comment create --pr --publish report.md
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.