star-whale/starwhale

an MLOps/LLMOps platform

starwhale's Issues

add some probes for agent daemonset

Current Behavior
No check for agent daemonset.

Proposed Behavior
Add livenessProbe/readinessProbe/startupProbe for agent daemonset.

• add some api or cmd for agent pods.
• config helm charts.

[Pending]add huggingface transformer example

Proposed Behavior

• add 1-2 transformer model example
• tune sdk and report

Solution Proposal

• https://github.com/pytorch/serve/tree/master/examples/Huggingface_Transformers
• https://huggingface.co/

refactor swds in input.json

input.json structure can be used by swds and jsonline, so we should refactor swds field.

config timezone in controller

Current Behavior
use UTC timezone.

Proposed Behavior
controller can config timezone for cluster, then task/agent use it.

display system version on UI

Is your feature request related to a problem? Please describe.
user don't know what the version of SW is when he/she is attempting to make a bug report

Describe the solution you'd like

1. make the version info avaliable in the OS envirenment variables when building controller and agent images.
2. agent reads the OS ENV variable and reports it's version info to controller
3. controller reads the OS ENV and agents' reports
4. controller offers an API to UI and cli to expose these version info through

auto to detect python dependencies

Is your feature request related to a problem? Please describe.
Swcli only use pre-defined pip-req.txt or pip freeze in venv environment or conda export in conda environment today. Can we auto detect python dependencies by static-code-analysis ? It will more simple for end-user.

Describe the solution you'd like
WIP.

design swmp 2.0

Current Behavior
swmp 1.0 is a very simple structure, which is a single tar file. one line changed will make a new swmp tar, it is uneconomical.

Proposed Behavior

• we will introduce layer mechanism in swmp.
• add blake2b checksum into meta file.
• if no files changed, should swcli build a new swmp version?

[Pending]complete ui swmp creator and editor

Current Behavior
swcli build swmp only.

Proposed Behavior
users can use web-ui to create, edit the swmp.

merge controller and agent docker images into one image

Current Behavior
controller and agent have indepent docker images.

Proposed Behavior
use one single docker image to deploy controller and agent.

swmp list too too slow

Describe the bug
when we run swcli model list, it will take a lot of time to read meta in swmp tar file for meta.

Expected behavior
quick、efficient method for list.

support tag for swmp and swds

Current Behavior
Only very long version to track swmp/swds.

Proposed Behavior
Tag can reduce the users short-term memory load and provide more human friendly interactive.

Solution Proposal
WIP.

controller login failed when login mutlitime in a short time

design starwhale cloud demo version

Current Behavior
only in-premise version.

Proposed Behavior
Provide a cloud version, include some features:

• user isolation
• resource isolation
• user register
• cloud host plan
• deployment and release

use short version name to push/pull/run for swmp/swds

Current Behavior
use full version name.

Proposed Behavior
use short version name, such as only seven prefix of version.

add index files at ppl output phase

Current Behavior
no file index field.

Proposed Behavior
add index files for ppl output

[Pending]design evaluation report schema

Current Behavior

• Only multi_classification report.

Proposed Behavior

• Define a general report schema, which can descibe a variety of machine learning problems. The schema will be used by web-ui and client-terminal-ui to show some tables, graphs, summary etc.
• a schema
• a validator
• some machine learning examples
• define some frontend ui primitive

Ref

• https://github.com/evidentlyai/evidently

add validator for cmp sdk

Current Behavior
no validator.

Proposed Behavior
add output validator.

python ci

Current Behavior
no ci

Proposed Behavior

• add ci framework for python #291
• python black lint : #299
• flake8 check: #302
• mypy check: #305
• unittest: #306
• code test coverage: #310
• release pypi by tag action: #316

refactor client setup.py requirements

Current Behavior
setup.py and requirments.txt include the same python requirements.

Proposed Behavior
Only one place include python requirements.

model.yaml/dataset.yaml runtime bug

Describe the bug
Users can write wrong runtime field which is not actual python runtime. This issue will lead import wrong python version.

To Reproduce
Write runtime: 3.9, but venv local mode use python3.7.

Expected behavior

• use _manifest.yaml runtime first. #147
• auto check runtime field.

conda export sometimes omit starwhale package

Describe the bug

• conda export omit starwhale python package randomly.
• Reproduce Case: conda cannot export editable starwhale.

Expected behavior
conda export MUST include starwhale, if not , it will lead to import error in ppl/cmp phase.

Proposal
add some doc and console warnings

add text classification example

Current Behavior
no text classification.

Proposed Behavior

• add 1-2 text classification example
• tune sdk
• tune ppl amd cmp report

Solution Proposal
ref: https://github.com/pytorch/serve/blob/master/examples/text_classification

design experiment management

add object detection example

Current Behavior
no object detection example.

Proposed Behavior

• add 2-4 object detection example
• tune sdk
• tune ppl amd cmp report

Solution Proposal

• https://github.com/pytorch/serve/tree/master/examples/object_detector

complete swds/swmp push methanism

Current Behavior
run push cmd, and wait wait cmd exit. no upload progress, no auto retry.

Proposed Behavior

• more friendly output.
• head first for version.

Some Controller API suggestions

Enhancement

• createTime vs createdTime vs startTime
  We can use one uniform verb to describe create time in all apis. pr: #190
  
• /api/v1/project/{pid}/dataset/{did} may return serialized meta field directly.
• /api/v1/project/{pid}/model/{mid}/version add meta field.
• /api/v1/login return user's details, such as role, createdtime.
• should we add owner field in model/dataset list api? It will bring a lots of redundant fields.
• could /api/v1/project api add username query string to show the specific projects? in default, only show the projects of current login user.

Bugs

• /api/v1/project/{pid}/job/{jid}/task
  • this api is lack of finishTime and duration field.
  • this api is lack of task type field, which can describe cmp or ppl task.
• modelName is required for /api/v1/project/{pid}/model ?
  • make it as a optional field？
    

Feature Request

• need one api to show all models with external parameters
  • scenario: run swcli model list --remote cmd in local environment. users may specify multi dimensional parameters, such as --project , --model-name, --self (own projects models).
• need one api to show all datasets with external parameters
  • scenario: same as models api.
• /api/v1/project/{pid}/model/{mid} api add version into url path or params.
  • this api only show the latest version files now. client need fetch the different version's files.
  • model and dataset may use consistent schema, which make cli or frontend write simply code.
• `/api/v1/project/{pid}/model/{mid}' show more details in one api.
  • such as model create time, meta, files etc.
• `/api/v1/project/{pid}/dataset/{mid}' show more details in one api.
  • such as dataset create time, meta, files etc.

cc @dreamlandliu @anda-ren

api: job/result return with wrong response

Describe the bug
job/result expect to be job metrics, but log content returned

To Reproduce
Steps to reproduce the behavior:

1. Go to http://localhost:3000/projects/1/jobs/64/results
2. Open Browser Dev Tools
3. Check Network Panel

Expected behavior
expect to be job metrics

Screenshots

Additional context
none

agent uniqueness change

Is your feature request related to a problem? Please describe.
Now we give an unique constrain on anget through ip address. But the ip address of an agent is constantly changing in K8S context. A new uniqueness field of agent is required

Describe the solution you'd like

• add an unique field of agent called serial_number
• agents report the serial_number to controller and make sure these numbers are universal unique

add CI/CD workflow based on github actions

Current Behavior
Manual trigger

Proposed Behavior
automation

Solution Proposal
based on github actions

add more details into swcli version

Current Behavior
show 0.1.0-dev15 only.

Proposed Behavior
show more details for swcli version, such as build date, git commit sha.

define a `.swignore` file to ignore some files when build swmp/swds

Is your feature request related to a problem? Please describe.
We need write exclude_pkg_data in dataset.yaml or model.yaml twice. In another side, a lot of ignore fields may be disturb the concision of model.yaml/dataset.yaml.

Describe the solution you'd like
Define .swignore file, REMOVE exclude_pkg_data field.

starwhale python package and its dependecies use more than 500MB space.

Current Behavior
too large.

Proposed Behavior
remove useless dependecies.

typo mutlilabel

Describe the bug
/api/v1/project/{pid}/job/{jid}/task/result use typo world.

To Reproduce
visit result api.

Expected behavior
mutlilabel -> multilabel

run list cmd will raise exception when no swmp/swds in local storage

Describe the bug
raise CreateFailed: root path '/home/xxx/.cache/starwhale/pkg' does not exist exception.

add image classification example

Current Behavior
only mnist.

Proposed Behavior

• add 2-3 image classfication examples, use pytorch/tensorflow/sklearn lib
• tune sdk and ppl/cmp report

Solution Proposal

• https://github.com/pytorch/serve/tree/master/examples/image_classifier
• https://towardsdatascience.com/exploring-classifiers-with-python-scikit-learn-iris-dataset-2bcb490d2e1b

remove docker-in-docker, use kubernetes native to refactor controller-agent

Current Behavior
docker-in-docker

Proposed Behavior

• remove agent pod
• add agent sidecar
• define starwhale crd, write some operator

add auc/roc output for multi_classification report

Current Behavior
multi_classification does not include auc/roc data.

Proposed Behavior
auc/roc is very common and import for multi_classification problem.

Solution Proposal
add methods for multi_classification decorator.

block in loading data from s3/minio

Describe the bug
if minio/s3 cannot be connected, the load program will be blocked unitl the user kill the process.

Expected behavior

• config timeout
• auto retry
• active exit proccess
• more user friendly log

add dataset name in swds input.json

Current Behavior
one input.json only include one dataset.

Proposed Behavior
include dataset name dimension into input.json.

agent dynamically generate OSS connections

Current Behavior
agent generate fixed oss connections via properties when bootstrap

Proposed Behavior
dynamically generated according to the connection information sent by the controller

Solution Proposal
dynamicallly generated

eval run cmd in local environment

Current Behavior
We only run the complete-flow evaluation in controller.

Proposed Behavior
When we run swcli eval run --local，cli will use local swmp and swds to run complete-flow evaluation. The feature make a great help for debug.

• Use docker to reproduce eval flow.
• Do not depend on minio.

Design Proposal

• cmd schema: swcli eval run [--local/--remote] --model xx --dataset xx --dataset yy [--project xx] [--baseimage xx] [--resource gpu:1] [--name xx] [--description xx] [--phase]
  • --local/--remote: optional, in local or remote cluster, remote cluster is the default option.
  • --model: required, model id or model name:version(local mode)
  • --dataset: required, dataset id or datset name:version(local mode)
  • --project: optional, project id, only for remote cluster mode.
  • --baseimage: optional, task run image. if omitted, starwhale will use the latest baseimage, name is: starwhaleai/starwhale:latest
  • --resource: optional, only for remote cluster mode, fmt is [resource:gpu|cpu]:[cnt:int >0], default is cpu:1
  • --name: optional, eval job name, the username-timestamp-randomstr is the default name.
  • --desc: optional.
  • --gencmd: optional, only generate docker run cmd in local mode.
  • --phase: optional, only for local mode. choices: all|ppl|cmp, default is all.
• local mode:
  • search swmp and extract swmp into workdir
  • generate swds fuse input.json
  • render/run docker run cmd for ppl
  • render/run docker run cmd for cmp
  • show progress
  • parse and render cmp report
  • eval storage for local mode
  • in {@snapshot_dir}/run/eval/{version}/, we will store all result artifacts.
  • swcli eval list --local will show local eval list.
  • swcli eval inspect xxx --local will show local result and report.
• external cmd:
  • swcli dataset fuse xxx : generate fuse input.json
  • swcli model extract xxx: extract swmp tar

log methanism for evaluation task

Proposed Behavior

• Web visit realtime or archived task log.
• #518
• #517

tune size for mnist swmp

Current Behavior
very simple mnist example will use more than 2GB size for swmp, should we do some optimizations for reduce swmp size?

Proposed Behavior

• swmp layers
• base image pre-install pytorch

js/ts ci

Current Behavior
no ci

Proposed Behavior

• add js/ts lint ci
• add compile checker

java ci

Current Behavior
no ci

Proposed Behavior

• java lint
• java unittest
• release controller/agent docker image

agent online & offline management

Is your feature request related to a problem? Please describe.

• controller can't tell if an agent is being down or long time reporting
• controller can't tell if an agent is removed or being down

Describe the solution you'd like

• when controller receives agent's report it will define the deadline before which agent should report next time. The deadline is dispatched to agent.
• controller will set agent status to down if it does't receive report from the agent before the deadline
• user remove removed agent on the UI (let user make the removing decision)

Subtasks

• #247
• #248

add neural machine translation example

Current Behavior
only mnist example

Proposed Behavior

• add one nlp model
• tune sdk and report

Solution Proposal

• ref: https://github.com/pytorch/serve/tree/master/examples/nmt_transformer

homepage for starwhale.ai

Current Behavior
no homepage

Proposed Behavior
use github-pages to host starwhale.ai homepage

controller proxy/cache for pypi/docker image/conda

Current Behavior
controller does not support proxy or cache. when agent reproduce swmp python package or pull image, it will spend a lot for time.

Proposed Behavior

• cache: pypi or docker image or conda channel
• settings: upstream address, conda channel, cache expire time, timeout settings
• storage: use minio or s3
• auto-inject: agent will auto inject envs or files into task container.
  • ~/.pip/pip.conf
  • docker /etc/docker/daemon.json mirror field
  • ~/.condarc

controller support https

Current Behavior
only support http.

Proposed Behavior
Cloud and on-premise both https.