microsoft/openpaimarketplace

A marketplace which stores examples and job templates of openpai. Users could use openpaimarketplace to share their jobs or run-and-learn others' sharing job.

Openpaimarketplace

---

A marketplace which stores data and job templates of OpenPAI. Users could use openpaimarketplace to share their jobs or run-and-learn others' sharing job.

Components

There are two components of openpaimarketplace, rest server and webportal, which are responsible for backend service and frontend UI seperately.

Getting Started

• For Admin

  To the admin user who is responsible for deploying and managing OpenPAI and openpaimarketplace, please check admin manual for help.

• For User

  To the normal user who wants to use marketplace templates in OpenPAI, please check user manual for help.

Reference

• OpenPAI: A complete solution for AI platform. HiveD will be more user-friendly when working in tandem with OpenPAI.

• OpenPAI Protocol: The protocol interface between marketplace and OpenPAI platform.

Developing Guide

This section is a guide for developers who are new to openpaimarketplace. Openpaimarketplace contains 2 sub projects, rest_server and webportal. For the detailed developing guide, you should refer to the readme doc of each sub project.

rest_server

Rest server uses nodejs as backend service framework. The api follows RESTful API specification.

webportal

Currently openpaimarketplace webportal is used as a pai webportal plugin. It uses react as frontend framework, and builds with webpack bundler.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

openpaimarketplace's Issues

2021 Jan. Release Plan

Release Manager

@debuggy

End Game

code freeze: Dec. 24
Bug bash: Dec. 28 ~ Jan. 8

Plan Items

Deployment and use #165

• ⭐ Deploy database with more rubustness -- @debuggy
• ⭐ Blob is not required when using job template -- @debuggy

market list page

• ⭐ Add search bar -- @debuggy

Create templates

• Refine UI of marketplace create page -- @yangou1988 @yiyione ETA: 12/18 #163

• ⭐ Create data item (basic feature: upload data to Azure blob)

• UI design -- @yangou1988 @TobeyQin ETA: 12/9
• Feature implementation -- @yiyione ETA: 12/18

• ⭐ Create item from job list, or add "save to template" feature -- for discussion --@everyone

Documentation

• ⭐Refine manual book of marketplace deploy flow #154 -- @debuggy @TobeyQin @yiyione ETA: 12/24
• Deployment flow discussion @debuggy @yiyione @TobeyQin ETA: 12/11

Test Cases

Test owner: @TobeyQin

• Marketplace home page components and items #185 #186
• Download dataset
• Upload dataset #169 #170 #171 #172 #173 #174
• Create job template #187
• Delete redundant items(using delete API) #188
• Run each templates successfully @SwordFaith
• Successfully deploy marketplace follow the document @debuggy

Backlog

• 拖拽式UI
• Version control
• Easily transfer GitHub project to Marketplace
• Introducing marketplace in submit job page
• Export template to marketplace from job list and job detail page

Examples Template

• P2 - NNI tasks NNI student program
  (controller, trials) -- @TobeyQin @qfyin (+liuzhe-lz )

• P2 - NNFusion example -- @TobeyQin @qfyin (+wenxaing)

Submission Related Backlog

• P2 ~Submit job ui should show protocol prerequisites parts #73 @debuggy @yiyione @TobeyQin ~

• P1 - Add feature to select dataset by click checkbox -- @hzy46

• P2 - Refine Submit Job page layout -- @yangou1988

• P0 - Fix data storage config mismatching issue -- @debuggy

• P1 - add "Upload" feature for dataset, set upload standard and workflow
  1. For public dataset (Specify the source path)
  2. For private dataset (Could we store data to Azure blob? or GitHub?)

Collaboration with SHAIIC -- discussion

• Marketplace deploy flow
• Training examples
• Jupyter Notebook interaction

Test casse for Sept. release

Test Owner

@TobeyQin

Test Items

1. view market list

• refresh to view market list
• check if all items list in the page
• check if filter works

2. view market item detail

• click an item to item detail page and see if any bugs
• back to market list

3. submit and delete

• check if UI and config match in submit job page
• submit an item template to pai as job and run to see if success

4. check known bug fixed

• check if corresponding data team storage is checked

Use pai icon to replace project icon if author is OpenPAI

covid chest xray data.url parse error and need create /data dir

UI exceed margin in Covid19 Outbreaks Dataset description

[create data template] New created data template do not have "download" button

Bug Description

New created data template do not have "download" button

Reproduce Steps

1. Create a new data template
2. Click the template just created
3. No "download" button

Screenshots

Additional Info

Doc Update

1. Github Readme

• Introduction @debuggy
• Getting Started
  • For admin (admin manual link) @yiyione
  • For user (user manual link) @SwordFaith
• Reference @debuggy
• Contributing @debuggy

2. User manual in readthedocs

• admin manual @yiyione
• user manual @SwordFaith

3. Update description of examples @TobeyQin

The adapt between job templates and cluser device driver

If you have a job templates which requires AMD GPU, but the cluster only has Nvidia GPUS, how to check and meet the requirements of the job template?

cifar10-single-mul-DDP-nccl-gloo mp.spawn lack gpu arg for train

https://github.com/microsoft/pai/blob/master/examples/Distributed-example/cifar10-single-mul-DDP-nccl-gloo.py

line 63: train function signature: def train(gpu, args, trainset)
line 42: mp.spawn(train, nprocs=args.gpus, args=(args, trainset)), where args= should be a tuple with size of 3.

Cannot read property commands of undefined error on use commands

Not work:

• cifar10-single-mul-DDP-gloo-Apex-mixed
• cifar10-single-mul-DDP-nccl
• cifar10-single-mul-DDP-gloo
• cifar10-single-mul-DDP-nccl-Apex-mixed
• tensorflow_serving_mnist
• Caffe2 ResNet50 Example
• Horovod Pytorch
• Caffe2 Minist Example
• Keras Tensorflow Mnist Example
• Mxnet Autoencoder Example
• rocm_tensorflow2
• Pytorch Mnist Example
• Rocm Pytorch Mnist
• Chainer Example
• tensorflow_cifar10
• Pytorch Regression Example
• sklearn_text_vectorizers

Work:

• cifar10-single-node-gpus-cpu-DP
• GPU Sharing Example
• COVID-19 Chest X-ray Diagnosis Training Job Template
• COVID-19 Chest X-ray Dataset
• Covid19 Outbreak Statistics Dataset
• Couplet Training Job Template
• Covid19 Inference Service
• Couplet Inference Job Template
• Grammar Check Inference Job
• Grammar Check Model Training
• Covid19 Prediction Model

Cannot read property 'commands' of undefined

Reproduction step

1. Click any old template in marketplace list
2. Click "use" button, and show below error

2020 Sept Release Plan

Release Manager

@TobeyQin

End Game

Feature freeze: 9/30

Code freeze: 10/12

Scrum demo date: 9/30 ~ 10/12

Plan Items

1. P0 - New templates

   1. add new datasets

   • P0 - X-ray chest check -- @yiyione

   • P0 - Grammar check -- @debuggy

   • P0 - Covid19 bug fix -- @debuggy

   2. add new job templates

   • P0 - X-ray chest check -- @yiyione

   • P0 - Grammar check -- @debuggy

   • P0 - Distributed job -- @hzy46

   • P2 - Multi-job GPU Sharing -- @hzy46 @TobeyQin

   • P2 - NNI job integration -- Feature Engineering -- @debuggy @TobeyQin

   3. Description document template -- @TobeyQin

   • P1 - Add dataset doc template

   • P1 - Add job template doc template

   4. • P0 - add info for items in submit job page -- @debuggy @hzy46

   5. other features

   • P1 - add "Upload" feature for dataset, set upload standard and workflow - @debuggy / @yangou1988
     1. For public dataset (Specify the source path)
     2. For private dataset (Could we store data to Azure blob? or GitHub?)

   • P2 - add "topic" filter feature - @yangou1988

2. • P0 - Refine Submit Job page -- @yangou1988
3. • P1 - Fix data storage config mismatching issue -- @debuggy
4. • P0 - User experience survey -- @yangou1988 @TobeyQin
5. • P1 - Submit job page export YAML prerequisite variables design
6. • P1 - For dataset, add "download" feature -- @debuggy

sklearn_text_vectorizers install fail

marketplace edit and delete function should be limit to author or admin

Error no available connectionString

Reproduction step:

1. In the Marketplace List page, click any template.
2. click "use" button, and then show below error:

Root cause:

Empty Connection String when initializing the marketplace.

Hotfix

Add a certain connection string when initialization.

Long term solution

<TBD>

Oct. ~ Nov. Test Cases & Known Issues

test cases:

• Webpage layout and items display correctly.
• All the link and button works.
• Test data download
• Test Create template logic and if it can successfully add template.
• Test Azure storage URL is correct and accessible.
• Test if all the jobs can finish successfully.
• Test if can get output files when job finished.

known issues:

1. • Failed to fetch job log (#5094 )
2. • Error no available connection strings #104 PR in review
3. • Cannot read property 'commands' of undefined #105
4. • covid chestxray dataset job failed #106

covid chestxray dataset job failed

reproduction step

1. In marketplace page, click covid-chestxray-dataset
2. Click "use" and "Submit" to submit job.
3. Click “stderr” to show the job failed log

Job Name: covid-chestxray-dataset_e5b25a75

Error message:

rocm_pytorch GPG error

Marketplace service deployment test for resources and throughput

http request will be blocked in webportal https

failed to pull the webportal_plugin image

After I run the pull command, I get the following error:

openpai@dev-box:~$ docker pull docker.pkg.github.com/microsoft/openpaimarketplace/webportal_plugin:latest
Error response from daemon: unauthorized: Your request could not be authenticated by the GitHub Packages service. Please ensure your access token is valid and has the appropriate scopes configured

Lack old example description

In current HEAD, old examples don't have description markdown file, so can't display description in their detail page.
There is description info in each old example protocol. It should be better to read from protocol file when there is not any description markdown.

Marketplace item submmitted by admin also needs approval

If an admin submit a marketplace item, it will also needs approval from admin. It is somewhat confusing.

Autotest all marketplace official templates

auto fill statue column in #129

2020 July ~ Aug Release

Release Manager

@TobeyQin

End Game

Code freeze: 8.17

Scrum demo date: 8.20

Endgame & retrospective date: 8.31

Plan Items

1. Marketplace - @debuggy / @TobeyQin / @yiyione

• P0 - merge old job templates to new marketplace, add label "old" #41

• P0 - Unified pai protocol

• P0 - add new modules

  1. add new datasets

  • X-ray chest check

  • Grammar check

  2. add new job templates

  • X-ray chest check

  • Grammar check

  • P2 - NNI job integration -- Feature Engineering

• P1 - add "open in new tab" option when right-click "use" button

• P1 - add "Upload" feature for dataset, set upload standard and workflow - @debuggy / @yangou1988

  1. For public dataset (Specify the source path)
  2. For private dataset (Could we store data to Azure blob? or GitHub?)

• P2 - add "topic" filter feature - @yangou1988

2. GPU Utilization @hzy46 / @TobeyQin

• P0 - Connect with people in SHAIIC, get teaching jobs and usage scenario

• P0 - Discuss about the user demand

• P0 - Get baseline of GPU usage

• P1 - Make plan to improve the utilization

• P1 - Code Implementation

• P1 - Measure the GPU utilization methods

Use expose instead of ports to avoid conflicts with the openpai service on master node.

On the master node, ports 5432 and 3000 are binded by openpai. I have been exploring the Internet for a long time before finding a solution. This solution needs to modify the Postgres port declaration to expose instead of ports in docker-compose.yml.

Before

    ports:
      - "5432:5432"

After

    expose:
      - "5432"

For more details, please refer to my Github repository siaimes/openpaimarketplace .

Bug when viewing taskroles in some item

Speed up download button

Pull request build image check will fail if pushed by forked repo

#143

Webportal generate job protocol wrong upload path

[create template] Can we show template name but not ID in the create successfully confirm page?

Bug Description

Can we show template name but not ID in the create successfully confirm page?

Reproduce Steps

1. Create a data template
2. The template ID but not template name showed in the confirmation page

Could we replace the item id to the template name in the page?

Screenshots

Additional Info

Template type: Data template Job template

[User Experience] Hope to add right-click open in new tab feature

Could we add right-click to "open in new tab" feature? It is so hard to return to Marketplace home page after submit a template job.

User story of marketplace

• AI-EDU
• covid-19
• nni
• grammar check

Refine document

When click the "back to marketplace" link in the template description page, can it go back to the last filtered page, but not show all components in the marketplace?

Bug Description

"back to marketplace" link cannot go to last filtered page.

Reproduce Steps

1. Click "Data Template" button in the marketplace list page
2. Click any data template in the filtered list.
3. Click "back to marketplace" in the template description page.

I will not go to the list with the "data template" filter result, but will list all items. Can we just show the filtered list?

Screenshots

Additional Info

Create item UI issues

1. Upload YAML should be Upload OpenPAI Protocol.
   

2. In drop-down button, data template option should be disabled. It could be set as job template by default.

Font style guide of webportal

Webportal Font Solution:

1. FontClassNames (Use to custom text)
   Different classname has different default styles. Incudes:

• FontSize
• FontWeight
• FontFamily: A font map for different language with a fallback const FontFamilyFallbacks = `'Segoe UI', -apple-system, BlinkMacSystemFont, 'Roboto', 'Helvetica Neue', sans-serif`;

2. Text component (Recommended to use as default text)
   Has a parameter variant to control the font styles.

3. Tachyons. The tachyons css module is not recommended right now. However the history code still has tachyons styles. Will be removed step by step.

2020 Oct. ~ Nov. Release Plan

Release Manager

@TobeyQin

End Game

code freeze: Nov. 16

Plan Items

P0 items for this release

1. Data -- @debuggy

• P0 Move NFS to Azure Blob as backend -- ETA: Nov. 5
• P0 Upload Job output to Azure Blob -- ETA: Nov. 9
• P0 Download data from azure blob to local -- ETA: Nov. 2
• P1 Use Azure storage SDK for privacy -- ETA: TODO
• P1 Refactor data use logic after change storage to blob

2. Service Deployment -- @yiyione

• P0 Start Local Rest Server -- ETA: Nov. 2
• P0 Deployed Rest Server in PAI -- ETA: Nov. 5
• P0 Start database and load items into webpage -- ETA: Nov. 9
• P0 Register in PAI pylon -- ETA: Nov. 5
• P0 Figure out the cost of marketplace service -- ETA: Nov. 24

3. • Create marketplace item -- @yiyione

4. Documentation -- @TobeyQin @debuggy

• GitHub readme for users and developers -- ETA: Nov.25

• User Manual on readthedocs -- ETA: Nov.25

• Update all the description docs -- ETA: Nov.20

5. UI Layout

• Marketplace plugin default to toggle up -- ETA: Oct. 30

6. UI Design -- @yangou1988

• Add download button in Marketplace data template -- ETA: Oct. 30

Submission Related Backlog

• P1 The defaulting should only be done after submission phase #4576 @debuggy ETA: TODO after 11/6

• P2 Submit job ui should show protocol prerequisites parts #73 @debuggy @yiyione @TobeyQin

• P1 - Add feature to select dataset by click checkbox -- @hzy46

• P2 - Refine Submit Job page layout -- @yangou1988

• P0 - Fix data storage config mismatching issue -- @debuggy

• P1 - add "Upload" feature for dataset, set upload standard and workflow
  1. For public dataset (Specify the source path)
  2. For private dataset (Could we store data to Azure blob? or GitHub?)

Other items

2. User create templates

• P1 - Create from marketplace page (Provide protocol and description)

• P1 - Save protocol from job submission page

• P2 - Support filter and version control

• P2 - Design create feature from job list -- @yangou1988

• P1 - Job rotate to marketplace

4. Others

• P1 - Survey of integrate GitHub project to Marketplace -- @yiyione

---

Bug Fix

2. • P0 - Add Distributed Job template -- @hzy46

Create item page refine (job/data template)

under construction

marketplace test cases for Aug release

1. view market list

• refresh to view market list
• check if filter works

2. view market item detail

• click an item to item detail page and see if any bugs
• back to market list

3. submit and delete

• submit an item template to pai as job and run to see if success

Status Table of official templates

Type	Job Template Name	Finish successfully	Fail issue	Upload Successfully
Job	cifar10-single-mul-DDP-nccl-Apex-mixed	Y		-
Job	cifar10-single-mul-DDP-gloo	Y		-
Job	cifar10-single-mul-DDP-gloo-Apex-mixed	Y		-
Job	cifar10-single-node-gpus-cpu-DP	Y		-
Job	cifar10-single-mul-DDP-nccl	Y		-
Job	GPU Sharing Example	Y		-
Job	COVID-19 Chest X-ray Diagnosis Training Job Template	Y	#133	N
Job	Couplet Inference Job Template	Y		-
Job	Covid19 Inference Service	Y		-
Job	Grammar Check Model Training	Y	#133	N
Job	Couplet Training Job Template	Y	#133	N
Job	Grammar Check Inference Job	Y		-
Job	Covid Prediction Model	Y	#133	N
Old	tensorflow_serving_mnist	Y		-
Old	Caffe2 ResNet50 Example	Y		-
Old	Horovod Pytorch	Y	#128
Old	Caffe2 Minist Example	Y		-
Old	Keras Tensorflow Mnist Example	Y		-
Old	Pytorch Mnist Example	Y		-
Old	Pytorch Regression Example	Y		-
Old	Rocm Pytorch Mnist	N	need amd sku
Old	tensorflow_cifar10	N	#128
Old	rocm_tensorflow2	Y
Old	Mxnet Autoencoder Example	Y		-
Old	Chainer Example	Y
Old	sklearn_text_vectorizers	Y
Data	COVID-19 Chest X-ray Dataset	Y
Data	Grammar Check Dataset	Y		-
Data	Couplet Dataset	Y		-
Data	Covid19 Outbreak Statistics Dataset	Y		-

test cases of bug bash

1. create new marketplace item

• create a new marketplace item from scratch
• create a new marketplace item from success job list

2. view market list

• refresh to view market list
• check if filter works
• check if pagination works

3. view market item detail

• redirect to item detail page and see if any bugs
• back to market list

4. diff between admin and non-admin

• edit cookie isAdmin, to see if pending list shows or not
• as admin, approve / reject some items

5. submit and delete

• submit some item template to pai as job and run to see if success
• delete a market item

Use gravatar as users' icons

https://en.gravatar.com/

Marketplace quick start script

chainer example utf8 character in stdout can't display

Add searchbox in market list page

Demo:

Test Cases:

Check if the filter works well in each step:

• Open marketplace list page
• click Data Template at sidebar
• search Couplet and press enter
• click Data Template again to clear type filter
• clear search and type enter

Known Issues:

• search keyword is case sensitive because [Op.substring] is case sensitive

Left work

• show results number and keyword in itemlist like github

the admin in cookie is string instead of bool

Horovod Pytorch can't submit due to wrong taskrole

An issue for testing purpose

Check if the Teams connector works.