mail-ecnu/vmagent

Our VMAgent is a platform for exploiting Reinforcement Learning (RL) on Virtual Machine (VM) scheduling tasks.

VMAgent is a platform for exploiting Reinforcement Learning (RL) on Virtual Machine (VM) scheduling tasks. It is developed by the Multi-Agent Artificial Intelligence Lab (MAIL) in East China Normal University and Algorithm Innovation Lab in HUAWEI Cloud. VMAgent is constructed based on one month real VM scheduling dataset called Huawei-East-1 from HUAWEI Cloud and it contains multiple practicle VM scheduling scenarios (such as Fading, Rcovering, etc). These scenarios also correspond to the challanges in the RL. Exploiting the design of RL methods in these secenarios help both the RL and VM scheduling communities. To emphasis, more details about VMAgent can be found in our paper VMAgent: Scheduling Simulator for Reinforcement Learning. Our another paper Learning to Schedule Multi-NUMA Virtual Machines via Reinforcement Learning has employed this VMAgent simultor to design RL-based VM scheduling algorithms.

Key Components of VMAgent:

• SchedGym (Simulator): it provides many practical scenarios and flexible configurations to define custom scenarios.
• SchedAgent (Algorithms): it provides many popular RL methods as the baselines.
• SchedVis (Visulization): it provides the visualization of schedlueing dynamics on many metrics.

Scenarios and Baselines

The VMAgent provides multiple practical scenarios:

Scenario	Allow-Deletion	Allow-Expansion	Server Num
Fading	False	False	Small
Recovering	True	False	Small
Expanding	True	True	Small
Recovering-L	True	False	Large

Researchers can also flexibly customized their scenarios in the vmagent/config/ folder.

Besides, we provides many baselines for quick startups. It includes FirstFit, BestFit, DQN, PPO, A2C and SAC. More baselines is coming.

Installation

git clone https://github.com/mail-ecnu/VMAgent.git
cd VMAgent
conda env create -f conda_env.yml
conda activate VMAgent-dev
python3 setup.py develop

To use runx module and .cv2 module, you may need to do next instructs:

pip install runx
sudo apt update
sudo apt install libgl1-mesa-glx

Quick Examples

In this quick example, we show how to train a dqn agent in a fading scenario. For more examples and the configurations' concrete definitions, we refer readers to our docs.

config/fading.yaml:

N: 5
cpu: 40 
mem: 90
allow_release: False

config/algs/dqn.yaml:

mac: 'vectormac'
learner: 'q_learner'
agent: 'DQNAgent'

Then

python vmagent/train.py --env=fading --alg=dqn

It provides the first VM scheudling simulator based on the one month east china data in HUAWEI Cloud. It includes three scenarios in practical cloud: Recovering, Fading and Expansion. Our video is at video. Some demonstrations are listed:

Docs

For more information of our VMAgent, we refer the readers to the document. It describes the detail of SchedGym, SchedAgent and SchedVis.

Data

We collect one month scheduling data in east china region of huawei cloud. The format and the stastical analysis of the data are presented in the docs. one month east china data in huawei cloud.

Visualization

For visualization, see the schedvis directory in detail.

References

• Junjie Sheng, Shengliang Cai, Haochuan Cui, Wenhao Li, Yun Hua, Bo Jin, Wenli Zhou, Yiqiu Hu, Lei Zhu, Qian Peng, Hongyuan Zha and Xiangfeng Wang, VMAgent: Scheduling Simulator for Reinforcement Learning. IJCAI Demos, 2022.
• Junjie Sheng, Yiqiu Hu, Wenli Zhou, Lei Zhu, Bo Jin, Jun Wang and Xiangfeng Wang, Learning to Schedule Multi-NUMA Virtual Machines via Reinforcement Learning, Pattern Recognition, 121, 2021, pp.108254.

License

Licensed under the MIT License.

vmagent's Issues

奖励函数bug情况

在运行train.py文件之后，生成的csv文件和tensorboard文件中的tot_reward和tot_length完全相等的情况。
一开始我用sched_env.py里面的单位奖励，然后得到两者相等是没问题的；但是我把单位奖励注释掉，奖励函数换成了代码中的另一个request['cpu']/self.cpu时，这时的结果应该是reward要小于length，但却没有按预想发生，仍然是两者相等；然后我就把奖励函数换成我自己设计的，发现实验结果里面两者还是一样相等。我在调用train.py文件里面的run函数时print每一次放置虚拟机的reward，发现不管是何种奖励函数，print处理的都是1。
下面是更神奇的事，我把sched_env.py里面的reward函数全部注释掉了，然后我运行train.py文件发现还是能运行，并且print出来的reward还是1。
请问作者这个是什么原因？以及如何解决？谢谢！

setup.py 的 stable-baselines 需要 0.11版本 gym

根据文档运行 python3 setup.py develop 时会出现下句话：
error: gym 0.10.9 is installed but gym[atari,classic_control]>=0.11 is required by {'stable-baselines'}
是不是需要写明 stable-baselines 的版本，或者提升 gym 的版本？

PPO、DQN、SAC训练奖励值为什么基本都没啥变化

如题，怎么训练过程中奖励值基本不变化，RL的目标不是最大化回合奖励吗？看起来并没有效果。。。

运行SAC算法时，出现了avail_actions的输出值突然变成了None这个情况。

我在运行SAC算法的时候，提示AttributeError: 'NoneType' object has no attribute 'cpu'。定位代码在action_selectors.py文件中Class NormalACActionSelector里的action_probs = softmax(agent_outputs)*avail_actions.cpu()这行，然后我在这行代码前加了一行print(avail_actions)，开始一段时间输出的都是正常tensor张量，但是突然就输出了一个None，提示错误，但是回头看到sac_controller.py文件的Class SACMAC里的mask_invalid函数已经对avail_actions是否为None进行了判断。故来请教这个bug该如何解决？