Training and Generalization

[kauray]

Training and Generalization

[hongzimao]

Notice that Decima is evaluated with number of incoming jobs at least 10x that during training. The example training and testing script in README.md also indicates this point: the training has tag --num_stream_dags 200 while testing has tag --num_stream_dags 5000. The way to think about whether 200 or 2000 jobs during training is enough is that we need to make sure the agent experience a "full cycle" of workload, meaning, sees the empty system gets loaded by incoming jobs and then drain the jobs to empty (might repeat this ups and downs for multiple times).