About the file “env.py”

[hongzimao]

About the file “env.py”

[hongzimao]

1. It is not the jobs the agent decides to schedule at this round --- that will be the action. source_job is the job that frees up the current executor. In other words, if some job is indeed the source_job, there won't be migration cost when assigning new nodes from this job to the executor.

2. No, get_executor_limits tells you how many more executors you can still assign to a job. It's just when the job is source_job, we need to count in the executors that already bind to the job, that's what you saw in https://github.com/hongzimao/decima-sim/blob/master/spark_env/env.py#L148-L151

3. exec_commit describes the destination of some executor committed to be scheduled. To understand this term, we need to explain when scheduling event occurs: at the end of a previous node, let's say the node had 5 executors running, so when the first executor of out this 5 doesn't have new tasks to run, we invoke the scheduling module (because we don't want that single executor to wait the other 4, causing unnecessary idle time). Now the scheduling module determines the new node for all 5 executors (because we don't want to invoke the scheduling module too many times). But at this decision, only 1 executor can move, so we put the other 4 executor in this exec_commit so that when they are free later on, they can consult exec_commit instead of the scheduling agent to determine the next node to run.

Hope these help!

[bywshr]

1、what is “source executors”mean?
2、what is "source job/source"mean?
3、what is "use_exec"mean?
4、what is "exec-commit"mean?
5、Why calculate use_exec in this way?

[bywshr]

1. Does "self.source_job" means the job that you decide to schedule in this step?
2. In "get_executor_limits" this function, will all the jobs' limit except souce job be 0 ?
3. What does "self.exec_commit" mean?

Thank you!
May I ask you some questions？
1、what is “source executors”mean?
2、what is "source job/source"mean?
3、what is "use_exec"mean?
4、what is "exec-commit"mean?
5、Why calculate use_exec in this way?