Unexpected dense task placement for k8s cluster

[kubernetes-retired]

Unexpected dense task placement for k8s cluster

[dalanlan]

Sorry again that I am being vague here. The example pods I am trying to run are from k8s doc. They r not computation or I/O sensitive to me, no, but I just want to make sure it's not like the case that poseidon cannot see other options but the machine resource it runs on.

[ms705]

@dalanlan This is entirely plausible: depending on the value of --max_tasks_per_pu that you use (the default is 10), the scheduler may put several pods on each machine.

The specific policy applied depends on what cost model the Firmament library uses under the hood. This is read from the --flow_scheduling_cost_model flag and the default is the TRIVIAL cost model (0). The trivial cost model does not have any sophisticated assignment policy: it just sets a higher cost for leaving tasks unscheduled than for scheduling them anywhere. If you want to look into using more complex policies with Poseidon, you'll want to specify a different cost model. That said: since Poseidon does not yet allow for passing K8s metrics through to Firmament's knowledge base, the policies will have little information to work with, and may not do anything particularly useful when using from Poseidon.

Depending on what your goal is, you could either patch Poseidon with the necessary plumbing to get your cost model to work (we can possibly help you with this), or you could develop your own cost model that reflects the scheduling policy you need (again, we can help with this, but it will require building the Firmament library (and Poseidon) from scratch.

[ms705]

Closing, as things have moved on (although above still remains essentially valid).

Let us know if any questions remain, or if you encounter issues with the new code.