"Job reported running on node no longer exists or is not in running state" msgs

[openpbs]

"Job reported running on node no longer exists or is not in running state" msgs

[bhroam]

The function collect_jobs_on_nodes() turns the 'jobs' attribute on a node into a resource_resv ** array. The race condition that is being reported here is when a job ends between when we query the nodes and query the jobs. The problem arises because we call the function not only for a node, but also for a reservation. The 'jobs' attribute lists all jobs on the nodes, in reservations or not. When we call the function for a reservation, the jobs not in the reservation are not found and the function considers them ghost jobs. This should only happen when a reservation requests part of a node, and you have jobs running inside and outside a reservation.

Probably the best way to fix this issue is to just not print anything. The scheduler recovers from the ghost job race condition pretty well. The race condition happens pretty often on a busy system, so there is no real benefit to seeing this message.

This should be a pretty simple fix if you want this fixed now. I would suggest just removing the message and building PBS. You could even submit a PR if you wanted to give back to the community.

To build PBS, see:
https://pbspro.atlassian.net/wiki/spaces/PBSPro/pages/13991940/Building+PBS+Pro+Using+rpmbuild