Resolve long wait times for hosted runners

[microsoft]

Resolve long wait times for hosted runners

[MIchaelMainer]

Configured to support two agents.

[gavinbarron]

We know the work around at present and it's annoying. but workable.

We're hoping to be able to use this repo as a blueprint for how we configure our open source repos going forward so it's important to me that we try to work with the 1ES systems as much as we can.

Given that I'd like to stick with the 1ES hosted agents and perhaps @MIchaelMainer can investigate the option of recreating the runner to see if that has any impact. If it becomes too onerous, we should pivot to using the GitHub runners.

[MIchaelMainer]

Ugh, I'm sorry this is happening. I just canceled the last remaining job. It appears that some jobs got picked up and completed. Did you do any actions that resulted in those jobs getting picked up?

I may need to go back and recreate the runner as believe there is some brittleness around setting config at creation time versus update. I don't think there is much you can do but cancel jobs and wait some time for the vm to be deallocated before requesting a new run.

[MIchaelMainer]

@waldekmastykarz

I suspect that this continuous getting stuck state is happening when a VM is allocated. If the VM stays allocated and is in the bad state, the subsequent jobs regardless of whether they are new or rerunning of existing jobs, won't be assigned. If the VM is shut down, and existing jobs are cancelled, and a new VM is allocated in response to the running of a job, that may be how we get unstuck.

If this is too disruptive, we can go back to using GitHub images, and we can figure this out later. Probably more like retry later and hope that the runners are more reliable by then.

[gavinbarron]

Yes, closing

[waldekmastykarz]

@gavinbarron can we consider this issue as closed given @MIchaelMainer comment #36 (comment)?

[waldekmastykarz]

Hey @MIchaelMainer, @gavinbarron, it seems like the issue still not solved unfortunately. Here's a screenshot of recent tasks that have been queued for over 11 hours:

We had something similar with the recent release that was queued for almost 20h and had to be restarted: https://github.com/microsoftgraph/msgraph-developer-proxy/actions/runs/3831328422/attempts/1.

Is there anything else that we could check to address this issue?

[waldekmastykarz]

Thank you, and I appreciate your help @MIchaelMainer. The only thing I've done was to follow your last recommendation, cancel and then re-start the jobs stuck in queue.

[waldekmastykarz]

@gavinbarron any preference for the path forward from your side?

To me it's an inconvenience but just that. We know that it can happen and we know a workaround. If switching back to GitHub images has other downsides, then we should keep the current setup.

[waldekmastykarz]

Shall we close this issue for now then and reopen in the future if need be?