Failing to create LVM Volumes (due to Github Runner changes)

[easimon]

Failing to create LVM Volumes (due to Github Runner changes)

[easimon]

According to https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#cloud-hosts-used-by-github-hosted-runners , GitHub has started to use different Azure VM types for public repositories -- which is consistent with what you see.

What all runner VMs have in common, is that they have two disks/partitions mounted. /, which hosts both the operating system installation and the working directory of the build, and /mnt, which is, by default, used for a small swap file only, leaving unused space on /mnt. I created this action back then mainly to utilize these otherwise unused ~14 GB of temp space on the previous runner generation.

Previously, the public runners were DS2_v2 VMs (2 vCPUs, 7GB RAM). This is a quite old VM generation, and I assume, Azure will retire them at some point -- or at least they do not add new capacity. One way or the other, this might have gotten a scarce resource for GitHub. These runners have (if I recall correctly) 84 GB on / and 14 GB on /mnt.

Now they seem to have switched to D4ads_v5 (4 vCPUs, 16GB RAM), which is the most recent VM generation on Azure at the moment, and which comes with a different disk layout. It seems like they just cut the 150GB disk the VM has as a "ephemeral disk", in half: 75GB for / and 75 GB for /mnt).

The 29 GB @vynight mentioned on root were actually never "guaranteed". Instead, GitHub always documented the old runners to have 14 GB of disk space, which was incorrect all the time. The 14 GB were the size of the /mnt volume on the old generation, which was usually unused - and the 29GB, were the free space available on / -- which is what was left after installing the ubuntu image on a 84 GB(?) disk, and it always varied a bit because of software updates.
This (the / partition) is also where also the work directory resides.

But it seems like this news can be turned into good news for many:

• The new VMs have double the vCPU cores and more than double the RAM
• While the root volume has gotten smaller, the total disk space has increased (~150GB instead of ~100GB)

The downside is, that the root volume is now around 10GB smaller, and you might have to fiddle with the "root reserve" parameter to get your build green again. For some, these 10GB might have been crucial (e.g. because of large docker images residing in /var or other things), and these are out of luck for now. I never tried to fiddle with moving /var/lib/docker around as a part of this action, because I assume this is also used by GitHub container actions, and might break easily.

Some side notes

• I always considered this action a hack for the desperate. Any change to the GitHub runners (VM types, OS version) may totally break the action (and it happened in the past). GitHub does not guarantee the given amount of disk space, nor the disk layout.
• Actually, I do not understand why GitHub partitions the disk on the new runner VM types, downsizing the root volume compared to the old gen, and leaving so much disk space (usually) unused on /mnt. It should be easy for GitHub to change the partition layout to not be 75GB/75GB, but e.g. 100GB/50GB. It should even be possible to just create one 150GB root volume, with no /mnt at all. This is probably not true for the old VM types, where the 14GB temp disk is separate.

[mkoeppe]

We started to see this too, e.g., in https://github.com/sagemath/sage/actions/runs/7810559253/job/21304178537?pr=36181#step:2:286

[easimon]

Out of curiosity: why do you require so much free space on the root partition? Can't you rework your build to put everything in the workspace folder (which is where this action makes space)?

[vyknight]

Out of curiosity: why do you require so much free space on the root partition? Can't you rework your build to put everything in the workspace folder (which is where this action makes space)?

Since we only needed a little bit more than the 29GB that were given to us, it was easier to just clear out the root directory and not bother about anything else. At least that's what I suspect since I did not configure our actions.

[mkoeppe]

why do you require so much free space on the root partition?

"docker build" needs space in /var

[mkoeppe]

Thanks very much for sharing this detailed analysis! Adjusting root-reserve-mb accordingly did the trick for us (sagemath/sage#37252).

[vyknight]

This worked for us as well, thank you so much.