Bad VMs leading to Inaccurate results

[treosh]

Bad VMs leading to Inaccurate results

[alekseykulikov]

Thank you @exterkamp for a great research!
I hope it will go away as the Github Actions platform matures. Otherwise, we have to request a new VM to provide consistent performance testing results. Any ideas on how to force a new VM and not fail the workflow?

[paulirish]

Reading https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops it doesn't provide many expectations around CPU load.. or any options for it.

But it does say "(Linux only) Run steps in a cgroup that offers 6 GB of physical memory and 13 GB of total memory" so that's something.

cc @connorjclark

[paulirish]

Oh and much like azure pipelines... github actions apparently also allows self-hosted runners: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners

[alekseykulikov]

As a solution, we may collect Lighthouse reports using PageSpeed Insights. It's more consistent (https://treo.sh/demo/6) and will allow running audits in parallel to speed up a build.

- name: Audit URLs using Lighthouse
  uses: treosh/lighthouse-ci-action@v2
  with:
   urls: |
      https://example.com/
      https://example.com/demo
      https://example.com/dashboard
   psiToken: page-speed-insights-token
   runs: 3 # use median for performance runs