Measuring Core Web Vitals (RUM)

[tastejs]

Measuring Core Web Vitals (RUM)

[addyosmani]

Thanks Anton. In this case, I think Lighthouse CI using GitHub actions might be the perfect fit for continuous performance recordings. When I've used them, I've found the clear "diff" of user-experience changes in the enhanced Lighthouse CI report to help me understand what metrics or audits were seeing a shift.

Perhaps for this project, we can setup Lighthouse CI and only do periodic checks using WebPageTest where we feel we really need to be able to compare the filmstrip or waterfalls the way they visually present them?

[anton-karlovskiy]

Hi @addyosmani @pi0 @danielroe

Could you please pause merging any PRs to the dev branch until we have set up measurement integrations like Lighthouse or WebPageTest etc?

Now as we have set up Vercel configuration, we can obtain a unique URL for a PR.
I think the purpose of getting such a unique URL is to measure the performance like Core Web Vitals at a specific point.
Actually, in the last Next.js project, I used to run Lighthouse and WebPageTest after a major PR of changes, and it was sort of hard work that consumes a lot of time. :)

I think it's very important for us to know whether a specific PR brings improvement or regression in the performance so that we can make sure that our optimization approaches have been right and correct them if a regression occurs at an early stage.
To be honest, SpeedCurve is good for monitoring the trends of the performance but it runs testing daily not per PR.
As far as I'm concerned, it's good for us to have a system where performance testing is done per PR rather than daily.

I think we can set up an automatic measuring process as a CI/CD action.
I guess there are some options:

• Vercel integration
  
• Lighthouse CI and GitHub Actions
  https://css-tricks.com/continuous-performance-analysis-with-lighthouse-ci-and-github-actions/

I'm going to dig into which one is fit for us. @addyosmani I'd like your suggestions here. :)
I guess it would be ideal if we could use Vercel Analytics but it appears that it's not available for Nuxt.js projects.

Re: nuxt-modules/web-vitals#13, https://github.com/vercel/nuxt-plugin-vercel

[anton-karlovskiy]

Thanks Anton. In this case, I think Lighthouse CI using GitHub actions might be the perfect fit for continuous performance recordings. When I've used them, I've found the clear "diff" of user-experience changes in the enhanced Lighthouse CI report to help me understand what metrics or audits were seeing a shift.

Perhaps for this project, we can set up Lighthouse CI and only do periodic checks using WebPageTest where we feel we really need to be able to compare the filmstrip or waterfalls the way they visually present them?

Hi @addyosmani
I've checked all about Lighthouse CI using GitHub actions and found some issues.

1. The main one is the period during which the collected Lighthouse reports will remain in temporary-public-storage.
   
   We would have to set up Lighthouse CI Server in order to keep the records for as long as we want.
2. And the other one is that it seems like Lighthouse CI would run locally rather than against the Vercel URL.
   

On the other hand, it appears that the Lighthouse integration via Vercel provides all we need:

I guess it's free too so I just wondered if using the Vercel's Lighthouse would be quicker.

What are your thoughts?

[danielroe]

You can set up a GitHub Actions workflow that triggers on Vercel deployments - it will receive the unique preview URL from Vercel and you can then run lhci (via Action or custom script) against that deployment URL.

[anton-karlovskiy]

@danielroe Thank you for your information.

We have already integrated Vercel into our repository so every time a PR against the dev branch is created, the unique preview URLs are being generated now.
The point is:

1. can we run LHCI against those preview URLs from Vercel, not from GitHub actions?
2. we would still have to set up Lighthouse CI server in order to keep the records for as long as we want.

cc @addyosmani

[danielroe]

I was responding particularly to this line in your previous comment:

And the other one is that it seems like Lighthouse CI would run locally rather than against the Vercel URL.

More details here

[anton-karlovskiy]

@addyosmani

I've tried Vercel's Lighthouse and I think we do not have much control over the process.

For example, it appears that we are not able to set up the number of counts, several URLs etc that we can do with Lighthouse CI using GitHub actions.

https://vercel.com/nuxt-movies/nuxt-movies/integrations/icfg_nYDnSDMCuBZzqdMbpRsmD5ac

[anton-karlovskiy]

@addyosmani

Could you please set up the repository secrets like so?
FRONTEND_URL: http://localhost:3000
API_KEY: 9a31326bc179b029cd4513c489628e79
API_LANG: en-US
API_COUNTRY: GB
API_YOUTUBE_KEY:

And also please set up up a GitHub status check by following https://web.dev/lighthouse-ci/#github-status-checks?
Thank you.

[addyosmani]

Hey Anton. The above steps should now be completed.  

[anton-karlovskiy]

@addyosmani

I've got a question about measuring the CLS.

I've tried measuring the CLS using Web Vitals Chrome extension.

• Version 87.0.4280.141 Google Chrome
  
• Version 90.0.4392.0 Google Chrome Canary
  

The measured CLS results are quite different.
Could you please teach me the reason?
Thank you.

[addyosmani]

One reason this may be is that CLS as implemented in Chrome has received some updates to better account for how developers render modern content in Chrome 88 and 89: https://chromium.googlesource.com/chromium/src/+/master/docs/speed/metrics_changelog/cls.md.

[anton-karlovskiy]

@addyosmani

Then which version should we rely on in checking CLS?

[addyosmani]

In this case newer Chrome is always the more reliable (Chrome 90) :)

[pi0]

Hi Addy, indeed that is the module. We moved best community implementation to nuxt-community org and almost done a full rewrite. I will try to publish it this weekend and updating you :)

[addyosmani]

I will try to publish it this weekend and updating you :)

That sounds promising. Thanks so much, @pi0! 😄

[anton-karlovskiy]

@addyosmani

Is Heroku being used here as a workaround for GitHub actions not doing the full suite of checks? I'd rather that we fix this directly vs. adding another service into the mix. Here's what I will do tomorrow (today is public holiday in the U.S): I'll add our maintainer for LHCI to the repo and ask for his help debugging what we are seeing. Ideally it shouldn't require someone self-hosting to work around this issue (I worry if we're having to do this the product is not working correctly for others...).

Now I think we can see the full suite of checks:

The reason why I think we should use Heroku as a Lighthouse CI Server is to keep the records for long term and other benefits the Lighthouse CI Server can give like:

The Lighthouse CI server provides a dashboard for exploring historical Lighthouse reporting. It can also act as a private, long-term datastore for Lighthouse reports.

Re: https://web.dev/lighthouse-ci/#server-setup