treosh/exthouse

Analyze the impact of a browser extension on web performance.

Exthouse

Analyze the impact of a browser extension on web performance.

Motivation

When measuring real user performance engineers take to the account factors like device and network conditions. But there is one more factor, that is not in direct control - web extensions. They add additional scripts, DOM manipulations, and impact the user experience.

Exthouse is a tool powered by Lighthouse that provides a report about web extension impact on web performance. It measures an extension performance score that helps developers to improve the performance of their extensions and web in general.

Goals:

1. Highlight one more performance factor affecting web performance.
2. Identify web extensions that harm web performance.
3. Provide developers with reports they can use to improve performance.
4. Show that desktop users may experience unexpected performance issues related to web extensions.
5. Try to figure out the way to notify users about a negative impact of the installing extension, at best at Chrome Web Store level.

Methodology

Exthouse performs several steps to do analysis:

1. Launches a browser without extension to evaluate the default performance and store results ./exthouse/result-default-1.json

2. Launch a browser with installed extension using Puppeteer and stores results to ./exthouse/MY_EXTENTION-1.json

3. Extends Lighthouse performance categories with additional audits to estimate the impact of the extension:

   • exthouse-new-long-tasks - The value represents a sum of Long Tasks. Long Tasks (weight: 1).
   • exthouse-max-potential-fid-change - The change for the longest task duration highlights the impact on potential First Input Delay (weight: 1).
   • exthouse-extension-files - Extension files add extra CPU consumption for every URL visit. Bundle resources into one and leverage hot chaching. Learn more (weight: 1).
   • exthouse-default-metrics - All metrics collected from the default run (without extension) (weight: 0).

4. Generates Lighthouse style report using the Lighthouse scoring algorithm.

Environment conditions:

• Browser: Chromium
• Emulated form factor: desktop
• CPU slowdown multiplier: 2
• More settings in Lighthouse config.

Most of the extensions add tasks to the main thread and affect interactivity metrics:

• Time to Interactive (TTI) - Time to interactive is the amount of time it takes for the page to become fully interactive. Learn more.
• First Input Ddelay (FID) - The change for the longest task duration highlights the impact on potential First Input Delay. Learn more.

Analysis of top 10 extensions from Chrome Web Store

This analysis evaluates the top 10 extensions from Chrome Web Store by users count. Extensions are manually filtered to exclude login requirement, not relevant extensions in categories like PLATFORM_APP, or related to specific URLs like *://*.google.com/*.

Name	Score	Users Count	FID Δ ( ms )	Scripting Δ ( ms )
Grammarly for Chrome	50	10M	114	530
Adblock Plus	59	10M	118	760
Skype	82	10M	150	120
Avira Browser Safety	94	10M	60	30
Avast SafePrice	99	10M	62	0
AdBlock	100	10M	0	0
Google Translate	100	10M	0	0
Pinterest Save Button	100	10M	0	0
Tampermonkey	100	10M	0	0
uBlock Origin	100	10M	0	0

Usage

Install CLI using npm:

npm install --global exthouse

exthouse --help

Usage: exthouse [path/to/extension.crx] [options]

Options:
  --runs <number>    amount of runs to evaluate median performance value (default: "1")
  --url <url>        url to evaluate extension performance (default: "https://example.com/")
  --format <format>  output format options: [json,html] (default: "html")
  --disableGather    disable gathering and use /exthouse to produce results
  -V, --version      output the version number
  -h, --help         output usage information

CLI usage examples

# Evaluate extensions with several runs.
# It performs do 3 runs, get median value and generate a report.
exthouse Grammarly-for-Chrome.crx --runs=3`

# Generate a report based on existing data:
# It reads results from `/exthouse` folder and generate report.
exthouse Grammarly-for-Chrome.crx --disableGather

# Output report in json format
exthouse Grammarly-for-Chrome.crx --format=json`

Evaluate any extension

1. Download extension using https://chrome-extension-downloader.com/
2. Copy path to the MY_EXTENTION.crx and pass to cli exthouse MY_EXTENTION.crx --runs=3
3. The process takes a few minutes and results are stored in the Lighthouse report.
4. All debug data is stored in exthouse folder.

Find downloaded examples extensions folder.

Future Work

• add support for login script to test extensions required authentication (#22)
• perform extensions analysis required authentication and compare score w/o authentication. #25
• experiment with cache (try: warm, hot) to see how scripts are effected by v8 caching. More about cache.
• experiment with results, running in Chrome and Edge. Add flag browserBinaryPath
• expose node.js API (#24)
• experiment with Firefox add-ons (all related work is in the branch firefox-experimental).
• make repo smaller using bfg-repo-cleaner.

Credits

This tweet has kick-started the initial research and this project.

Development is sponsored by Treo.sh - Page speed monitoring made easy.

exthouse's Issues

Error: Unexpected crx format version number

I tried to evaluate the final version of Block Origin.

1. Download extension using https://crxextractor.com/ (Because on https://chrome-extension-downloader.com/, when I input the URL of Block Origin. (https://chrome.google.com/webstore/detail/ublock-origin/cjpalhdlnbpafiamejdnhcphjbkeiagm?hl=en), It shows the. error: Empty Extension!)

2). Copy path to the ublockOrigin.crx and pass to cli: exthouse MY_EXTENTION.crx --runs=3
It shows error:

Unexpected crx format version number.
Run:
exthouse --help

Do you have any suggestions for me?
Note that, when I used the extension in your folder. (https://github.com/treosh/exthouse/tree/master/extensions), it works ok.
.

npm install error

I got an error when installing exthouse, saying that

ERROR: Failed to download Chromium r686378! Set "PUPPETEER_SKIP_CHROMIUM_DOWNLOAD" env variable to skip download. Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/exthouse/node_modules/puppeteer/.local-chromium'

There is a workaround for people who also got this error. Try:

sudo npm install --global exthouse --unsafe-perm=true --allow-root

Bug: throw exception :(

After simple run of exthouse recived an exception:

Error: Protocol error (Tracing.end): Tracing failed
    at Function.fromProtocolMessage (/Users/gregoryshehet/.nvm/versions/node/v12.14.1/lib/node_modules/exthouse/node_modules/lighthouse/lighthouse-core/lib/lh-error.js:132:19)
    at /Users/gregoryshehet/.nvm/versions/node/v12.14.1/lib/node_modules/exthouse/node_modules/lighthouse/lighthouse-core/gather/connections/connection.js:123:25
    at processTicksAndRejections (internal/process/task_queues.js:94:5) {
  protocolMethod: 'Tracing.end',
  protocolError: 'Tracing failed',
  friendlyMessage: undefined
}
Cannot destructure property `extFile` of 'undefined' or 'null'.

So I tried to fix it locally by wrapping extFile destructor inside if condition:

 if (completeRes[medianIndex]) {
      const { extFile } = completeRes[medianIndex]
      const medianFileName = join(tmpDir, `median-${extFile}`.replace(/-\d+(?=\.\w+$)/, ''))
      await remove(medianFileName)
      await symlink(extFile, medianFileName)
}

But got another error, and think to create an issue :)

ENOENT: no such file or directory, open '<SOME_PATH>/median-result-collamark.json'

So I created such a file manually. Some note, script creates another json file median-result-default.json with a new line symbol in the end '\n' of the file name.

After I received an error:

Error: Protocol error (Tracing.end): Tracing failed
    at Function.fromProtocolMessage (/Users/gregoryshehet/work/extension-perfomance/node_modules/lighthouse/lighthouse-core/lib/lh-error.js:132:19)
    at /Users/gregoryshehet/work/extension-perfomance/node_modules/lighthouse/lighthouse-core/gather/connections/connection.js:123:25
    at processTicksAndRejections (internal/process/task_queues.js:94:5) {
  protocolMethod: 'Tracing.end',
  protocolError: 'Tracing failed',
  friendlyMessage: undefined
}
Cannot read property 'main-thread-tasks' of undefined

It seems there is some bugs in the library.

Consistent scoring between different URLs

Hello,

First of all thanks for the work you've done ! it looks very promising !
I had some question about how you guys calculate the new score ? I imagine there's some sort of comparaison between the score with and without the extension ?

I wonder how the url you pass in as test url can impact the new score !
Indeed I tried it with a plugin we've created that display a bar when you arrive on a partner website.

If I go on a website that is not a partner it does nothing.

What I expected was to have a difference between the partner and non partner site as there is more javascript executed on partners' sites.

Here is the scrore I had by running the exthouse command with --runs=3

• Non partner => 98 (pretty good)

• Partner 1 => 53

• Partner 2 => 98

• Partner 3 => 70

The script executed should be the same between the three partners !

Do you guys have any idea how this can happen ?

Thanks again for the great work !

Investigate js-self-profiling API

Currently we track additional CPU work with kinda Long Tasks API and intention to do that for Firefox is having this API implemented.

In case we can relay on js-self-profiling API, we have second option to have Firefox support sooner (depends on which API implemented sooner).

js-self-profiling under implementation in Chrome, hence we should just enable it via puppeteer.

Analyze extensions performance impact for Firefox

The purpose of the project improving speed on a web. All results we were gathering related only to Chromium mostly. We’d like to extend results and see extensions (Add-ons) influence for Firefox users.

Some part of work was already done.
We are able to install add-ons using [Puppeteer(https://github.com/GoogleChrome/puppeteer).
Thanks for the help to extend API for web-ext repo to @aslushnikov @rpl @Rob--W.

More info can be found puppeteer/puppeteer#4162 and puppeteer/puppeteer#4162

Next step - use performance API which isn’t implemented yet in Firefox.
Related bugtracking issues:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1463069
• https://bugzilla.mozilla.org/show_bug.cgi?id=1348405

All experiments are stored separate branch - https://github.com/treosh/exthouse/tree/firefox-experimental

Analyze extension score w/o authentication

Motivation:

Honey extension requires authentication, but even without login, it affects perf.

It will be really interesting to know the results after authentication, compare how FID and other values changes between these 2 cases.

expose node.js API

Need to figure out final options and response as array or object.

custom login script

Add support for custom login script to evaluate extensions requiring authentication (more than 50% of all extensions).

Solution

Puppeteer powers Exthouse and allows us to get an extension page. A custom script could do some extra steps to finish the authentication and extension install.

Possible API

login.js

exports.handler = async (extension, page) => {
  // perform steps for login
}

Use the login script:

exthouse my-ext.crx --require=login.js

Extensions candidates to be added

• Social Fixer - 179,196 users, probably has influence on user interaction. Since social media used a lot, looks like a good candidate.

anyone has more?