krausest/js-framework-benchmark

A comparison of the performance of a few popular javascript frameworks

js-framework-benchmark's Issues

[Suggestion] Record Framerates in addition to JavaScript execution times

A lot of these frameworks touch the DOM only sparingly and such frameworks should also measure frame rates in addition to just JavaScript execution times. Even if the JavaScript takes a lot of time to run, frameworks could technically stagger it such that the frame rates are not impacted.

Frame Rates can be calculated in the same way Chromium's perf test suite calculates is - using Log entries like DrawFrame and BenchmarkImplStats. If this is something the project could use, I could send in a pull request with the change.

Switch to User Timing API

I think the good idea would be to switch to User Timing API instead using performance.now. With performance.mark you can only mark timestamps without much impact on test and after all tests have passed performance.measure should be used to calculate.

References:
https://www.html5rocks.com/en/tutorials/webperformance/usertiming/
https://developer.mozilla.org/en-US/docs/Web/API/User_Timing_API

Update all results

There may be some wrong in the result. binding.scala-v10.0.1_06_remove-one-1k should not be as slow as the result.

On my computer, binding.scala-v10.0.1_06_remove-one-1k is 10 times faster than vanillajs_06_remove-one-1k !

after initialized  06_remove-one-1k 0 binding.scala-v10.0.1
after run  06_remove-one-1k 0 binding.scala-v10.0.1
result binding.scala-v10.0.1_06_remove-one-1k [ { type: 'click', ts: 138156995841, dur: 348, end: 138156996189 },
  { type: 'paint', ts: 138157055751, dur: 1099, end: 138157056850 },
  { type: 'gc', ts: 138156900336, mem: 36.346763610839844 } ]

after initialized  06_remove-one-1k 0 vanillajs
after run  06_remove-one-1k 0 vanillajs
result vanillajs_06_remove-one-1k [ { type: 'click', ts: 138258041670, dur: 3743, end: 138258045413 },
  { type: 'paint', ts: 138258123467, dur: 1444, end: 138258124911 },
  { type: 'gc', ts: 138258135172, mem: 41.954978942871094 } ]

I suspect that vanillajs_06_remove-one-1k benchmark was running on a different Chrome version, which is the cause of the difference.

(Moved from #94)

please update to angular-v2.4.2

please update to angular-v2.4.2

consider adding bundle size and parse/init time metrics

probably not too difficult, but really useful.

Inconsistency in implementation of 10th row updates

Howdy. I was looking at a few implementations to see how differences are normalized and noticed some significant differences in how the "every 10th row" code is implemented:

• Inferno implementation
• Ember implementation

The Inferno implementation mutates the array, where the Ember one rebuilds the entire set of data. Which is the intended implementation?

Further, it seems like the data generation itself should perhaps not be included in the benchmarked values (though if the items in the array are part of a change-tracking solution like Ember objects or MobX it seems unfair to exclude that work, so maybe not).

Webdriver doesn't start on Chrome 54

Webdriver doesn't run, instead throwing an error:

SessionNotCreatedError: session not created exception

If you update chromedriver to 2.24.1 this issue goes away but I can't get chromedriver to open a window. :/

binding.scala doesn't build on windows

Works on OSX, but fails on windows (see #91).

Updating Preact to 7.1.0

Why? https://twitter.com/_developit/status/804830051531456513

vanillajs-keyed "replace all rows" baseline is wrong

@krausest

vanillajs-keyed doesnt actually replace rows for "replace all rows", which is why it's so fast while other keyed implementations are slower. at least for the keyed version, it needs to do the correct thing (which should clock in at slightly faster than the kivi impl)

i made a DOM-instrumented version of of vanillajs-keyed [1] [2]. you can watch the console to see a summary of DOM ops performed for any action.

feel free to use my DOMInstr lib (it's MIT) to check what the other implementations do when porting to keyed versions.

[1] https://rawgit.com/leeoniya/js-framework-benchmark/vanillajs-keyed-instrumented/vanillajs-keyed/index.html
[2] leeoniya@8f1acaa

Add Elm

Is there any chance we can add Elm to this benchmark, I'd love to see another benchmark like the creator's blog benchmark in this suite of benchmarks!

Let me know if that seems appropriate or not :-)

hope add choo

https://github.com/yoshuawuyts/choo

React one can be a little faster

Didn't try but in theory by making the static part a component and mark shouldComponentUpdate: -> false will make it faster.
https://github.com/krausest/js-framework-benchmark/blob/master/react-v15.3.1/src/Main.jsx#L94

My Macbook Air is too slow to try...

Add a painfully imperative implementation

@krausest

The vanillajs implementation tries to be a purpose-built vdom lib which means it has to do keyed lookups in the data, do diffing, etc. This is not the fastest possible implementation of the requirements. Since there is no explicit requirement for one-way data flow from a datastore, you could create an implementation that is probably 30%-50% faster (and maybe smaller) with jquery-like semantics. For example, clicking "X" would literally remove the clicked node (via a delegated handler) and simultaneously remove the backing item from the store.

This would really be the true baseline for all other implementations. And you could theoretically drop vanillajs altogether rather than maintaining it with whatever new perf hacks are discovered by fast vdom libs.

If you prefer a bit of fluent sugar without using jQuery, cash.js, sprint.js you can use the tweet-sized bala.js [1].

[1] https://github.com/finom/bala

Will you be interested in my in-depth comparisons between mobx & redux?

Hi, @krausest . I've done these in branch benchmark/mobx-vs-redux my fork. 8 more columns have been added to the result table. Don't know if you would like a PR for this, as it might goes too much detail into inferno & react, which's a little bit off the intention of this repo.

New comparisons.

• inferno v0.7.26-mobX-v2.4.4
• inferno v0.7.26-redux-v3.5.2
• inferno v0.7.26-redux-v3.5.2-immutable-js-v3.8.1
• inferno v0.7.26-redux-v3.5.2-immutable-js-v3.8.1-pure-render-mixin-15.3.0
• react v15.3.0-mobX-v2.4.2
• react v15.3.0-redux-v3.5.2
• react v15.3.0-redux-v3.5.2-immutable-js-v3.8.1
• react v15.3.0-redux-v3.5.2-immutable-js-v3.8.1-pure-render-mixin

50 shades of red

all red buckets are not created equal, though most green and yellow are. looking at [1], "select row" 13.38ms is the same color as 73.94ms.

it would be useful to have the color scales be tuned to be statistically meaningful within each test. 4300ms and 500ms are very different levels of unacceptable when the baseline is 207.55ms :)

[1] https://cdn.rawgit.com/krausest/js-framework-benchmark/4e47158a10e52122bf4244724cf99b6de4ef7ac1/webdriver-java/table.html

Solution to tracking or non-tracking (keyed vs. non-keyed)

I'd like to ask for feedback to fix the tracking or non-tracking (keyed vs. non-keyed) issue. Please keep all general discussion out of this issue and put it into issue #22 instead.

Problem: Non tracking / non-keyed frameworks are faster than those which track / use keys. Regarding this benchmark non tracking is a correct solution. The "big" frameworks like react and angular can also perform much faster if track by if is disabled as can be seen here.
The real problem is that for real life applications there's a big chance that non-tracking will cause wrong results (e.g. for transitions or dom manipulation outside of the framework).

Goal: Require all frameworks to use a tracking implementation.

Proposal: Add a simple functional requirement: The button to remove a row shall show a zoom transition when holding the mouse button. After releasing the button the transition must be stopped immediately, the next row must not show a zoom effect.
This is a very simple change that just needs an update of the css (74e7436) and might make the results a bit less biased and shouldn't influence the measurement via webdriver.

The following frameworks currently appear to be non-tracking :

• aurelia
• cycle.js
• dio
• domvm
• inferno
• riot
• TSERS
• vanillajs
• vue

(please note that kivi is not included in that list)

What do you think?

Add build times into the benchmarks

Let's face it – long build times suck. They affect CI and CD builds with automated tests/builds and effect local build times too. I feel it's important to emphasise this in the benchmarks as it's a HUGELY important metric in any job where engineering of software is important.

Benchmarks don't describe resources used to test

Pardon me if I'm missing it. But for these comparisons, yes, it's good to have a relative value. But it would be good to know what machine is you're running these on.

Update riot to v3.0

http://riotjs.com/release-notes/

Vanilla version can be faster (maybe a lot)

Hi @krausest

Here are some notes on boosting the vanilla impl.

get/setAttribute is slow, use node properties when available:

setAttribute("class") -> .className
elem.getAttribute("data-id") -> elem.dataset.id
elem.setAttribute("data-id") -> elem.dataset.id

elem.innerText -> elem.textContent (when creating) (wont boost perf, but is a better practice)
...and use elem.firstChild.nodeValue (when updating)

.childNodes is slow; firstChild, lastChild, nextSibling, previousSibling are fast.

tr.childNodes[0] -> tr.firstChild
tr.childNodes[1].childNodes[0] -> tr.firstChild.nextSibling.firstChild

removeAllRows() -> tbody.textContent = ""

createDocumentFragment is no longer necessary in modern browsers [1], it's actually slower than plain appendChild and insertBefore

the vanilla version uses event delegation, while many implementations instead attach 1000's of onclick handlers, this is not an apples-to-apples comparison, especially when the libs allow for event delegation (many do).

also, I noticed the vanilla version does not maintain the row highlight (is present) during mutations/appending, etc.

/$0.02

[1] infernojs/inferno#223 (comment)

npm install failure

did a brew update.. Have the latest everything.. should I just revert to [email protected] ? or do I need to manually add a repo?

> [email protected] postinstall ~/Downloads/js-framework-benchmark/ember-v2.6.1
> bower install

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/6.2.2/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5

npm ERR! No compatible version found: [email protected]
npm ERR! Valid install targets:
npm ERR! 1.0.0-alpha8, 1.0.0-alpha7, 1.0.0-alpha5, 1.0.0-alpha4, 1.0.0-alpha3, 1.0.0-alpha2, 1.0.0-alpha11, 1.0.0-alpha10, 1.0.0-alpha1, 0.8.0-alpha6, 0.8.0-alpha4, 0.8.0-alpha3, 0.8.0-alpha2, 0.8.0-alpha1, 0.7.27, 0.7.26, 0.7.25, 0.7.24, 0.7.23, 0.7.22, 0.7.21, 0.7.20, 0.7.17, 0.7.16, 0.7.15, 0.7.14, 0.7.13, 0.7.12, 0.7.11, 0.7.10, 0.7.9, 0.7.8, 0.7.7, 0.7.6, 0.7.5, 0.7.4, 0.7.3, 0.7.2, 0.7.1, 0.7.0, 0.6.4, 0.6.3, 0.6.2, 0.6.1, 0.6.0, 0.5.22, 0.5.21, 0.5.20, 0.5.18, 0.5.17, 0.5.16, 0.5.13, 0.5.12, 0.5.11, 0.5.10, 0.5.9, 0.5.8, 0.5.7, 0.5.6, 0.5.5, 0.4.5, 0.4.4, 0.4.3, 0.4.0, 0.1.11, 0.1.10, 0.1.9, 0.1.7, 0.1.6, 0.1.4, 0.1.3, 0.1.2
npm ERR!
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     ~/Downloads/js-framework-benchmark/inferno-v1.0.0-alpha7/npm-debug.log
child_process.js:529
    throw err;
    ^

Error: Command failed: npm install
    at checkExecSyncError (child_process.js:486:13)
    at execSync (child_process.js:526:13)
    at ~/Downloads/js-framework-benchmark/install.js:7:3
    at arrayEach (~/Downloads/js-framework-benchmark/node_modules/lodash/lodash.js:520:11)
    at Function.forEach (~/Downloads/js-framework-benchmark/node_modules/lodash/lodash.js:9202:14)
    at Object.<anonymous> (~/Downloads/js-framework-benchmark/install.js:5:3)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/6.2.2/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] postinstall: `node install.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script 'node install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ~/Downloads/js-framework-benchmark/npm-debug.log

consider removing unused bootstrap styles

a large part of this benchmark is browser restyling overhead which has no bearing on lib speed and is a fixed unnecessary cost for everyone. in many cases it also masks lib efficiency. if for no other reason than to allow the benchmark runs to complete in less time, it may make sense to strip out unused styles.

you can use something like https://github.com/purifycss/purifycss to avoid having to figure things out by hand.

unfair benchmarks vs real life tests

I spent a couple of hours going over the last additions of the benchmarks.

I noticed one thing that may be important:
Some frameworks have an extra overhead the first time you interact with the DOM.
You can observe that when you open the console and click several time the "Create 1,000 rows" button.

Some frameworks get a 5x factor on the next calls (the 1rst call is very slow). Among them:

• ractive
• riot
• react
• ...

Some have always the same performance:

• angular
• vue
• react-lite
  -vidom
  -...

I am still wondering whether it is unfair, or not?
I think the current way of doing this test (inserting a lot of rows on a blank page) is close to a real life test.
Do you?

Update React to 15.3.0

There’s been an unmounting time regression in 15.x that should be fixed in 15.3.0.
Would be great to see if it improves results!

React keyed and non-keyed are misclassified

Failure postinstall npm dependencies

Is probably not an issue related to this repo on its own, but most likely you guys can let the aurelia owners know that their package seems to be broken:

node v6.2.1
npm  v3.9.3
npm ERR! code ELIFECYCLE
npm ERR! [email protected] postinstall: `jspm install -y`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script 'jspm install -y'.

Highlighted row should stay selected when clicking swap rows, update and append

The highlight handling for swap rows, update and append is not consistent among the frameworks. The highlighted row should stay selected when clicking swap rows.

Update Inferno to 1.2.2

Apparently guys at Inferno managed to boost createElement() by 50%

Might be interesting to bench it ?

split into 2 tables "keyed" and "non-keyed" ?

Looking at the table [which is getting way too wide IMO], the perf numbers are very difficult to compare because many impls are keyed and many are unkeyed. these are really 2 completely different perf profiles.

if you prefer to have a single huge table at the very least, it would make sense to order/group them by keyed vs non-keyed. a trivial way to see this is to assume everything < 130ms for "replace all rows" is unkeyed and > 130ms is keyed.

i think 2 tables will make sense and would provide much more meaningful comparisons (and also not span a 2 x 1080p screens :)

the vanillajs-keyed impl would need to be fixed for this as well: see #103

Is it possible to run the tests on mobile devices?

As the title states: I would like to run the benchmarks on my android phone. Is that possible?

[Suggestion] Convert JS Benchmark to use Javascript instead of Java

Given that most of the code here uses Node, it would be interesting to convert the WebDriver tests that use Java into Node based tests - will remove dependency on Java from this repo.

I have written a [tool] that runs performance benchmarks and uses a Node based test suite. If this is something you are ok with, I can send in a Pull request.

setting `this.start` in angular 1.5?

Looking at https://github.com/krausest/js-framework-benchmark/blob/master/angular-v1.5.8/src/HomeController.es6.js#L4 i see each part sets this.start to perf.now() but afaik this isn't used anywhere.

is it neccessary?

(Haven't looked to see if this is used in any other ones)

kivi-v0.11.0-alpha3 fails to build

I've had several people ask me why this is: it appears to be a premature merge as kivi-v0.11.0-alpha3 does not build. @localvoid @krausest

I've spoken to @localvoid and he knows what the issue is, so hopefully he can make a PR to fix.

Aurelia specific logic

Can you please, explain why we need this long tsAfter = isAurelia ? tsEventFire : tsEvent; Thanks.

Ractive Add Test Optimization

You can optimize the ractive test by using "array.push" rather than providing a completely new array for ractive to process.

add() {
        // this.data = this.data.concat(this.buildData(1000));
        this.data.push.apply(this.data, this.buildData(1000));
    }

sub-optimal "append rows to large table" in vanillajs (keyed and non-keyed)

@krausest kivi and inferno score non-trivially better than vanillajs in "append rows to large table" for both keyed and non-keyed impls. something to look into.

Please add Riot.js

http://riotjs.com

Updated infernojs to v1.1.1

https://github.com/infernojs/inferno/releases/tag/1.1.1

add cycle.js into benchmark

Please add cycle.js into your benchmark.

https://cycle.js.org/

Thanks!

Vue.js measurements missing?

I don't want to create a pull request for that
;-)

'use strict';

var Vue = require('vue');
var {Store} = require('./store');

var store = new Store();

var startTime;
var lastMeasure;
var startMeasure = function(name) {
startTime = performance.now();
lastMeasure = name;
}
var stopMeasure = function() {
var last = lastMeasure;
if (lastMeasure) {
window.setTimeout(function () {
lastMeasure = null;
var stop = performance.now();
console.log(last+" took "+(stop-startTime));
}, 0);
}
}

new Vue({
el: '#main',
data: {
rows: store.data,
selected: store.selected
},
methods: {
add() {
startMeasure("add");
store.add();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
remove(item) {
startMeasure("remove");
store.delete(item.id);
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
select(item) {
startMeasure("select");
store.select(item.id);
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
run() {
startMeasure("run");
store.run();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
update() {
startMeasure("update");
store.update();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
runLots() {
startMeasure("runLots");
store.runLots();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
clear() {
startMeasure("clear");
store.clear();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
},
swapRows() {
startMeasure("swapRows");
store.swapRows();
this.rows = store.data;
this.selected = store.selected;
stopMeasure();
}
}
})

Building (npm run build) doesn't work

Hello!
Building doesn't work, it's log I have

# node --version
v6.4.0

# git clone https://github.com/krausest/js-framework-benchmark
# cd js-framework-benchmark/

# npm install
# npm run build

> [email protected] build /tmp/0/js-framework-benchmark
> node build.js


> [email protected] build-prod /tmp/0/js-framework-benchmark/angular-v1.5.7
> webpack -p -c webpack.config.js

sh: 1: webpack: not found

npm ERR! Linux 3.13.0-93-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build-prod"
npm ERR! node v6.4.0
npm ERR! npm  v3.10.3
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] build-prod: `webpack -p -c webpack.config.js`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the [email protected] build-prod script 'webpack -p -c webpack.config.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark-angular package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     webpack -p -c webpack.config.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark-angular
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark-angular
npm ERR! There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?

npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/0/js-framework-benchmark/angular-v1.5.7/npm-debug.log
child_process.js:526
    throw err;
    ^

Error: Command failed: npm run build-prod
    at checkExecSyncError (child_process.js:483:13)
    at execSync (child_process.js:523:13)
    at /tmp/0/js-framework-benchmark/build.js:7:3
    at arrayEach (/tmp/0/js-framework-benchmark/node_modules/lodash/lodash.js:522:11)
    at Function.forEach (/tmp/0/js-framework-benchmark/node_modules/lodash/lodash.js:9011:14)
    at Object.<anonymous> (/tmp/0/js-framework-benchmark/build.js:5:3)
    at Module._compile (module.js:556:32)
    at Object.Module._extensions..js (module.js:565:10)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)

npm ERR! Linux 3.13.0-93-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v6.4.0
npm ERR! npm  v3.10.3
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: `node build.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script 'node build.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node build.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /tmp/0/js-framework-benchmark/npm-debug.log

non-keyed description is not true for Binding.scala

In the section Non keyed results on page http://www.stefankrause.net/js-frameworks-benchmark5/webdriver-ts/table.html:

In consequence inserting or deleting an element in the data array might append after or delete the last table row and update the contents of all elements after the inserting or deletion index.

It's not true for Binding.scala. Thanks to its precise data binding mechanism, the DOM managed by Binding.scala always precisely follows the same changes on upstream data.

installation issue

Just wanted to try on my machine, but errors occured at end of npm install:
npm-debug.txt

EDIT: same goes for npm run build, it fails at end with "unknown command 'tsc'"
hope it helps.

Set NODE_ENV to "production" so libraries like React do not include debugging code

This strongly influences the benchmark results.

add Elm in the benchmark

Please add http://elm-lang.org/ into the benchmark.

Allow a nicer way to run for a single framework

The benchmarks seem to take a really long time to run, it would be nice if there was a way to pass an argument in to the mvn command to signal only running for a single framework. Right now I am just using comments to block the others from running so it's doable would just be a nice to have. Thanks for putting this together by the way cool stuff.

mithril v1.0 alpha bug

When I build and run the benchmark...

1. create 1000 rows (ok, logs run took xxx)
2. create 1000 rows again. (bug, deletes old rows and creates next 1000 rows, but logs "delete took xxx". Then if I select, it deletes all the items. Or if I append, swap, or update, the log keeps saying "delete took xxx" instead of correct action)
3. (alternative). append, update, swap, select all work fine until I create a 2nd time.
4. once I hit clear, things go back to step 1.

i've been trying for a while to fix it, but i can't trace down how doing anything other than .remove() can trigger startMeasure with the name 'delete'.

https://youtu.be/pNpMYPMVduA

npm build failure

Thanks @krausest for the npm install fix with dependencies..
npm install 81.69s user 12.57s system 106% cpu 1:28.52 total

running into a build issue:

$ npm run build

> [email protected] build ~/Downloads/js-framework-benchmark
> node build.js


> [email protected] build-prod ~/Downloads/js-framework-benchmark/angular-v1.5.8
> webpack -p -c webpack.config.js

Hash: 7c41281ac1dc4335b953
Version: webpack 1.13.2
Child
    Hash: 7c41281ac1dc4335b953
    Version: webpack 1.13.2
    Time: 5878ms
      Asset    Size  Chunks             Chunk Names
    main.js  250 kB       0  [emitted]  main
       [0] multi main 40 bytes {0} [built]
       [5] ./src/angularmin.js 75 bytes {0} [built]
       [6] ./webpack.config.js 525 bytes {0} [built]
        + 4 hidden modules

    WARNING in main.js from UglifyJs
    Side effects in initialization of unused variable jQuery [./src/main.es6.js:3,4]
    Condition always true [./~/jquery/dist/jquery.js:10035,0]

> [email protected] build-prod ~/Downloads/js-framework-benchmark/angular-v2.0.0
> npm run precompile && node rollup.config.js --prod


> [email protected] precompile ~/Downloads/js-framework-benchmark/angular-v2.0.0
> ngc

The `this` keyword is equivalent to `undefined` at the top level of an ES module, and has been rewritten
The `this` keyword is equivalent to `undefined` at the top level of an ES module, and has been rewritten
closure compiler optimizing...
closure compiler optimizing complete
closure compiler optimizing...
closure compiler optimizing complete
built angular2

> [email protected] prebuild-prod ~/Downloads/js-framework-benchmark/aurelia-v1.0.0
> npm run clean:dist


> [email protected] clean:dist ~/Downloads/js-framework-benchmark/aurelia-v1.0.0
> rimraf dist


> [email protected] build-prod ~/Downloads/js-framework-benchmark/aurelia-v1.0.0
> cross-env NODE_ENV=production npm run webpack -- --progress --profile


> [email protected] webpack ~/Downloads/js-framework-benchmark/aurelia-v1.0.0
> cross-env ./node_modules/.bin/webpack "--progress" "--profile"

Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration has an unknown property 'htmlLoader'. These properties are valid:
   object { amd?, bail?, cache?, context?, devServer?, devtool?, entry, externals?, loader?, module?, name?, dependencies?, node?, output?, plugins?, profile?, recordsInputPath?, recordsOutputPath?, recordsPath?, resolve?, resolveLoader?, stats?, target?, watch?, watchOptions? }
 - configuration.entry['app'] should be one of these:
   string | [string]
   The entry point for one output file.
 - configuration.entry should be one of these:
   object { <key>: string | [string] } | string | [string]
   The entry point(s) of the compilation.
 - configuration.resolve has an unknown property 'root'. These properties are valid:
   object { modules?, descriptionFiles?, plugins?, mainFields?, aliasFields?, mainFiles?, extensions?, enforceExtension?, moduleExtensions?, enforceModuleExtension?, alias?, symlinks?, unsafeCache?, cachePredicate?, fileSystem?, resolver? }

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/6.2.2/bin/node" "/usr/local/bin/npm" "run" "webpack" "--" "--progress" "--profile"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] webpack: `cross-env ./node_modules/.bin/webpack "--progress" "--profile"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] webpack script 'cross-env ./node_modules/.bin/webpack "--progress" "--profile"'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark-aurelia package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cross-env ./node_modules/.bin/webpack "--progress" "--profile"
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark-aurelia
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark-aurelia
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ~/Downloads/js-framework-benchmark/aurelia-v1.0.0/npm-debug.log

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/6.2.2/bin/node" "/usr/local/bin/npm" "run" "build-prod"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build-prod: `cross-env NODE_ENV=production npm run webpack -- --progress --profile`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build-prod script 'cross-env NODE_ENV=production npm run webpack -- --progress --profile'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark-aurelia package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cross-env NODE_ENV=production npm run webpack -- --progress --profile
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark-aurelia
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark-aurelia
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ~/Downloads/js-framework-benchmark/aurelia-v1.0.0/npm-debug.log
child_process.js:529
    throw err;
    ^

Error: Command failed: npm run build-prod
    at checkExecSyncError (child_process.js:486:13)
    at execSync (child_process.js:526:13)
    at ~/Downloads/js-framework-benchmark/build.js:7:3
    at arrayEach (~/Downloads/js-framework-benchmark/node_modules/lodash/lodash.js:520:11)
    at Function.forEach (~/Downloads/js-framework-benchmark/node_modules/lodash/lodash.js:9202:14)
    at Object.<anonymous> (~/Downloads/js-framework-benchmark/build.js:5:3)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/6.2.2/bin/node" "/usr/local/bin/npm" "run" "build"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: `node build.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script 'node build.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the js-framework-benchmark package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node build.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs js-framework-benchmark
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls js-framework-benchmark
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     ~/Downloads/js-framework-benchmark/npm-debug.log

consider running benchmark with CPU/net throttling

@krausest

Now that the benchmark has been split into keyed/unkeyed, both tables are largely green. They're green in part due to the relatively high specs of the machine on which these benches run. This performance of your Mac is sufficient not to break the the 16ms frame boundary in many cases.

This situation is not great because mobile phones - even modern ones - are much slower...and they account for 50% of web traffic today. Rather than re-scaling the colorization, I propose reducing the max workload (maybe 10k rows -> 5k rows) but running the bench with Chrome's 3x CPU throttling and maybe network throttling at Good 3G that's representative of mobile performance. These changes would help de-greenify the tables and help better assess mobile overhead in real life.

FWIW, in my experience Chrome's throttling can in fact be tweaked to be representative of actual perf i get on my 5 year old, not-so-fast device in 3G. [1]

This should probably wait till after Round 5, which i assume is coming soon.

[1] http://www.gsmarena.com/htc_evo_4g_lte-4665.php