Weekly benchmark report - 2021

[jupyterlab]

Weekly benchmark report - 2021

[fcollonval]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table

Test file	large_code_100_notebook	large_md_100_notebook	longOutput - A single output with 100x100 divs	errorOutputs - 10x100 errors output collected
open
chromium
actual	1620 <- [1643 - 1659 - 1678] -> 2048	1099 <- [1152 - 1167 - 1183] -> 1272	1377 <- [1404 - 1421 - 1437] -> 1626	2436 <- [2519 - 2537 - 2559] -> 2759
expected	1585 <- [1627 - 1640 - 1651] -> 1682	1108 <- [1150 - 1166 - 1182] -> 1543	1357 <- [1400 - 1414 - 1432] -> 1617	2453 <- [2507 - 2532 - 2554] -> 2715
Mean relative change	1.9% ± 0.7%	-0.7% ± 1.1%	0.7% ± 0.8%	0.2% ± 0.5%
switch-from-copy
chromium
actual	70 <- [136 - 165 - 213] -> 1649	51 <- [64 - 74 - 184] -> 344	52 <- [70 - 85 - 413] -> 559	33 <- [54 - 64 - 120] -> 322
expected	337 <- [379 - 400 - 441] -> 1143	232 <- [263 - 272 - 362] -> 473	591 <- [628 - 645 - 820] -> 950	101 <- [130 - 148 - 179] -> 278
Mean relative change	-50.2% ± 5.9%	-55.5% ± 3.7%	-73.4% ± 2.8%	-41.0% ± 4.1%
switch-to-copy
chromium
actual	109 <- [138 - 158 - 170] -> 276	49 <- [61 - 66 - 76] -> 173	51 <- [67 - 77 - 86] -> 191	35 <- [52 - 59 - 71] -> 174
expected	325 <- [366 - 382 - 401] -> 474	230 <- [258 - 265 - 277] -> 331	589 <- [619 - 633 - 644] -> 722	98 <- [122 - 138 - 149] -> 262
Mean relative change	-59.2% ± 0.8%	-73.0% ± 0.9%	-87.7% ± 0.3%	-51.5% ± 2.2%
switch-from-txt
chromium
actual	53 <- [109 - 121 - 148] -> 286	49 <- [59 - 64 - 70] -> 166	52 <- [62 - 73 - 81] -> 162	34 <- [55 - 61 - 73] -> 155
expected	93 <- [122 - 129 - 138] -> 222	84 <- [107 - 113 - 120] -> 160	98 <- [121 - 130 - 144] -> 269	50 <- [74 - 80 - 93] -> 187
Mean relative change	-2.8% ± 2.9%	-40.6% ± 1.7%	-44.4% ± 1.6%	-22.3% ± 3.3%
switch-to-txt
chromium
actual	37 <- [49 - 63 - 68] -> 221	35 <- [46 - 56 - 65] -> 131	49 <- [70 - 83 - 98] -> 205	23 <- [41 - 49 - 56] -> 74
expected	288 <- [312 - 331 - 343] -> 470	192 <- [219 - 227 - 238] -> 299	548 <- [582 - 598 - 692] -> 864	74 <- [87 - 104 - 112] -> 174
Mean relative change	-79.6% ± 1.0%	-75.1% ± 0.7%	-85.7% ± 0.6%	-53.3% ± 1.5%
close
chromium
actual	441 <- [512 - 553 - 567] -> 713	353 <- [377 - 387 - 400] -> 471	648 <- [681 - 693 - 707] -> 763	186 <- [193 - 198 - 205] -> 259
expected	270 <- [311 - 374 - 397] -> 436	233 <- [248 - 259 - 266] -> 338	340 <- [355 - 365 - 374] -> 434	150 <- [161 - 166 - 175] -> 273
Mean relative change	51.0% ± 4.9%	49.3% ± 2.7%	89.3% ± 2.1%	16.9% ± 3.5%

Changes are computed with expected as reference.

Expected = 16a81be
Actual = a783add

Go to action log
Changelog covered

[fcollonval]

The major gain in switching time (and the increase in close time) is due to: jupyterlab/jupyterlab#11074

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table

Test file	large_code_100_notebook	large_md_100_notebook	longOutput - A single output with 100x100 divs	errorOutputs - 10x100 errors output collected
open
chromium
actual	1694 <- [1786 - 1814 - 1847] -> 2186	1192 <- [1259 - 1305 - 1354] -> 1530	1575 <- [1626 - 1652 - 1676] -> 1934	1880 <- [1927 - 1960 - 2001] -> 2390
expected	1732 <- [1787 - 1812 - 1838] -> 2196	1208 <- [1279 - 1337 - 1387] -> 1675	1577 <- [1635 - 1654 - 1670] -> 1800	1891 <- [1947 - 1977 - 2006] -> 2409
Mean relative change	0.4% ± 1.3%	-2.3% ± 1.5%	0.1% ± 0.8%	-0.7% ± 1.3%
switch-from-copy
chromium
actual	102 <- [162 - 196 - 241] -> 1920	54 <- [80 - 150 - 217] -> 436	60 <- [82 - 91 - 192] -> 688	41 <- [67 - 102 - 156] -> 365
expected	131 <- [161 - 198 - 242] -> 1885	55 <- [89 - 142 - 212] -> 391	57 <- [83 - 95 - 199] -> 628	43 <- [65 - 98 - 156] -> 275
Mean relative change	1.1% ± 10.5%	1.3% ± 9.1%	5.3% ± 15.9%	2.0% ± 7.9%
switch-to-copy
chromium
actual	81 <- [160 - 188 - 203] -> 257	54 <- [73 - 92 - 146] -> 225	59 <- [79 - 87 - 97] -> 232	27 <- [61 - 70 - 103] -> 240
expected	120 <- [153 - 179 - 201] -> 278	55 <- [75 - 111 - 147] -> 225	60 <- [80 - 87 - 99] -> 254	34 <- [62 - 68 - 105] -> 201
Mean relative change	2.9% ± 2.6%	-2.4% ± 6.0%	-0.9% ± 4.1%	-1.1% ± 6.7%
switch-from-txt
chromium
actual	124 <- [168 - 177 - 185] -> 306	50 <- [69 - 76 - 97] -> 200	60 <- [79 - 86 - 93] -> 206	36 <- [63 - 72 - 100] -> 180
expected	85 <- [167 - 174 - 181] -> 358	53 <- [68 - 76 - 94] -> 204	58 <- [77 - 84 - 94] -> 178	42 <- [60 - 70 - 102] -> 204
Mean relative change	1.7% ± 1.7%	0.2% ± 5.7%	0.8% ± 3.2%	1.4% ± 6.0%
switch-to-txt
chromium
actual	46 <- [59 - 65 - 74] -> 235	41 <- [62 - 70 - 81] -> 205	55 <- [75 - 84 - 108] -> 207	22 <- [32 - 40 - 51] -> 199
expected	47 <- [57 - 64 - 75] -> 282	41 <- [64 - 70 - 77] -> 176	55 <- [75 - 83 - 106] -> 258	24 <- [33 - 42 - 53] -> 189
Mean relative change	1.1% ± 7.8%	2.4% ± 5.3%	0.2% ± 5.7%	-0.5% ± 7.8%
close
chromium
actual	522 <- [637 - 656 - 680] -> 725	419 <- [437 - 447 - 466] -> 538	757 <- [786 - 803 - 818] -> 893	206 <- [233 - 243 - 271] -> 370
expected	525 <- [629 - 659 - 678] -> 736	416 <- [441 - 456 - 475] -> 584	760 <- [785 - 796 - 813] -> 883	210 <- [234 - 243 - 284] -> 339
Mean relative change	1.4% ± 2.1%	-1.7% ± 1.7%	0.3% ± 0.9%	-0.2% ± 3.7%

Changes are computed with expected as reference.

Expected = b80475fbd409385eae2017fbb82c72c638fd6959 | Actual = c7d67b08fb904b104a077ad272096ebd326823c6
Go to action log
Changelog covered

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table

Test file	large_code_100_notebook	large_md_100_notebook	longOutput - A single output with 100x100 divs	errorOutputs - 10x100 errors output collected
open
chromium
actual	1912 <- [1985 - 2014 - 2109] -> 3289	1254 <- [1379 - 1424 - 1480] -> 1788	1747 <- [1803 - 1839 - 1885] -> 2295	2019 <- [2093 - 2136 - 2198] -> 2786
expected	1865 <- [1933 - 1948 - 1968] -> 2159	1267 <- [1343 - 1411 - 1494] -> 1900	1731 <- [1770 - 1798 - 1826] -> 2093	2041 <- [2101 - 2137 - 2179] -> 2607
Mean relative change	5.7% ± 1.7%	-0.1% ± 2.0%	2.2% ± 1.2%	0.3% ± 1.7%
switch-from-copy
chromium
actual	139 <- [177 - 213 - 241] -> 1781	61 <- [130 - 240 - 327] -> 583	59 <- [80 - 93 - 202] -> 675	33 <- [64 - 98 - 168] -> 323
expected	85 <- [160 - 192 - 218] -> 1727	54 <- [129 - 165 - 224] -> 425	68 <- [82 - 94 - 201] -> 722	42 <- [66 - 102 - 164] -> 359
Mean relative change	13.7% ± 13.7%	31.3% ± 11.0%	0.1% ± 14.9%	0.7% ± 8.7%
switch-to-copy
chromium
actual	96 <- [164 - 192 - 221] -> 298	57 <- [106 - 139 - 246] -> 365	63 <- [79 - 88 - 98] -> 258	40 <- [64 - 74 - 113] -> 242
expected	128 <- [156 - 178 - 201] -> 360	55 <- [100 - 136 - 164] -> 243	64 <- [78 - 86 - 96] -> 222	37 <- [64 - 72 - 122] -> 210
Mean relative change	6.4% ± 2.9%	21.6% ± 7.9%	2.7% ± 5.0%	0.4% ± 7.3%
switch-from-txt
chromium
actual	86 <- [169 - 179 - 186] -> 239	51 <- [76 - 93 - 130] -> 234	61 <- [77 - 87 - 94] -> 224	32 <- [62 - 80 - 121] -> 220
expected	97 <- [163 - 170 - 179] -> 222	56 <- [72 - 81 - 118] -> 208	65 <- [77 - 86 - 96] -> 197	36 <- [67 - 82 - 121] -> 201
Mean relative change	4.6% ± 1.7%	8.4% ± 6.6%	-2.1% ± 3.7%	-2.0% ± 6.1%
switch-to-txt
chromium
actual	46 <- [61 - 73 - 128] -> 324	40 <- [64 - 71 - 103] -> 248	53 <- [72 - 82 - 113] -> 248	23 <- [34 - 40 - 52] -> 246
expected	44 <- [56 - 68 - 120] -> 200	40 <- [63 - 70 - 107] -> 212	51 <- [73 - 82 - 111] -> 265	25 <- [34 - 40 - 52] -> 186
Mean relative change	12.8% ± 8.2%	2.1% ± 6.7%	2.9% ± 6.4%	-2.9% ± 8.7%
close
chromium
actual	593 <- [664 - 717 - 734] -> 934	473 <- [514 - 535 - 556] -> 623	769 <- [819 - 847 - 873] -> 960	220 <- [235 - 247 - 267] -> 364
expected	553 <- [669 - 693 - 721] -> 772	464 <- [498 - 515 - 529] -> 687	801 <- [831 - 850 - 867] -> 972	227 <- [239 - 247 - 265] -> 385
Mean relative change	2.6% ± 2.1%	3.6% ± 1.7%	-1.0% ± 1.2%	-1.8% ± 3.5%

Changes are computed with expected as reference.

Expected = 9addcd3a48915716d31ccb57fa7270ef54f48328 | Actual = 59c1ee11b0443c407c155b8a2cee72728221e12c
Go to action log
Changelog covered

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table

Test file	large_code_100_notebook	large_md_100_notebook	longOutput - A single output with 100x100 divs	errorOutputs - 10x100 errors output collected
open
chromium
actual	1720 <- [1782 - 1812 - 1853] -> 2210	1192 <- [1279 - 1328 - 1394] -> 1847	1621 <- [1669 - 1691 - 1717] -> 1882	1917 <- [2004 - 2052 - 2094] -> 2590
expected	1719 <- [1784 - 1822 - 1850] -> 2131	1222 <- [1297 - 1345 - 1403] -> 1759	1606 <- [1659 - 1688 - 1722] -> 1899	1872 <- [1979 - 2021 - 2067] -> 2536
Mean relative change	0.1% ± 1.1%	-1.5% ± 2.2%	0.6% ± 0.9%	1.3% ± 1.5%
switch-from-copy
chromium
actual	121 <- [164 - 200 - 239] -> 2069	60 <- [104 - 201 - 306] -> 486	60 <- [81 - 96 - 206] -> 666	36 <- [65 - 92 - 146] -> 350
expected	126 <- [166 - 201 - 231] -> 1944	59 <- [85 - 207 - 307] -> 506	61 <- [83 - 98 - 205] -> 703	42 <- [68 - 105 - 155] -> 273
Mean relative change	2.2% ± 10.7%	0.6% ± 9.8%	1.3% ± 15.4%	-2.4% ± 7.8%
switch-to-copy
chromium
actual	123 <- [157 - 189 - 205] -> 312	58 <- [80 - 120 - 157] -> 321	58 <- [81 - 90 - 102] -> 190	31 <- [60 - 69 - 102] -> 217
expected	99 <- [157 - 187 - 206] -> 256	56 <- [84 - 124 - 170] -> 481	60 <- [80 - 90 - 102] -> 261	33 <- [60 - 71 - 109] -> 231
Mean relative change	0.4% ± 2.6%	-4.0% ± 7.6%	-1.2% ± 4.0%	-4.9% ± 6.9%
switch-from-txt
chromium
actual	100 <- [161 - 170 - 178] -> 241	51 <- [72 - 82 - 114] -> 271	62 <- [79 - 85 - 96] -> 196	30 <- [61 - 72 - 106] -> 227
expected	102 <- [161 - 169 - 178] -> 254	54 <- [73 - 84 - 114] -> 228	59 <- [79 - 86 - 95] -> 179	30 <- [60 - 73 - 103] -> 192
Mean relative change	-0.0% ± 1.8%	0.1% ± 5.9%	-0.1% ± 3.1%	2.4% ± 6.5%
switch-to-txt
chromium
actual	48 <- [57 - 64 - 77] -> 277	41 <- [63 - 70 - 80] -> 281	53 <- [76 - 84 - 98] -> 231	24 <- [33 - 41 - 51] -> 216
expected	44 <- [57 - 64 - 77] -> 306	43 <- [62 - 69 - 80] -> 261	55 <- [74 - 83 - 107] -> 281	24 <- [33 - 42 - 54] -> 211
Mean relative change	2.7% ± 8.0%	0.4% ± 6.0%	-1.7% ± 5.6%	-5.9% ± 7.3%
close
chromium
actual	521 <- [643 - 678 - 696] -> 746	422 <- [454 - 471 - 499] -> 770	777 <- [809 - 820 - 837] -> 914	201 <- [227 - 240 - 260] -> 355
expected	507 <- [633 - 662 - 686] -> 730	418 <- [455 - 472 - 490] -> 550	762 <- [790 - 806 - 830] -> 914	209 <- [232 - 244 - 272] -> 353
Mean relative change	2.3% ± 2.3%	0.8% ± 2.1%	1.5% ± 1.0%	-2.9% ± 3.4%

Changes are computed with expected as reference.

Expected = a0a911276a4642aea0fb1546dc25f4f1224e5fe2 | Actual = 9ad40ceed4f74470f2ea1d99f12c883f8d460a16
Go to action log
Changelog covered

[github-actions]

Benchmark report

The execution time (in milliseconds) are grouped by test file, test type and browser.
For each case, the following values are computed: min <- [1st quartile - median - 3rd quartile] -> max.

The mean relative comparison is computed with 95% confidence.

Results table

Test file	large_code_100_notebook	large_md_100_notebook	longOutput - A single output with 100x100 divs	errorOutputs - 10x100 errors output collected
open
chromium
actual	1489 <- [1536 - 1558 - 1576] -> 2354	1021 <- [1110 - 1170 - 1205] -> 1415	1382 <- [1436 - 1452 - 1469] -> 1619	1654 <- [1708 - 1727 - 1753] -> 2094
expected	1511 <- [1547 - 1567 - 1586] -> 1751	1072 <- [1129 - 1185 - 1218] -> 1297	1389 <- [1440 - 1454 - 1469] -> 1516	1625 <- [1694 - 1719 - 1743] -> 2134
Mean relative change	-0.1% ± 1.2%	-1.1% ± 1.5%	0.0% ± 0.5%	1.0% ± 1.2%
switch-from-copy
chromium
actual	112 <- [145 - 181 - 226] -> 1678	53 <- [68 - 89 - 335] -> 503	54 <- [70 - 84 - 299] -> 565	34 <- [55 - 65 - 119] -> 250
expected	114 <- [149 - 185 - 222] -> 385	51 <- [66 - 81 - 292] -> 464	56 <- [72 - 83 - 415] -> 590	34 <- [54 - 62 - 119] -> 260
Mean relative change	11.1% ± 10.5%	5.7% ± 13.6%	-0.1% ± 14.8%	3.5% ± 9.7%
switch-to-copy
chromium
actual	112 <- [146 - 171 - 189] -> 262	48 <- [67 - 76 - 105] -> 262	51 <- [68 - 79 - 88] -> 224	29 <- [50 - 59 - 76] -> 190
expected	118 <- [144 - 175 - 189] -> 216	47 <- [63 - 70 - 82] -> 225	54 <- [67 - 75 - 83] -> 224	33 <- [52 - 59 - 70] -> 160
Mean relative change	0.9% ± 2.6%	18.1% ± 8.8%	5.5% ± 3.8%	0.7% ± 6.1%
switch-from-txt
chromium
actual	53 <- [116 - 147 - 162] -> 236	52 <- [63 - 69 - 83] -> 181	54 <- [67 - 75 - 84] -> 199	30 <- [55 - 63 - 75] -> 176
expected	65 <- [114 - 142 - 158] -> 201	53 <- [62 - 67 - 75] -> 144	54 <- [66 - 75 - 83] -> 186	30 <- [55 - 62 - 75] -> 163
Mean relative change	3.3% ± 3.3%	7.4% ± 4.0%	2.1% ± 3.4%	1.3% ± 5.5%
switch-to-txt
chromium
actual	40 <- [53 - 62 - 70] -> 233	36 <- [52 - 65 - 72] -> 142	49 <- [72 - 84 - 103] -> 220	23 <- [38 - 46 - 53] -> 93
expected	41 <- [51 - 62 - 71] -> 197	39 <- [53 - 66 - 73] -> 115	52 <- [73 - 85 - 102] -> 184	23 <- [40 - 49 - 56] -> 106
Mean relative change	3.2% ± 7.0%	-0.8% ± 3.7%	1.7% ± 5.3%	-6.6% ± 3.7%
close
chromium
actual	463 <- [558 - 577 - 599] -> 677	374 <- [395 - 410 - 421] -> 570	670 <- [687 - 695 - 710] -> 772	187 <- [197 - 204 - 214] -> 285
expected	469 <- [572 - 587 - 601] -> 658	379 <- [401 - 410 - 419] -> 442	670 <- [680 - 691 - 698] -> 746	184 <- [193 - 197 - 208] -> 281
Mean relative change	-1.2% ± 1.9%	0.8% ± 1.4%	1.3% ± 0.7%	2.4% ± 3.1%

Changes are computed with expected as reference.

Expected = 26335b4d936925a7cbbdf31a86fa8bd4495efa83 | Actual = 602b05399b0ca762613c8f560a49b15abdefee39
Go to action log
Changelog covered