Investigate TravisCI Flakiness

[material-components]

Investigate TravisCI Flakiness

[traviskaufman]

Here's where I'm at so far with this.

I started by looking at some of our old MDL logs from TravisCI, e.g. https://travis-ci.org/google/material-design-lite/jobs/184278985

Combing through the massive logs led me to stuff like this:

Basically, it looked like the tests were finishing but the browser couldn't communicate with them.

I ran the tests locally in debug mode, and profiled the test execution using the Chrome Devtools timeline. Here's the result of the first three seconds:

Note the one massive function call that seems to run for the entirety of the test suite. This call completely locks up the browser, and in fact delays all of the console logging and tap reporting in karma until the suite has finished running. It could also very well delay any sort of script execution dealing with communicating with karma. This, I believe, may be the reason we're dealing with massive test execution times and disconnection issues using SauceLabs. My hypothesis is supported by the fact that are test infra became more flaky as the number of our tests increased.

When I further expected that function, I found the following to be the offending code: https://github.com/substack/tape/blob/master/lib/results.js#L68-L75

This runs all tests synchronously within the same frame, which basically means the more tests we have, the more karma's test function will degrade. Thus I came to the conclusion that tape may not be able to scale to the size of our codebase. It could be the fact that we're using tape in the browser, but nonetheless this presents a problem for us.

Once I had discovered this, I switched out tape for mocha in the following way:

• Installed mocha, karma-mocha, and chai, which replaced both tape and karma-tap.
• Created a fake-tape.js file that essentially duck-punched mocha into using tape's interface. This way, I could run the tests using mocha but not have to modify any of the tests' source code.
• Fooled the karma-webpack config into thinking that import test from 'tape' was importing tape, when really it was importing fake-tape.

Yes, this is the ugliest hack ever, but it worked!. Here's the timeline output running the same sweet of tests with mocha under the hood. Additionally, rather than the first 3 seconds it captures more of the test execution:

As you can see, there's no longer one single function for running tests but many different function calls spaced apart. The browser felt immediately responsive, and all of the superfluous console logging and erroneous reporter output from karma-tap has disappeared.

Next steps

Next steps are to test this setup on SauceLabs, and ideally on TravisCI using SauceLabs, perhaps via a fake pull request. If it seems to fix the flakiness, we should evaluate what the next steps our in terms of where we take our test infra.

[traviskaufman]

Discussed this offline with the team. Link to chat log from convo: https://gist.github.com/traviskaufman/d47416257b86c1bbf979d7f3eee119d3

Decision was made that we definitely need to test in real browsers, since it's the only way we can verify that our code will work in all browsers we support.

We looked at bootstrap's TravisCI file and test infra for inspiration.

Decision was made to move off of tape, as mocha seems to handle the scale of our tests much better, and has first-class browser support.

Implications of this are as follows:

• Removing dependency on karma
• Rewriting / reshaping tests for mocha, rather than for tape
• Changing the way we write some of our tests, e.g. using before/after hooks rather than setupTest()

Next steps

• Determine what type of mocha interface we want to use. The Qunit-style interface looks most similar to what we already have.
• Investigate different types of setups for running mocha locally. Options include jsdom + jsdom-global, or electron-mocha.
• Investigate what it would take to refactor all tests. Investigate potentially using something like recast.
• Ensure that debugging still works in development, and coverage still works on single runs / CI.

@amsheehan @sgomes should we close this issue and create a follow-up?

[sgomes]

👍 for closing the issue; sounds like we have a much more well-defined set of tasks. Let's create issues for those and add them to our tracker.

Thanks for the great investigative work, @traviskaufman!