lennym/choma

Random execution order for mocha suites

choma

Random ordering for mocha

Why?

It is very easy to accidentally create a fragile test suite by leaving lingering state after a test case, that is then inadvertently used as a precondition for a subsequent test. This can then cause problems trying to run single tests or test suites in isolation.

By executing files in a random order on each execution of a test suite the risk of accidental introduction of state is eliminated, or at least reduced, since any dependency on leftover state will result in test failures.

Usage

Pass choma as a require option to mocha, either by appending to your test command:

> mocha ./tests/ --require choma

or by adding a line to mocha.opts

--require choma

Seeds

The seed used to generate the random order is output to the console when running your test suite.

If you would like to re-use a seed to re-test a particular execution order, you can set an environment variable of CHOMA_SEED.

> CHOMA_SEED=myseed mocha ./tests/ --require choma

choma's Issues

Add a way to read the seed in the test

If you use a test suite with, for instance, faker, you might want to use the same seed for choma and faker to be able to reproduce a test run. You should not only be able to set the seed by an environment variable, but also to read that seed to pass it to faker.seed(...).

CHOMA_SEED is Not Consistent Across Different Machines

On my CI/CD:

Then, locally:

I see the choma seed printed matches the one in my CI/CD system, but the tests don't run in the same order. I expected the ordering to be the same across different machines.

But!

I noticed when I created a small PoC to test this, that things consistently ran as expected:

CI/CD:

Locally:

The files containing tests are exactly the same locally & in the CI/CD system. They're run on different OSes (Debian Stretch vs Ubuntu 16.04) but otherwise things are the same (node version and all deps)

I don't know the choma code well. What might cause this difference between a simple few test files vs a complex bunch of tests?