How to Run ECMA Test262

[JitCompiler]

How to Run ECMA Test262

[avih]

FWIW, @ccxvii wrote an initial harness, and I extended it a bit to skip tests which hang/fail (with current master - 643d428 - about 25 tests).

The harness is not necessarily complete, and it's possible that some harness functions are missing and causing some failures. But it does manage to run all tests without hanging.

The harness and its current output (on Ubuntu 18.04 LTS) are attached here for reference:
mujs-test262-harness-and-output.zip

To run it:

clone https://github.com/tc39/test262.git && cd test262
git checkout es5-tests
# Save the attached harness as mujs-harness.js

# And actually run it
find test/suite -name '*.js' | mujs mujs-harness.js

Summary:

Total: 11701
Pass: 7275
Failed: 4426

[ccxvii]

The test suite here https://github.com/tc39/test262 tests a different language than MuJS implements. MuJS only implements ES5 but that test suite tests ES2015+.

[LeeYunhang]

@ccxvii Is it possible to checkout ECMA5.1 from tc39/test262 to get test suite ECMA5.1?

[ccxvii]

You can check it out from the 'es5' branch, but you'll need to write a new test harness. The one in the tc32/test262 suite assumes it will be run from inside a browser.

[avih]

I think it'd be nice if this repo had such "test frame". Maybe just a small program which exposes window as the global object and maybe few more bits that the test suits expects to see, and therefore allow it to run?

[avih]

I did continue to improve the harness after I posted it here, but eventually stopped because it didn't seem like upstream intends to integrate it into the repo.

For reference, that's the latest patchset I had locally - https://gist.github.com/avih/d4765c2d39d487737134682dbb08cf70

It was applied on top of mujs dcb3f03 (after 1.0.4 and before 1.0.5).

The first commit is probably conflicting because upstream did add a compile function to the mujs shell - but I don't recall if it has identical API as the first commit here.

The 3 other commits should apply cleanly but I didn't test if they still work (with current mujs sh compile function and/or other changes).

The main difference from the harness I posted earlier is that this patchset adds a shell script to run each test on its own, because some tests leave the js VM dirty enough to affect the later tests.

Beyond the shell script, the js test harness file itself was also enhanced to identify/support more test descriptions/errors/etc.

I still think it would be nice to integrate and maintain a test 262 launcher at the repo, but I won't work on it myself as long as upstream seem uninterested.

[avih]

I did continue to improve the harness after I posted it here, but eventually stopped because it didn't seem like upstream intends to integrate it into the repo.

For reference, that's the latest patchset

Ah, it's actually identical to the PR I opened 2018 - #73 and which is still open...

I just tested it again, and indeed except for the first commit which is no longer required because it was merged upstream (support compile function at mujs shell), the three other commits are still valid, they still run the test suite.

Results with mujs master are pretty similar to what they were at 2018, with 5 less hanging tests which can be removed from the list of crashing/hanging tests. Still ~2K failures out of ~11K tests.