Regression testing?

[DCNick3]

Regression testing?

[nepx]

I "test" for regressions by booting the following operating systems:

• DOS (floppy drive problems)
• Windows 95 (general system state)
• OS/2 Warp 4.5 (esoteric protected mode operations like call gates)
• Windows NT 4.0 (disk drives)
• Windows XP (VESA, uncommon instructions like bound, page-split traces)
• Mandrake Linux (hardware task switching)
• Windows 7 (MMX, SSE, SSE2, large disk images)
• Windows 10 (PAE, NX, only occasionally, though, since boot times are long)

These operating systems are pretty complete in their usage of the entire x86 instruction set. Now, they won't check for edge-case shift instructions like the ones described in #7 and #8, but they do tell me if some major bug has been introduced.

Another option is to randomly "fuzz" a bunch of instructions with random assignment of operands and inputs, but the process is too unpredictable and non-deterministic for me, and quite a few operations cause undefined behavior or faults, usually in flags (i.e. af for shifts) but occasionally in outputs as well. With all these constraints, you're better off just going with hard-coded inputs and outputs.

For a more rigorous test, I use the QEMU TCG test suite for i386, which can be found here. I use a version set up on this disk image, which includes test output created on a real processor. It exercises a large variety of arithmetic instructions, but is limited due to the fact that it runs entirely in user mode (so nothing crazy like far calls/far returns).

I've always wanted a series of tests for x86 like the ones that NESDev developers use. I'll have a go at writing some; it'll be fun to write some x86 assembly.

I'll create a separate repository for these tests so that other emulators can use them too; testing an x86 CPU is a project in itself.

[nepx]

Unit tests: https://github.com/nepx/verr.