Add new result types to Test Run

[open-mpi]

Add new result types to Test Run

[ompiteam]

Imported from trac issue 37. Created by jsquyres on 2006-08-26T09:59:26, last modified: 2006-11-29T15:44:30

[ompiteam]

Trac comment by jsquyres on 2006-08-26 10:01:30:

(In [243]) Refs #37

The mtt client now explicitly reports whether a test has timed out (to
differentiate it from failing to meet its "pass" criteria). This is
helpful for sysadmins to adjust the timeout value in their ini files.

The timed_out value is reported in the results. See ticket #37 for
details of the possible (pass, timed_out) tuple possibilities.

[ompiteam]

Trac comment by jsquyres on 2006-08-27 13:30:44:

Thinking about this a bit more, I wonder if we should actually have 4 possible results from an individual test run:

• passed
• failed
• timed out
• skipped

And therefore we should have a single field (bit-mapped?) passed back to the server indicating which of those results the tests fell into.

This would also imply a slightly different methodology both in the ini file and the client -- we should allow a specific "skipped" criteria (just like we allow specifying the "pass" criteria). The default can be an exit status of 77.

[ompiteam]

Trac comment by jsquyres on 2006-08-27 13:35:11:

Also see ticket #34.

[ompiteam]

Trac comment by [email protected] on 2006-08-28 13:04:16:

In the database, I would like to have two fields {{{exit_status}}} and {{{timed_out}}}.

{{{exit_status}}}[[BR]]
smallint which allows for any status code (so we will be
provisioned for any other possible status codes we want to report on in the future)

{{{timed_out}}}[[BR]]
boolean (which can be paired with an {{{exit_status}}})

[ompiteam]

Trac comment by jsquyres on 2006-08-28 15:12:20:

We talked about this a little more -- let's only add ''one'' field, and name it "result". Let's have this set of values:

• 0 = success
• 1 = fail (meaning: the test completed, but it did not meet the "pass" criteria)
• 2 = timed out (meaning: the test did not complete in the assigned time and the MTT killed it)
• 3 = skipped (meaning: the test indicated that it could not run, and therefore it didn't do anything useful)

More specifically -- what if the client returns one of four '''strings''' back to the server, and how the server chooses to store that is up to the server (i.e., the server could convert that to a smallint based on the mapping and only store that smallint in the database for efficiency, and then convert it back to a string whenever querying for or displaying that result on a web page). Doing it this way will:

• make the HTTP results easy for a human to read (particularly for debugging)
• ensure that the client and server never get out of sync in terms of which integer maps to which meaning

[ompiteam]

Trac comment by jsquyres on 2006-09-28 17:39:20:

The client is now reporting a value of 4 for the test_result field when tests timeout (1=pass, 2=fail, 3=skipped, 4=timeout)

[ompiteam]

Trac comment by jsquyres on 2006-09-29 09:11:38:

Reassigning this to Ethan because it's now a server-side problem. :-)