bfrymire/crispy

Unit testing framework built in GML for GameMaker LTS 2022+

crispy's Issues

Make base "class" for the Test constructors

There's a lot of code shared between each Test constructor, create a constructor that will act as a base "class" for each of them.

Add support for Feather

Update the JSDocs and argument names of Crispy to work with GameMaker's new Intellisense feature: Feather

Minor issue with "setUp() and tearDown()" wiki example

Hello.

There is a minor error on page the wiki page setUp() and tearDown(). The following code will not compile:

// Create TestCase
test_example = new TestCase(function() {
	// The _number variable is defined in the setUp() function
	self.assertEqual(_number, 24);
}, "test_example");

// Define setUp() function
test_example.setUp(function() {
	// Here we're defining a variable to use in our test
	_number = 24;
});

// Define tearDown() function
test_example.tearDown(function() {
	// We can call our _number variable in the tearDown() too
	show_debug_message("Your number is: " + string(_number));
});

The error lies in the incorrect argument passing, apparently, the format has changed in one of the versions. The correct code would be something like:

// Create TestCase
test_example = new TestCase("test_example", function() {
	// The _number variable is defined in the setUp() function
	self.assertEqual(_number, 24);
});

Due to the odd logic of GitHub, I cannot make a pull request to suggest change for the wiki, so I create an issue.

Thanks in advance.

Assertion of throw()

Hello.

Do you think functionality like pytest.rises would be useful for crispy?
The idea is that the test case expects a raise of exception and if the specified exception was raised, the assertion returns True. In the case of GML, this could be throw() with certain parameters.

Already now I can implement such functionality with crispy, but it turns out to be quite verbose:

function never_enough(number) {
	enum errors
	{
		toobig,
		toosmall,
	}

	if number > 42
		{
		throw(errors.toobig)
		}
		else
		{
		throw(errors.toosmall)	
		}
}


runner = new TestRunner("runner");
suite = new TestSuite("suite");
runner.addTestSuite(suite);

test_too_small = new TestCase("test_too_small", function() {
	try
	{
		never_enough(34)
	}
	catch(_err)
	{
		assertEqual(_err, errors.toosmall)
	}
});
suite.addTestCase(test_too_small);


test_too_big = new TestCase("test_too_big", function() {
	try
	{
		never_enough(43)
	}
	catch(_err)
	{
		assertEqual(_err, errors.toobig)
	}
});
suite.addTestCase(test_too_big);

runner.run();

Thanks for your attention and your project.

.setName() throws error when no name is supplied to TestCase

According to the old README.md file, the name argument is optional:

TestCase([name], [struct_unpack])

When not supplying a name, undefined will attempt to be passed through .setName(). The function only accepts strings and will throw an error otherwise.

TestCase and TestSuite both have the same function.

---

Error message:

ERROR!!! :: ############################################################################################
ERROR in
action number 1
of Create Event
for object obj_test:


Unable to find a handler for exception TestCase.setName() expected a string, received undefined.
NO CALLSTACK

 at gml_Script_TestCase (line 407) - 		setName(undefined);
############################################################################################
gml_Script_TestCase (line 407)
gml_Object_obj_test_Create_0 (line 5)

TestCase.assertIsUndefined() message type check uses incorrect variable and expression operator

Given the following:

runner = new TestRunner("runner");
suite = new TestSuite("suite");
runner.addTestSuite(suite);
suite.addTestCase(new TestCase("test_assertIsUndefined"), function() {
    assertIsUndefined(undefined, "Expected undefined.");
});

This will throw two separate exception messages within TestCase.assertIsUndefined().

1. Change the variable message to local variable _message.
2. Change the or (||) operator to and (&&) operator.

if !is_string(message) || !is_undefined(_message) {
    crispyThrowExpected(self, "assertEqual", "string", typeof(_message));
}

Don't allow names to contain line breaks

assertNotEqual error thrown when first and second are not the same type

When using assertNotEqual, if first and second are not the same type the test would pass as true. However, when comparing first and second of different types, GameMaker throws an error message saying that they can't be compared and will break the runtime.

Allow assertNotEqual to check if first and second are not equal while not throwing an error message if the data types are different.

Create option to silence duplicate outputs when using multiple asserts on a test

While it's generally best to have a single assertion testing a single function, there are times where it's advantageous having multiple assertions on a test. When multiple assertions are on a test, each assertion creates a log which is then output at the end of all the tests. This can create duplicate outputs for the same test and may be confusing for the user.

Create an option for the user to silence duplicate outputs for a single test in the event there is no error.

`CrispyLog` param `_test_case` type checking not using `type()`

Document parent variable

Allow custom assertion methods

As a developer writing unit tests for my project I want to be able to create custom assertion tests so that I can test for more complex mechanics that won't properly be tested using a single assertion check.

`assertFalse(undefined)` behavior 🤔

Hello.

Crude example first:

runner = new TestRunner("Test runner")
suite = new TestSuite("Test suite")
runner.addTestSuite(suite)

test_assert_false_undefined = new TestCase("Assertion to undefined", function(){
	assertFalse(undefined)
	assertIsUndefined(undefined)
	assertTrue(undefined)
})
suite.addTestCase(test_assert_false_undefined)

runner.run()

First two assertions are true, last one is false. The fact that the first test passes successfully looks like not a best way for me.

I understand that this is standard GML behavior due to its bool and undefined logic. But I think it's actually harmful: the difference between false and undefined is often that something went wrong during the test. In my opinion, this is a potentially very large field for false-positive cases.

I understand that the problem is conceptual and, in part, my proposal goes against the ideas of GML, but, in my opinion, the potential harm from the current behavior is much greater. I agree with any solution: return false in such a situation, return undefined (not sure if this is the best option for assertion expression). Due to the controversial nature of the subject, this issue is an invitation to discussion.

Thank you in advance.

Stop on failure

Allow the tests to abruptly stop running in the event of an error.
The user should be able to set a flag on each level separately if there should be a stop on failure:

• TestCase - if an individual case fails
• TestSuite - if a nested case fails
• TestRunner - if a nested case fails

Required arguments are not enforced

Given the following code:

runner = new TestRunner();
suite = new TestSuite();
runner.addTestSuite(suite);
var _test = new TestCase(function() {
    assertEqual();
}, "_test");
suite.addTestCase(_test);
runner.run();

While running _test, the assertEqual function will assert true. This is happening because when first and second arguments are not provided, they'll be defaulted to undefined, making the assert true.
First and second arguments are required by the jsdocs, but not through code.
Go through each of the functions and make sure arguments that are required throws an error when not supplied.

Add configuration to silence passing tests

The Output Window can become cluttered when several hundred tests are added to a project. This makes it difficult to find output text from other libraries or debug messages.

Add a configuration that allows passing tests to not be printed to the Output Window.

Passing a value other than a string as assertion custom message will cause addition error

Given the following code as a TestCase test:

assertTrue(false, -100);

An error will be thrown after TestRunner is compiling the logs:

ERROR in
action number 1
of  Step Event0
for object obj_test:

DoAdd :: Execution Error
 at gml_Script_anon_CrispyLog_gml_GlobalScript_CrispyLog_1271_CrispyLog_gml_GlobalScript_CrispyLog (line 70) -                                           _msg += "- " + msg;
############################################################################################
gml_Script_anon_CrispyLog_gml_GlobalScript_CrispyLog_1271_CrispyLog_gml_GlobalScript_CrispyLog (line 70)
gml_Script_anon_TestRunner_gml_GlobalScript_TestRunner_4027_TestRunner_gml_GlobalScript_TestRunner (line 193) -                             var _msg = logs[i].get_msg();
gml_Script_anon_TestRunner_gml_GlobalScript_TestRunner_3046_TestRunner_gml_GlobalScript_TestRunner (line 116) -               tear_down();
gml_Object_obj_test_Step_0 (line 5) -               runner.run();

Throw an error when anything other than a string is passed as an assertion custom message.

Move class variable init and setting above static method variables

There's currently a Feather bug where GM2043 will warn you that you're trying to access a variable outside of its scope if the variable is a method variable and defined later in the struct.

To make the constructor code cleaner, when this bug is fixed, move all of the variable initialization to the top of the constructor.

When using `TestRunner.discover()` it's possible to create tests for the same function multiple times

Create a struct that uses the function names as keys.