jnguyen1098/seer

Testing framework and auto-grader for C programs

Seer

Generalized test harness for small-to-medium-sized C projects.

The frontend of Seer is a simple makefile interface that can be brought up using the command make or make help.

In order to make the valgrind report more accurate and reduce the potential margin of error, nothing in this test harness is heap/dynamically-allocated. TestResults contain no pointers and are passed by value. Delegating memory management (outside of calling close()) makes Seer considerably more reliable. Surprisingly, no pointers were used.

Features

• Seamless test case creation -- all cases are routed through a single function with a switch statement that takes in a test number.

• Crash handling -- Seer catches as many crash signals as possible and records them into the test results

Typical Workflow

1. Edit config.h and add any needed #include statements

2. (Optional) Format test report using make plan

3. Create test cases and fixtures using make edit

4. Move .c and .h files to the student folder

5. Use make grade or make valgrade to grade a submission

FAQ

• Why not use signal/sigaction/sigsetaction?

It's not really possible to tell which parts of memory are safe within a given process during a SIGSEGV (segmentation fault) or SIGABRT (usually fired by free() during double-free or corruption). Using the current process after such an event is essentially UB.

• Why not use setjmp/longjmp?

This was actually my original intent, but this only reduces the inherent risk. If the offending event somehow hits the stack, then this method is just as useless.

• Why not use pthreads?

The default (SIG_DFL) action for a segmentation fault is to terminate the current process, not the current thread. The only thing that threads don't have in common is the call stack -- everything else is shared. This means one thread could potentially ruin the memory being touched by other threads. As such, the only solution I can use is to fork() entire processes.

• What makes fork() so much safer? Isn't it a waste of memory?

Not entirely. Although child processes inherit an entire copy of the parent's address space, it is marked as "copy on write". This means if a child process is created and immediately crashes, the only memory copied to them is the memory immediately used/abused. Although spawning a process is inherently more taxing, period, I still think safety matters more here. This holds true especially in the case of more complex assignments.

seer's Issues

note to clean

implement something to guarantee that the previous student's projects files are cleaned (the .o/binaries)

better comment printing

It might be desensitizing to see a line of Passed! (No comment). Perhaps it would be better to either

• only show the comment if there is one (and don't print (No comment)), or

• change the comment colour to maybe an off-colour to the original

Fix stack size stuff

I feel like 128 is a bit too short of a string, but I find that a struct size of (128+128+4) * [10000 tests] was too much for the stack. I since changed it to (128+128+4) * [1000 tests], but I should fix this or include a flag to increase the size of the stack.

mention compiler flags

Currently the makefile uses -std=c99 and -Wall because I made this harness specifically for CIS*1300. However, I imagine other courses/projects will use different flags. It would be preferable to have this in the documentation in order to make sure people know where to config

Warn user about limits

Warn them in the documentation about the max tests, max str_size, etc. (esp in the case of strcpy/sprintf)

Warning feature?

Perhaps a yellow warning message for passes ? idk

Create main() ignore directives

The only main() function that should run exists in instructor/run_main.c. Sometimes files in student/ will have a main() (usually when the program being graded has a user interface testing part).

Unfortunately, it is not possible to compile multiple main()s into a binary, especially in the case of Seer.

The solution I've barely tested is:

1. In the CFLAGS, use -D"main(...)"="old_main(__VA_ARGS__)". This is the equivalent of doing this to every file:

#define main(...) old_main(__VA_ARGS__)

1. In run_main.c, use the directive:

#ifdef main
#undef main
#endif

Which removes the main(...) definition just in that file. This means in every file but run_main.c, the main() function is nullified. This—in theory—guarantees that the only main() that makes it to compilation is the one in run_main.c.

Adding the -D option to every compiled function is very sloppy, so I will look into this. In every C program I can think of, the main identifier/symbol cannot be used. This is because functions and variables share the same namespace; as every proper C program has a main(), this means that you can never create a variable named main either. Turns out this is wrong. You are totally allowed to use a local main variable provided you don't use it as a function:

#include <stdio.h>

int main(void)
{
    int main = 4;
    printf("variable %d\n", main);
    return 0;
}

This shouldn't change anything though.

Streamline grading

perhaps more bash scripting where you take a moodle submission .zip and it iterates through it all... but then again that will depend on the assignment. more analysis needed.

`make run` normal run

Rather than link/compile with Seer, there should also be an option to just run the student's program (for more complex assignments where user acceptance testing from the student's main() is necessary).