comp1511unsw/dcc

dcc - a C compiler which explains errors to novice programmers

dcc's Issues

Improve/increase output for "illegal array, pointer, or other error" to specify the type of error

Tutors and students alike are struggling to debug memory errors.

dcc's "runtime error - illegal array, pointer or other operation" doesn't really give much information, and this can lead to even tutors being stumped over what the problem is.

I've written some guides/help for tutors on how to debug some common errors (especially with the current COMP1511 ass2), but maybe also adding slightly more information to dcc's output would be more effective.

I've made a slight modification to dcc which provides more output and explanation about three situations: heap-buffer-overflow (aka "you didn't allocate enough memory"), use-after-free (aka "you just freed curr so you can't do curr->next), and double-free.

I've also written a test program to demonstrate each of the situations.

The test program and diff for dcc can be found in my CSE account:

http://www.cse.unsw.edu.au/~andrewb/misc/dcc-asan.diff
http://www.cse.unsw.edu.au/~andrewb/misc/dcc_asan_reporting_tests.c

Here's what the output looks like (additions bolded):

---

andrewb@cse:~/$ dcc-test -o asan_test dcc_asan_reporting_tests.c && ./asan_test
[...]
=================================================================

dcc_asan_reporting_tests.c:46 runtime error - illegal array, pointer or other operation [heap buffer overflow]
Attempted to access past the end of allocated memory. Make sure you have allocated enough memory for the size of your struct/array (rather than size of a pointer).
Execution stopped here in test_heap_bof() in dcc_asan_reporting_tests.c at line 46:
[...]

andrewb@cse:~/$ dcc-test -o asan_test dcc_asan_reporting_tests.c && ./asan_test
[...]
=================================================================

dcc_asan_reporting_tests.c:54 runtime error - illegal array, pointer or other operation [heap use-after-free]
Attempted to access memory that has already been freed.
Execution stopped here in test_heap_uaf() in dcc_asan_reporting_tests.c at line 54:
[...]

andrewb@cse:~/$dcc-test -o asan_test dcc_asan_reporting_tests.c && ./asan_test
[...]
=================================================================

dcc_asan_reporting_tests.c:62 runtime error - illegal array, pointer or other operation [double free]
Attempted to free memory that has already been freed.
Execution stopped here in test_double_free() in dcc_asan_reporting_tests.c at line 62:
[...]

I've tried to keep the amount of "scary technical jargon" to a minimum -- hopefully COMP1511 students will have at least heard of a "buffer overflow" by the time they're writing the sort of code to trigger these bugs; if nothing else, it will make it easier for tutors to quickly identify what's happening in a student's code.

I'm definitely open to better wording for the buffer overflow explanation -- currently everyone in COMP1511 is (indirectly/non-obviously) allocating sizeof(struct blah*) rather than sizeof(struct blah), but I'm sure outside of my COMP1511 bubble there are other reasons for students to run into that sort of bug.

(note that overflows in arrays within malloc'd structs are picked up by something other than asan)

---

I've had a few instances where this information made my life significantly easier when debugging student code during consults; the version of dcc that does this can be accessed at ~andrewb/bin/dcc-test.

Reducing the size of massive output binaries

The constant runtime overhead of ASan is such that binaries produced by dcc are several orders of magnitude greater in size. The result is that students tear through quota, and (e.g.,) cs1511exam work directories have compression ratios of ~0.01 because of the same blob of runtime code.

ASan has experimental (but usable!) support for the runtime image to be loaded by the dynamic linker; this can be fragile with complex code (things like order of initialisation vectors start becoming flaky) but this should not be an issue for us. To enable this, simply set -shared-libasan. We also need -Wl,-rpath,/usr/lib/clang/4.0.1/lib/linux as the rtld doesn't know where to find the runtime shared object by default; this path should be possible to determine from clang.

As a demonstration, I compiled int main() { return 42; } with clang, clang+ASan, clang+sharedASan dcc, dcc+sharedASan, ran the resultant binaries, and sized them.

   text    data     bss     dec     hex filename
   1037     276       4    1317     525 a.clang.elf
1384795   14596 4551920 5951311  5acf4f a.clangasan.elf
   1498     324       4    1826     722 a.clangsharedasan.elf
1391083   15480 4552008 5958571  5aebab a.dcc.elf
   9348    1300      96   10744    29f8 a.dccsharedasan.elf

The elimination of the ~1.6 MB constant overhead makes a staggering difference in executable size. Reducing the executable size so much presents a better case for including more interesting data (e.g., bundled code, etc) instead of lugging around the ASan runtime.

Feature request: better explanation about assert-related compile errors

Compile errors relating to syntactically-incorrect asserts (or surrounding code) are very confusing.

Missing semicolon on the line before

assert

int i = 10
assert(i == 10);

compile error:

test.c:12:5: error: called object type 'int' is not a function or function pointer
    assert(i == 10);
    ^
/usr/include/assert.h:108:3: note: expanded from macro 'assert'
  ((void) sizeof ((expr) ? 1 : 0), __extension__ ({                     \
  ^
1 error generated.

---

"normal" code

int i = 10
printf("hello\n");

test.c:10:15: error: expected ';' at end of declaration
    int i = 10
              ^
              ;
EXPLANATION: Try including a semicolon at the end of line 10 of `test.c`.

---

Missing bracket inside assert

assert

int i = 10;
assert(getNumber(i == 10);

compile error:

test.c:12:5: error: unterminated function-like macro invocation
    assert(getNumber(i == 10);
    ^
/usr/include/assert.h:107:11: note: macro 'assert' defined here
#  define assert(expr)                                                  \
          ^
test.c:23:2: error: expected '}'
}
 ^
test.c:9:34: note: to match this '{'
int main(int argc, char *argv[]) {
                                 ^
2 errors generated.

"normal" code

int i = 10;
printf("%d", getNumber(i == 10);

test.c:12:36: error: expected ')'
    printf("%d", getNumber(i == 10);
                                   ^
test.c:12:11: note: to match this '('
EXPLANATION: Make sure that all opening parentheses `(` are matched with a closing parenthesis `)` in `test.c`.
 In particular, check to see if you are missing a closing parenthesis on line 12 of `test.c`.

stop printing irrelevant variables

when prinint relevant variable dcc should not print variables
which have been declared but not used yet

they print as unitialized which confuses novice programmers

Feature request: create a PPA for dcc (so that it can be updated with `apt-get`)

see e.g. https://itsfoss.com/ppa-guide/

Currently, updating dcc requires redownloading the .deb file. It would be nice if it could be updated via apt instead.

valgrind returns an incorrect value for posix_spawn

dcc could potentially work-around by wrapping posix_spawn

Multiple (different) copies of dcc exist

The canonical location for dcc is /home/cs1911/bin/dcc, but copies exist(ed?) in different locations in CSE.

COMP1511's autotest is broken

COMP1511's autotest uses /home/cs1511/bin/dcc, which is a copy of dcc.

I've moved that to /home/cs1511/bin/dcc.old, and made dcc be a symlink to /home/cs1911/bin/dcc.

1511 autotest seems to work now.

Is there a reason why there are multiple copies of dcc? (maybe from when some of the home directories were inaccessible during ?18s1?)

Should there still be multiple copies of dcc?

Update errors from help50

The help50 matcher code now lives at https://github.com/cs50/help50-server/blob/master/helpers/clang.py -- it would be useful to update our local copy of the errors it depends on.

(It would also be useful to upstream something slightly more robust than "return the first matching error").

simple example produces - signal 11

#include <string.h>

int main(void) {
	return strlen(NULL + 1);
}

dcc with dual sanitizers doesn't detect uninitialised error that dcc --valgrind does

Simplest example:

#include <stdio.h>
int main(void) {
    double d;
    int i;
    i = d;
}

You get a compiler warning from this since it's very obviously using an uninitialised variable, but a student's assignment code did something equivalent which wasn't picked up by the compiler.

The autotests for the assignment just used dcc (dual sanitizers), whereas the marking tests used dcc and dcc --valgrind separately.

I would guess that since dcc --valgrind does "valgrind + UBSan", whereas dcc does "ASan + UBSan" and "valgrind", something in the UBSan-instrumented code is causing more to happen which valgrind then picks up on?

$ dcc -o test_both test.c
test.c:5:9: warning: variable 'd' is uninitialized when used here [-Wuninitialized]
    i = d;
        ^
test.c:3:13: note: initialize the variable 'd' to silence this warning
    double d;
            ^
             = 0.0
dcc explanation: It looks like you're trying to use the variable `d` on line 5 of `test.c`.
 However, on that line, the variable `d` doesn't have a value yet.
 Be sure to assign a value to `d` before trying to access its value.

$ dcc -o test_valgrind test.c --valgrind
(identical warning to above)

$ ./test        (note: no output)

$ ./test_valgrind

Runtime error: uninitialized variable accessed.

Execution stopped in main() in test.c at line 5:

int main(void) {
    double d;
    int i;
--> i = d;
}

Values when execution stopped:

d = <uninitialized value>
i = <uninitialized value>

Better mechanism to communicate/coordinate with dcc users

As far as I know, several courses use dcc ...

• COMP1511 (no, really?)
• COMP1521
• COMP1911
• COMP2521?
• COMP9024?

Moving this repo to an organisation common to those teaching staff would be useful, to coordinate bug reporting etc. across those groups.

Alternatively, some sort of dcc-users mailing list or similar would also be useful.

gdb.error: ptrace: Operation not permitted

I suspect my yama/ptrace_scope settings were reset after a recent system upgrade, but running dcc should be possible with /proc/sys/kernel/yama/ptrace_scope set to 1:

https://www.kernel.org/doc/Documentation/security/Yama.txt

In mode 1, software that has defined application-specific relationships
between a debugging process and its inferior (crash handlers, etc),
prctl(PR_SET_PTRACER, pid, ...) can be used. An inferior can declare which
other process (and its descendants) are allowed to call PTRACE_ATTACH
against it. Only one such declared debugging process can exists for
each inferior at a time. For example, this is used by KDE, Chromium, and
Firefox's crash handlers, and by Wine for allowing only Wine processes
to ptrace each other. If a process wishes to entirely disable these ptrace
restrictions, it can call prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, ...)
so that any otherwise allowed process (even those in external pid namespaces)
may attach.

The sysctl settings (writable only with CAP_SYS_PTRACE) are:

0 - classic ptrace permissions: a process can PTRACE_ATTACH to any other
process running under the same uid, as long as it is dumpable (i.e.
did not transition uids, start privileged, or have called
prctl(PR_SET_DUMPABLE...) already). Similarly, PTRACE_TRACEME is
unchanged.

1 - restricted ptrace: a process must have a predefined relationship
with the inferior it wants to call PTRACE_ATTACH on. By default,
this relationship is that of only its descendants when the above
classic criteria is also met. To change the relationship, an
inferior can call prctl(PR_SET_PTRACER, debugger, ...) to declare
an allowed debugger PID to call PTRACE_ATTACH on the inferior.
Using PTRACE_TRACEME is unchanged.

dcc bails after warning explanation, even if later errors exist

char *f (void)
{
        char xs[4];
        return xs;

This syntax error doesn't compile, of course:

z5017851@williams:~$ clang x.c
x.c:4:9: warning: address of stack memory associated with local variable 'xs' returned [-Wreturn-stack-address]
        return xs;
               ^~
x.c:4:12: error: expected '}'
        return xs;
                  ^
x.c:2:1: note: to match this '{'
{
^
1 warning and 1 error generated.

But dcc doesn't tell us that the error happened.

z5017851@williams:~$ dcc x.c
x.c:4:9: warning: address of stack memory associated with local variable 'xs' returned [-Wreturn-stack-address]
        return xs;
               ^~
dcc explanation: you are trying to return a pointer to the local variable 'xs'.
  You can not do this because xs will not exist after the function returns.
  See more information here: https://comp1511unsw.github.io/dcc/stack_use_after_return.html

I can see the value of doing this -- yes, I should fix that catastrophe before I make progress -- but it's a little surprising if there are errors following. (This surprised me during a lecture.)

including binary on compile produces messages confusing to novice

dcc a.c -o a
dcc a.c a

Second compile (linker) messages are very confusing for novice

SIGPIPE and SIGUSR trapped unecessarily

dcc should only catch SIGPIPE and SIGUSR1 if multiple sanitizers are running.

Feature request: have `dcc --valgrind` complain loudly if valgrind doesn't exist

If dcc --valgrind is used to compile code, but valgrind isn't actually present on the system, no errors are visible to the user (when compiling or running), and the code runs as if it had just been compiled with clang normally.

Example:

On a system without valgrind present:

$ dcc -o address_uninitialised uninitialised.c -fsanitize=address
$ dcc -o valgrind_uninitialised uninitialised.c -fsanitize=valgrind
$ ./address_uninitialised 
*p is -1094795586
$ ./valgrind_uninitialised 
*p is 1392676000

With DCC_DEBUG=1:

$ DCC_DEBUG=1 ./valgrind_uninitialised 
execvp of /usr/bin/valgrind failed*p is 789974176
__real_main returning 0

Running the same code on a system with valgrind present:

$ /valgrind_uninitialised

Runtime error: uninitialized variable accessed.

Execution stopped in main() in uninit.c at line 6:

int main(void) {
    int *p = malloc(sizeof(int));
--> printf("*p is %d\n", *p);
    return 0;
}

Given that valgrind has been explicitly requested, I think it would make more sense to at least print some sort of user-visible warning saying that valgrind could not be run and so the binary is running without any information/checking by valgrind.

stack overflow produces uninformative error messages

dcc signal handler uses the same stack so the SIGSEGV from a stack overflow isn't caught

A separate signal handler stack would allow it to be caught. Although it may be easier to produce an informative message from valgrind?

Tests?

It would be very useful to have tests, to demonstrate that dcc is doing what we expect!

Execution stopping with "errors", doesn't say what the error is

minimal breaking example:

int main(void) {                                                         
                                                               
    char input[256];                                                     
    char output[256];                                                    
                                                                   
    int ch = getchar();                                                  
                                                                   
    int i = 0;                                                           
    while (ch != '\n') {                                                 
        input[i] = ch;                                                   
        i++;                                                             
        ch = getchar();                                                  
    }                                                                    
    strcpy(output, input);                                               
}

Output from autotest:

dcc -Werror remove_repeated_lines.c -o remove_repeated_lines
1511 c_check remove_repeated_lines.c

Test 0 (./remove_repeated_lines) - failed (errors)
Your program produced these errors:

Execution stopped in main() in remove_repeated_lines.c at line 18:

        int array_size = 0;
        while ((array_size < 256) && (scanned_in_value != '\n')) {
            character[array_size] = scanned_in_value;
            array_size = array_size + 1;
            scanned_in_value = getchar();
        }
-->     strcpy(line[row],character);
        int repeat = check(character, line);
        if (repeat == 0) {
            int counter = array_size - 1;

Values when execution stopped:

array_size = 22
character = "Simpson, Homer Simpson", <234 uninitialized values>
line = {<256 uninitialized values> <repeats 256 times>}
repeat = <uninitialized value>
row = 0
scanned_in_value = 10
character[array_size] = <uninitialized value>
line[row] = <256 uninitialized values>

The input for this test was:
Simpson, Homer Simpson
He's the greatest guy in history
 From the
 Town of Springfield
 He's about to hit a chestnut tree.

You can reproduce this test by executing these commands:
dcc -Werror remove_repeated_lines.c -o remove_repeated_lines
echo -e 'Simpson, Homer Simpson\nHe'"'"'s the greatest guy in history\n From the\n Town of Springfield\n He'"'"'s about to hit a chestnut tree.' | ./remove_repeated_lines

The same output happens with using just dcc without autotest.

just-valgrind reports the error as an uninitialised value, just-asan reports an error but not what the error is.

Use `--rtlib=compiler-rt` to resolve missing `__mulodi4` and friends

Clang supports choosing a runtime library, which provides "support operations not natively supported by the underlying hardware (for instance, 128-bit integer multiplications)". There's effectively only two choices, here: GNU's libgcc, and LLVM's compiler-rt. (There's probably something similar in the MS ecosystem.)

dcc currently lets Clang choose the platform default -- on GNU systems, it chooses libgcc, which is missing symbols needed by the sanitizer runtime. When we encounter the missing symbol, we recompile without certain sanitizers, potentially trimming some whinges about terrible student code.

In 3c, I instead explicitly opt into compiler-rt (and, for reasons I don't fully understand, that requires also linking in -lgcc and -lgcc_s for unwinding support), and don't have any issues. dcc could viably implement this, and retain sanitizer coverage.

[feature not a bug?] dcc --valgrind reports uninitialised variable errors when accessing an array that was created-inside-and-returned-from a function

Hard to describe, so example code:

int *copyNumbers(int cards[]) {
    int copy[N_CARDS];

    // have some sort of loop to copy all of the values from the `cards` array that's passed in
    // into the `copy` array that we've just created. 

    return copy;
}

Trying to run the following code:

int *copied = copyNumbers(cards);
int first_number = copied[0];

will give an uninitialised variable access error on the second line.

---

Example screenshot from student who ran into this:

While yes, it is technically correct that the array they're trying to access is uninitialised memory (because the function has returned so the array has gone out of scope and ??? undefined behaviour ???), I would have expected an error message that gave some information about what had happened (e.g. ASAN's stack-use-after-return -- which yeah I don't expect valgrind to be able to give ASAN error messages, so maybe this is more specific to the situation when you're running both dcc --valgrind and "normal" dcc?).

user definition of functions from unistd.h breaks dcc

In dual sanitizer mode global definition of any of these names potentially break the generated code

_exit
close
execvp
getpid
lseek
pipe
read
sleep
unlink
write

In single sanitizer mode these potentially break
_exit
close
execvp
getpid

`dcc` explanation missing: "array type [...] is not assignable"

[was COMP1511UNSW/course_materials#64]

As noted during Tuesday week 9 lecture, dcc is missing an explanation for assigning array type.

Trouble with installation on OSX

Hi,
I got a few questions about installing dcc on OSX.

1. On that gdb wiki page, I don't actually know what I'm supposed to look for...
2. I thought we can't modify /usr/bin anymore after some OSX version. Could it be /usr/local/bin instead?
3. Is that a typo in the URL of installation command? maybe it should be 'dcc' instead of 'dc'?

Need some help here. Thank you!

Potentially-confusing errors

XXXXXXX-Game.c:26:13: error: expected ';' at end of declaration list
  stack node discardPile;
            ^
            ;
EXPLANATION: Try including a semicolon at the end of line 26 of `XXXXXXX-Game.c`.

I suspect this student meant "struct node" not "stack node"; the explanation says to include a semicolon (but there's already a semicolon, it's a parsing problem)

---

XXXXXXX-Game.c:154:9: warning: implicit declaration of function 'fprinf' is invalid in C99 [-Wimplicit-function-declaration]
        fprinf(stderr, "Out of Memory\n");
        ^

Might be a good one to say "try fprintf instead"

---

XXXXXXX-Game.c:233:25: error: expected ')'
    destroyCardList(game->deck);
                        ^
XXXXXXX-Game.c:233:20: note: to match this '('
EXPLANATION: Make sure that all opening parentheses `(` are matched with a closing parenthesis `)` in `XXXXXXX-Game.c`.
 In particular, check to see if you are missing a closing parenthesis on line 233 of `XXXXXXX-Game.c`.

The real problem here is that their function opening statement has a semicolon not a {

// Destroys an existing game.
// Frees all existing memory used in the game including
// allocations for players and cards.
void destroyGame(Game game);
    destroyCardList(game->deck);
    destroyCardList(game->discardPile);
    int i = 0;
    while(i < NUM_PLAYERS){
        destroyCardList(game->playerHand[i]);
        i++;
    }
    destroyTurnList(game->pastTurns);

    free(game);
}

---

XXXXXXX-Game.c:215:46: error: called object type 'Card' (aka 'struct _card *') is not a function or function pointer
    mynewgame->currentColor = topDiscardColor(mynewgame->discardPile->top);
                              ~~~~~~~~~~~~~~~^

Might be a good one to explicitly explain "you're trying to call a function 'topDiscardColor', but you have a variable with that name"

---

XXXXXXX-Game.c:30:23: warning: declaration of 'struct list' will not be visible outside of this function [-Wvisibility]
void addToList(struct list *list, Card toAdd);
                      ^

Again, explaining that it means e.g. "you need to put your struct definitions before your function prototypes"

---

XXXXXXX-Game.c:94:26: warning: format specifies type 'int' but the argument has type 'unsigned long' [-Wformat]
        printf("%d\n\n", sizeof(*s));
                ~~       ^~~~~~~~~~
                %lu
EXPLANATION: Be sure to use the correct format code (e.g., `%d` for integers, `%lf` for floating-point values) in your format string on line 94 of `XXXXXXX-Game.c`.

Maybe worth not complaining about sizeof? shrug

---

XXXXXXX-Game.c:45:9: warning: implicit declaration of function 'err' is invalid in C99 [-Wimplicit-function-declaration]
        err(EXIT_FAILURE, "Could not allocate memory for Game");

err is found in err.h

---

XXXXXXX-Game.c:3:10: error: 'Game.h' file not found with <angled> include; use "quotes" instead
#include <Game.h> 
         ^~~~~~~~
EXPLANATION: Looks like you're trying to `#include` a file (`Game.h`) on line 3 of `XXXXXXX-Game.c` which does not exist.
 Check to make sure you spelled the filename correctly.

Explanation is a bit confusing (clang's error is nicer)

---

XXXXXXX-Game.c:18:2: error: unterminated conditional directive
#ifndef GAME_H

XXXXXXX-Game.c:327:2: error: #endif without #if
#endif // GAME_H
 ^

Having some sort of warning that you really don't want to have #ifdef ANYTHING_H / #endif in your code might be good?

---

XXXXXXX-Game.c:90:12: error: redefinition of 'Card' as different kind of symbol
static int Card topDiscard(Game game);
           ^
./Card.h:11:23: note: previous definition is here
typedef struct _card *Card;
                      ^
XXXXXXX-Game.c:90:16: error: expected ';' after top level declarator
static int Card topDiscard(Game game);
               ^
               ;

---

XXXXXXX-Game.c:289:37: error: incomplete definition of type 'struct _card'
    if(game->discard->cardHead->card->value == D){
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
./Card.h:11:16: note: forward declaration of 'struct _card'
typedef struct _card *Card;
               ^

Some way of telling students that they're violating abstraction might be nice -- or at least what "incomplete definition of type 'struct blah'" means.

If you google / stack-overflow the error, it tells you "you need to make sure you put your struct definition into your .h file so it will work" (but that defeats the purpose of an ADT and so is bad advice)

popen/pclose are in stdio.h but not implemented by dual sanitizers

This produces two output lines:

#include <stdio.h> int main(void) { pclose(popen("echo hello world", "w")); }

Feature request: use a color other than dark blue to print "dcc explanation:"

The dark blue color used when printing "dcc explanation: " is hard to read in the default color schemes used on CSE machines, and students often don't seem to notice that there's actually something written there.

It might be better to change it to cyan or bright blue instead?

Error when compiling on OSX

Sorry to trouble, could you help fix this?

macOS 10.14.5
Apple Clang 10.0.1

Following error messages:

$ dcc test.c
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/bin/dcc/main.py", line 48, in
File "/usr/local/bin/dcc/main.py", line 45, in main
File "/usr/local/bin/dcc/compile.py", line 102, in compile
UnboundLocalError: local variable 'clang_version_float' referenced before assignment

Thank you.

Use --Wshadow

This may be contentious, but I think it makes a lot of sense to add -Wshadow as a compilation flag. Students often cause strange issues that are hard to trace by shadowing variable names. and as far as I can tell there is no good reason for us to allow them to do so. If they're shadowing i or j, they're too deeply nested. If they're shadowing anything else... surely they can name things better, or keep a temporary copy.

Is there any reason a 1511 student (or a 1521/2521 student for that matter) would want to shadow a variable?

Uninitialised stack values are 0x0, not 0xbebebebe

Maybe I'm misremembering the behaviour, but I thought that uninitialised stack variables "defaulted" to 0xbebebebe (i.e. a massive chunk of stack-ish memory was set to 0xbebebebe in the hope that this would catch students' uninitialised stack variables)

However, on CSE now the behaviour is that uninitialised variables in my main function have the value 0 :/

This happens both in 64bit VLAB and 32bit login server (williams).

However, it also happens on my local machine, so maybe I am misremembering the expected behaviour (or maybe both CSE and I have updated LLVM / clang and it's changed how something works)?

explanation missing: "macro errors you get when messing up an assert"

reported by @andrewjbennett on Slack

-o is ignored when -c is specified

If I do clang -c test.c -o a.o, the result is stored at a.o

If I do dcc -c test.c -o a.o, the result is stored at test.o

Feature request: version numbers

Unless I've overlooked something (in which case this should be "feature request: obvious version numbers") there's no easy way to tell what """version""" of dcc you have.

A tutor said that dcc --leakcheck didn't exist; I grepped through my local copy of dcc and it wasn't there. I grepped through CSE's copy of dcc, and it was (it's --leak-check though, so the instructions that said --leakcheck need updating).

I suspect CSE's version is "newer", because it's 1823 lines, compared to my local copy which is a mere 1735 lines. But it would be really nice to be able to explicitly tell this.

Packaging for Debian

To make dcc's existence on CSE somewhat less magical (and somewhat less tied to a dozen symlinks), and to improve portability beyond CSE, a Debian-compatible package would be Nice To Have. Notably, this should include information about dependencies.

(I have the better part of a Debian package set up already, but if someone who actually knows what they're doing would like to take this over, I would have no objections at all.)

POSIX semaphores across processes are unreliable under dcc

A program that uses POSIX semaphores across processes loses mutual exclusion properties (!!!) when compiled with dcc. For example: https://cgi.cse.unsw.edu.au/~cs1521/18s2/tutes/week09/code/chars3.c is a fairly textbook example of using named semaphores to synchronise two processes' access to a shared resource (the TTY) ... and it doesn't successfully do that on CSE.

• It works with Clang.
• Also affects 3c, so potentially an ASan or UBSan bug.
• Also affects my LLVM binary bundle, so possibly a 32-bit bug?
• Also affects gcc-7 on CSE, but not gcc-4.9.
• Also happens when the undefined behaviour in that example is fixed.
• I can't replicate this on different architectures or with different LLVMs

Reported by Angela Finlayson.

add stdbuf-like capabilities to dcc to assist testing student code

bundle C source into binary

dcc shows source file lines for error by assuming source file is present with same pathname as used for compilation. This breaks, for example, if binary run in different directory.

Compressed source could instead be bundled into binary. Perhaps this can be combined with bundling dcc's error handling Python code into binary.

avoid malloc in dcc code after error

dcc's error handling code uses setenv/putenv to set environment variables pass information to Python controlling GDB after an error. It would be more robust if this was done directly without malloc because malloc may trigger another error.

explanation would be helpful fo null where non-null expected errors

e.g.:

``In function ‘execute_command’,
inlined from ‘main’ at cowrie.c:81:9:
cowrie.c:165:13: warning: argument 1 null where non-null expected [-Wnonnull]
printf("%s\n", exepath);
^~~~~~~~~~~~~~~~~~~~~~~
cowrie.c: In function ‘main’:
cowrie.c:165:13: note: in a call to built-in function ‘__builtin_puts’
In function ‘is_executable’,
inlined from ‘execute_command’ at cowrie.c:166:16,
inlined from ‘main’ at cowrie.c:81:9:
cowrie.c:235:9: warning: argument 2 null where non-null expected [-Wnonnull]
faccessat(AT_FDCWD, pathname, X_OK, AT_EACCESS) == 0;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cowrie.c:165:13: note: in a call to built-in function ‘__builtin_puts’
In function ‘is_executable’,
inlined from ‘execute_command’ at cowrie.c:166:16,
inlined from ‘main’ at cowrie.c:81:9:
In file included from cowrie.c:13:
cowrie.c: In function ‘main’:
cowrie.c:165:13: note: in a call to built-in function ‘__builtin_puts’
In function ‘is_executable’,
inlined from ‘execute_command’ at cowrie.c:166:16,
inlined from ‘main’ at cowrie.c:81:9:
/usr/include/unistd.h:304:12: note: in a call to function ‘faccessat’ declared here
extern int faccessat (int __fd, const char *__file, int __type, int __flag)
^~~~~~~~~
cowrie.c:163:13: warning: null destination pointer [-Wformat-truncation=]
snprintf(exepath, strlen(path[i])+strlen(program)+2, "%s/%s", path[i], program);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cowrie.c:165:13: note: in a call to built-in function ‘__builtin_puts’
In function ‘is_executable’,
inlined from ‘execute_command’ at cowrie.c:166:16,
inlined from ‘main’ at cowrie.c:81:9:
/usr/include/unistd.h:304:12: note: in a call to function ‘faccessat’ declared here
extern int faccessat (int __fd, const char *__file, int __type, int __flag)
^~~~~~~~~

Installation Issues: Failed to install it on Ubuntu 18.04

Hi, I just use curl to download the .deb file and install it as the installation part command
curl https://github.com/COMP1511UNSW/dcc/releases/download/1.11/dcc_1.11_all.deb -o /tmp/dcc_1.11_all.deb sudo apt install /tmp/dcc_1.11_all.deb
Then I got an error like below:

Also, I tried to dpkg to install it but I still failed...:( Any tips or something I did wrong?

dcc -fsanitize=valgrind binary fail if python3 not in PATH

no explanation for -Wliteral-conversion warnings

An explanation could be added for the warning from

int main(void) { int i = 6.7; }

Should dcc use `-Werror`?

Proposal:

Use -Werror by default with dcc.

Advantages:

• Forces students to acknowledge (and fix) any compiler warnings, which do generally mean that there's a problem with their code.

Disadvantages:

• Attempting to demonstrate what happens with obviously-uninitialised variables (which give compiler warnings) will no longer work
  • Workaround: add -Wno-error manually when trying to demo something that dcc will otherwise stop you from doing.
• There might be some compiler warnings that students really don't need to care about ever.
  • Workaround: add -Wno-xxx flags to dcc where appropriate.
• There might be some compiler warnings that students only sometimes need to care about?
  • I can't think of any examples for this, can't suggest any workarounds (if there are indeed any needed).
• Code that works "just fine" with e.g. gcc -o blah blah.c (and no other flags) but that doesn't compile with -Werror will fail to compile and thus fail the automarking
  • I would consider this a feature, not a bug -- "your code doesn't work unless it works on CSE".

separate dcc source into separate files

split source into multiple files with build script combining them into (current) monolithic script for deployment

Feature request: make it more obvious when an "illegal array, pointer, or other operation" is caused by out-of-bounds access vs NULL pointer

This error could be because game is NULL, or it could be because the allocation for game wasn't large enough and so you're writing off the end of your allocated memory.

I'm not sure how easy/hard this would be to do, but I suspect it would at least be possible, given that ASan knows they're going off the end of their allocation

Excavate more history

I filter-branch'd scripts/dcc out of the 17s1 and 18s1 COMP1511 course materials repos.
It would be interesting (though not critical) to dig out more history ...

• ~teachadmin/lib/dcc
• ~cs1911 in {13,14,15,16}s1

(I can't readily get at any of these... but I'm p<0.05 sure they exist and plausibly interesting history could be extracted.)

KeyError: 'DCC_BINARY' in difficult to reproduce circumstances per andrewb

Test 1_both_in_out_lines______4 (./cs_paint) - failed (errors)
Your program produced these errors:
Traceback (most recent call last):
File "", line 9, in
File "", line 125, in
File "", line 29, in start_gdb
File "/usr/lib/python3.7/os.py", line 678, in getitem
raise KeyError(key) from None
KeyError: 'DCC_BINARY'
Your program produced these 60 lines of output:
[output etc as usual]

unistd.h use not detected with absolute pathnames

Updated help (and more robust argument parsing)

The help message is somewhat out-of-date --

Usage: dcc [--valgrind|--memory|--no_explanation] <c-files>

-- as dcc supports more flags. But, to mitigate this problem and to improve the somewhat impromptu argument parsing, switching to something like argparse should improve matters.

RecursionError: maximum recursion depth exceeded, [dcc-2.6.4]

#file_1.c
#include "file_2.c"

int main(void) {
    return 0;
}

#file_2.c
#include "file_1.c"

dcc-2.6.2

In file included from ./file_1.c:2:
In file included from ./file_2.c:2:
In file included from ./file_1.c:2:
In file included from ./file_2.c:2:
... # repeats many times
./file_2.c:1:10: error: #include nested too deeply
#include "file_1.c"

In file included from ./file_1.c:2:
In file included from ./file_2.c:2:
In file included from ./file_1.c:2:
In file included from ./file_2.c:2:
... # repeats many times
./file_1.c:4:5: error: redefinition of 'main'
int main(void) {
    ^
./file_1.c:4:5: note: previous definition is here
int main(void) {
    ^

dcc-2.6.4

Traceback (most recent call last):
  File "/web/cs2041/Python-3.7.2-pc.amd64.linux/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/web/cs2041/Python-3.7.2-pc.amd64.linux/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/cs1521/bin/dcc-2.6.4/__main__.py", line 48, in <module>
  File "/home/cs1521/bin/dcc-2.6.4/__main__.py", line 45, in main
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 34, in compile
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 338, in get_options
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 409, in parse_args
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 492, in parse_arg
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 508, in parse_clang_arg
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 521, in process_possible_source_file
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 521, in process_possible_source_file
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 521, in process_possible_source_file
  [Previous line repeated 982 more times]
  File "/home/cs1521/bin/dcc-2.6.4/compile.py", line 517, in process_possible_source_file
  File "/web/cs2041/Python-3.7.2-pc.amd64.linux/lib/python3.7/codecs.py", line 310, in __init__
    IncrementalDecoder.__init__(self, errors)
RecursionError: maximum recursion depth exceeded

The previous error is of course not great and could probably easily be improved. if just cutting down on the spam, and hiding the second error that is directly caused by the fist.

But an actual error is far better than a Traceback.