test_ref : can't write a proper grader

[ocaml-sf]

test_ref : can't write a proper grader

[picdc]

Actually, it comes from how the grader works:

1. a toplevel is initialized
2. prepare.ml then prelude.ml are loaded into it
3. the student code is loaded into it as the module "Code"
4. solution.ml is loaded into the module "Solution"
5. test.ml is loaded

As such, solution.ml will execute its side-effects after the graded code, and sets the reference to the correct value.

One naive solution would be to load solution.ml first. While this would work for constant values, it will not work if the reference contains a value that is computed : in that case, you would expect the grader to check the reference result against the solution's result, which would have been overwritten.

For now, the simplest solution I can think of is:

1. If an exercise asks to modify a reference that is declared in prepare or prelude, use a second reference to store the result, which will be used specifically by test.ml (you still need to store into the original reference, otherwise the grader will reject the exercise).
2. Load solution.ml before the student's code.

i.e.

solution.ml

let forty_two_copy = ref 0

let () =
  forty_two := 42;
  forty_two_copy := 42

test.ml

 Section ([Text "Ref: "; Code "forty_two"] ,
           test_ref [%ty: int] forty_two (!Solution.forty_two_copy)) (* /!\ use introspection *)

However, I'm not sure loading solution first is ideal: if you grade with the CLI, you can access the solution from the student code, via Solution.<ident> (it does not work online since it first typechecks the code). One possible solution would be to also typecheck offline.

I'm actually not sure how test_ref can be usable without such hideous tricks.

[AltGr]

It seems test_ref wasn't intended to test student answers.
In the mooc, I found a couple of uses count application of certain functions during user code execution:

1. initialise the reference in prepare.ml (never prelude.ml, I agree with @OCamlPro-Couderc wrote, I can't find a reasonable use for it in that case)
2. (Re-)define the function you want to count uses of to increase the reference on each call
3. Because of what @OCamlPro-Couderc wrote above about loading both the student code and the solution, in test.ml, you need to check the value of the reference after substracting the calls made from solution.ml
4. Don't forget to add, in test.ml, an AST check that a clever student didn't redefine the function to not increase your reference