jacquard-autograder/jacquard

Platform-independent Java autograder with Gradescope support

Jacquard

Jacquard is a Java autograder developed at Northeastern University with Gradescope integration. It has been used at multiple universities.

Features

• Static analysis with Checkstyle and PMD.
• Syntactic analysis using JavaParser
• Test coverage and cyclomatic complexity measurement with JaCoCo.
• Unit testing with JUnit 5, including:
  • running staff tests against student code
  • running student tests against
    • student code
    • intentionally buggy staff-written code
    • correct staff-written code

Examples

The best way to learn Jacquard is to review the examples, which include video explanations.

You should start with Jacquard Example 0, which shows how to grade based on Checkstyle, PMD, and JUnit 5 tests.

Jacquard Example 1 adds syntactic analysis based on the parse tree, such as whether students use certain language features and methods.

Jacquard Example 2 demonstrates:

• Changing the default configuration values:
  • visibility level of results
  • timeouts
  • language level
• Running Checkstyle and PMD on multiple student files
• Measuring code coverage of student tests
• Running unit tests on student code
• Fine-grained control of visibility
• Cross-testing, i.e., running student tests against
  • student code
  • correct code
  • buggy code

Example 2 does not depend on Example 1, which may be skipped if you are not interested in syntactic analysis.

Further Information

• Javadoc
• FAQ

There are low-volume Google groups jacquard-announce and jacquard-discuss.

Information for Contributors

I welcome contributors, especially ones interested in integrating Jacquard with learning management systems (LMSs) and GitHub Classroom. See tips for contributors.

I also welcome reports suggestions of how to improve Jacquard, the documentation, or examples.

Credits

Jacquard was influenced by Tim Kutcher's JGrade and includes some of its Visibility and GradedTest code.

jacquard's Issues

Add test for PMD/Checkstyle suppression

Bug: collapsing tests do not giving the correct visibility

There appears to be a bug in the new "collapsing" tests feature.

When all of the tests with identical names are set to "VISIBLE" the visibility will be set to the default visibility.

Deploy javadoc to github pages

Enable users to view javadoc on github pages.

Make Java 11 version

This was requested at the SIGCSE workshop.

Enable combining targets for scoring

If I run a single CheckStyleGrader over multiple targets in a single call to Grader.grade(List<Target>), each target can earn (or lose) the maximum score.

For example, consider:

List<Target> targets = Target.fromPathStrings("src/main/java/student/",
        "Class1.java", "Class2.java");
CheckstyleGrader checkstyleGrader = new CheckstyleGrader(
        "config/checkstyle-rules.xml",
        1.0,
        10.0);
List<Result> results = checkstyleGrader.grade(targets);

Because there are two targets, Checkstyle is worth 20 points, not 10. This violates the
Principle of Least Astonishment.

The same thing happens with other graders.

There are workarounds, such as constructing multiple graders or postprocessing the returned List<Result>, but there should be a better way.

This applies to jacquard-examples/hw1.

Enable multi-part package names

Currently, the package for submitted files must not be nested (e.g., student, not edu.neu.cs2500.hw1. Subpackages should be allowed.

[feature request] Automatic deployment from GitHub

Hello @espertus !

Thank you for your work on jacquard. I am evaluating it as a potential replacement for our hacky autograder code.

One small improvement I've been able to make on our own autograder setup is automatic deployment from GitHub. This allows the upload of a single zip file which automatically pulls new changes from main whenever it is deployed.

I have sent an invite to a private repo which contains the code. The repo contains a GitHub CLI extension which automatically configures repo-level deploy keys (only works on unix-based OSes at the moment, there is a windows bug) and creates the autograder.zip.

I'd love to discuss jacquard more ([email protected]). Your jacquard-discuss group appears to be private or otherwise inaccessible.

Incorrect description of cross-test result

When I ran jacquard-example2, the output included:

    {
        "output": "Test concatWorksForLength1() SUCCEEDED by not falsely reporting a bug in the hidden correct implementation\n",
        "score": 20,
        "visibility": "after_due_date",
        "max_score": 20,
        "name": "Tests of student.concat()"
    },

The output should not have referred to the student implementation as "the hidden correct implementation".