thepyrotechnic/pygrader

pygrader's Issues

Allow caching of last assignment, to allow fast re-grading

Caching should be implemented:

• with a warning if the assignment is not yet past due that the grader cannot verify whether a new submission has been added
• with the ability to toggle automatic caching
  • and an option for auto-cache only if past deadline
• with a question to ask to restore cache after an assignment has been chosen

Notify user that blank manual comment will not delete existing

Also add ability to delete comment

Timeout does not work properly on Linux

The process will not be killed properly, leaving the grader blocked.

PyCanasGrader class should contain the chosen course_id and assignment_id

This will greatly simplify most of the code, and make many functions much cleaner.

Update program flow to be more legible and usable

When running, the grader will present the user with a list of students and the ability to either select individual students to grade/update, or run the grader on all students. They will then be able to upload all grades at once, or on an individual basis. They will have the ability to manually edit individual grades.

Add ability to 'record' intitial setup, and then optionally skip 'setup' next time.

When selecting job type, class, and assignment, record this info to a file, used to optionally skip it next time. Should be readable (TOML or JSON)

Batch submissions do not work

The request is being improperly sent to canvas.

Add option for manual tests

Add commenting and/or automated rubric

The grader can currently upload the grades to Canvas, but currently it would be useful to allow it to either update a rubric, or add submission comments.

So if we have the following entry in failures.json:

{
   "John Doe": {
      "if-then": -2,
      "nand": -1
   }
}

Where John Doe has the id 48879, a comment would be uploaded

Failed if-then: -2
Failed nand: -1
- Ed Minnix on January 31 at 10:32

Implement proper batch updates for submitting grades

https://canvas.instructure.com/doc/api/submissions.html#method.submissions_api.bulk_update

Consider splitting pycanvasgrader.py into multile files

Could be:

• PyCanvasGrader.py
• AssignmentTest.py
• TestSkeleton.py
• Grader.py (includes helper functions)

Add ability to download new submissions and update old ones

Should be both a batch job and an option for a single user

Add optional-requirements.txt

Clear the screen when a new menu is shown

May need to use curses

Add late policies to skeletons

When I use the grader for programming assignments in the class for which I am a TA, I need to manually go back and apply late penalties. I think it would be useful to have a late_policy attribute on skeletons. This way, I can have the grader also apply a late penalty when it grades the submissions.

Proposed Usage

For the course, I use the following late policy:

Minutes Late	Percentage Lost
0-5	10%
5-30	20%
30-60	40%
60+	100%

This would translate to the following skeleton

descriptor = "Sample"
disarm = false

[late_policy]
5 = 0.10
30 = 0.20
60 = 0.40
'*' = 1.00

This would translate under the hood to an sorted association list under the hood.

late_policy = [(5, 0.10), (30, 0.20), (60, 0.40), (float('inf'), 1.00)]

Time specification language

Format	Translation
<int>, <int>m	number of minutes
<int>h	number of hours
<int>d	number of days
*	Anything larger than the largest concrete upper-bound (treated as inf)

You can then combine the specifiers with -. So 1h-30m translates to 1 hour 30 minutes.

Grader should not need to be in same directory

The grader currently relies on the grader executable/script on being present in the same directory. This causes some synchronization issues, particularly when using source control on skeletons.

The preferred behavior would be to have a separation of concerns, such as

• Grader
• Skeletons, test scripts, etc
• Temporary directories and caches

Failing the first test will not set ungraded user to 0

failures.json should be in sorted order

The failures.json file would be much easier to work with if it provided the names in a sorted order.

Canvas sorts names by the second word in the name. E.g., Edward Minnix is sorted by Minnix, where as Maria von Trapp is sorted by von Trapp.

Better documentation strategy for skeletons and preferences

Github site? Still would lke some form of offline reference as well.

Display a course identifier next to course name to resolve duplicate course names

If two courses have the same name, there is no way to differentiate them. Add some kid of ID so that the user can see the difference. Should be something identifiable, like the course date.

Handle SIGINT/Keyboard Interrupt

Currently the grader will flop over with an unhandled KeyboardInterrupt when ^C is pressed. It would be useful if the grader gracefully handled ^C.

test log displays point values as integers

When viewing the result of a user's tests, the point values they received will be shown as integers, instead of floats. This does not affect the actual grade.

Users are not shown if they don't have a submission

Should be shown like name [no submission]