diku-dk/staffeli

DIKU Support Tools for Canvas LMS

staffeli's Issues

Staffeli should not silently use the message 'See attached files' if I pass neither -m nor -f when correcting

Instead it should not upload a comment at all, or (if required), the comment 'No further comment' or similar.

Fix ReadTheDocs

It be broken, and I don't understand why.

Clarify/redo design

There are many layers in the code, and it is not clear to me how they work together.

For example, I'm perpetually confused by the Cachable* Python classes (though I kind of understand their purpose).

Also, when I had to add someone to a group on Absalon the other day, staffeli insisted on first downloading all groups before doing any changes. I suppose the idea here is to always have an up-to-date local copy of the Absalon groups (among other things), but I question whether this is necessary.

Upload to pip repo

typing module is required by package but not in setup.py

A documented workflow for handling resubmissions

We need a workflow for handling resubmissions. This may not involve writing new code, but merely documenting what's already there. There is a resubmissions.py script, but I do not understand what it does.

I think a good approach would be based on the existing logic for fetching submissions. The assumption is that at some point in the past, the original submissions were fetched, and this directory is still intact. When next fetching submissions for the same assignment, the tool will check for each submission whether it is identical to the previous one, and if so discard it. A special file should be created in directories for submissions that are only present in the "resubmission", and were not originally submitted.

Add a license declaration

Login using OAuth2

This would greatly reduce the entry curve to staffeli.

Moving

I am transferring to University in Oslo, but I intend to continue contributing/maintaining Staffeli, not least because they also use Canvas at UiO. Staffeli is mostly a DIKU project, so maybe it is best suited under the flag of the diku-dk organization. If you agree, how about a team like "eduhackers" for those involved in Staffeli development under diku-dk?

Typecheck the Python

See http://mypy-lang.org/

Actually download submissions via script

I now know how to do this.

Accept files for input the feedback messages when grading

Staffeli example

The guide at http://staffeli.readthedocs.io/en/latest/examples.html#list-students-who-did-not-submit :

• cat students/.staffeli.yml | grep sis_login_id
  change_to
  cat students/.staffeli.yml | grep login_id
  (changes in canvas ?)

Treating paths as strings is a bad assumption.

Currently Staffeli assumes specific string representations of paths. This leads to an incompatibility with e.g. PowerShell as it prepends .\ to all relative paths.

Canvas API functions for submissions

API functions for getting submissions for one or more assignments are only available as methods in the Assignment class that does filesystem walks and for which the constructor does heavy work.

• Move this to the Canvas class and make list_submissions alike list_assignments.

For some reason, the Canvas API supports listing submissions in a bunch of ways:

• List assignment submissions (the one used in Assignment):
  GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions

• Get a single submission (the one used in submission_history):
  GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions/:user_id, or
  GET /api/v1/sections/:section_id/assignments/:assignment_id/submissions/:user_id

• Add API functions for listing submissions for multiple assignments.

• List submissions for multiple assignments:
  GET /api/v1/courses/:course_id/students/submissions, or
  GET /api/v1/sections/:section_id/students/submissions

Improve StudentList class

• Use part before '@' as KUID instead of first six characters. This seems safer.
• Deprecate use of sis_login_id since this is actually not available Absalon's JSON.
• Define __contains__ for StudentList so student_id in students works when student_id is either a Canvas user_id or a KUID.
• Refer to StudentList instead of StudentList(...).mapping
• Use StudentList instead of canvas.list_users() when it makes sense.

Make contrib/groups/* safer and more well-documented

• The documentation for contrib/groups/create-groups.py isn't accurate wrt. number of arguments.

• check-groups.py should assert that the group.txts it finds are actually plaintext files and not RTF documents or pictures of kittens.

• create-groups.py fails when two individuals both upload a group.txt that put them in the same group:

    (staffeli) ~/Courses/ap17/ap-e2017-grading $ create-groups.py subs/1/ 'Assignment 1 Groups' subs/1/groups.txt    
    Traceback (most recent call last):
      File "/home/simon/Courses/ap17/staffeli/contrib/groups/create-groups.py", line 62, in <module>
        sys.exit(create(*sys.argv[1:]))
      File "/home/simon/Courses/ap17/staffeli/contrib/groups/create-groups.py", line 51, in create
        assert len(subpaths) == 1
    AssertionError
  

  Either the documentation should be clear about what to when check-groups.py fails in various ways, or the script should handle the situation gracefully, e.g. by detecting that groups (abc123 def456) and (def456 abc123) are the same and only create one group for them.

• It would be neat if the script had a "dry run" parameter that doesn't create groups but fetches the ones that are created and performs the filesystem operations with them. Currently this isn't possible because canvas.create_group() (used here) both creates a group on Canvas and returns the group's properties. If there were a canvas.get_groups(), that'd be neat.

`staffeli grade` does not parse attachments correctly

I would expect staffeli grade pass --no-feedback feedback.txt to upload feedback.txt as an attachment, per the usage message. Yet it does not.

resubmissions.py is wonky

• Check that grade was not given much later than the latest comment. This is likely to be due to the UI peculiarities of Canvas LMS rather than being intentional. If this is the case, download the resubmission as if not yet graded.

Add feedback.py to `bin/`

Add a setup.py

Manual PATH/PYTHONPATH manipulation is bad Python, I hear.

Also, people like less tedious installation processes.

Missing submissions

When I use the command staffeli --metadata fetch subs/<assignment> I get a lot of empty folders named by KUID_, but no submissions.

Submissions

Use the Submissions API to download submissions.

Python API should be something like this:

• Assignment object gets a submissions() function.
• This yields some Submission objects.
• A Submission object allows to fetch attempts and attachments one at a time.

Command-line API should be something like this:

staffeli clone <course name>
cd <course name>
staffeli subs <assignment>
cd subs/<assignment>
staffeli fetch # Fetch all the current submissions and resubmissions (attempts).

Make a demo video

Python vs. Golang vs. Rust — I think we're sticking with Python

Staffeli is written in Python 3 for the following reason: "Python 3, is deemed[1] a popular, modern, cross-platform language. Python 3 is the primary choice for every part of Staffeli. Cross-platform Python 3 code SHOULD be preferred over non-cross-platform code. For instance, when handling file-system paths." — DESIGN.md.

I have on occasion discussed rewriting Staffeli in Golang or Rust.

The Python implementation has grown to use mypy for static type checking, flake8 for style conformity, as well as pytest and hypothesis for (quick-check style) testing.

I believe that Staffeli should move on to have a simple, safe core (statically typed and tested quick-check style), but allow people to hack around it in the interest of deadlines, without being subject to both a pedantic type-checker and having to write thorough tests. (For instance, you might implement a Staffeli extension, write a simple test for it, send it through a continuous integration pipeline, and go on to the real work. Then, you can come back later, when you have more time, type your code and write more exhaustive tests.)

The benefit of Golang or Rust is that their type-systems are much more mature than mypy. The benefit of Python is that dynamic types are nice for when you want to just hack around Staffeli. That however, could be achieved with a mere Python language binding around a Golang/Rust core.

The quick-check sittuation is also much nicer in Python and Rust than it is in Golang, but Rust IMO is harder for newcomers to learn than Golang.

Also mentioning @kfl @athas @sshine.

__
[1]: By @oleks, @nqpz, @orkeren

Installation instructions with virtualenv

More informative name than "Assignment 2 Group 071"

When grading through the SpeedGrader (I don't, but others might) and assignment groups have been created for an assignment, the dropdown list contains group names that are 1) hard to distinguish or remember, and 2) hard to know if they belong to me or not, since I can no longer filter this list based on my class.

• A more informative name might be "Group 2-071 (Adolf, Eva)". Would this naming scheme pose any problems wrt. automation? I suspect not, since groups associated with a group category can be traced back to their group category through IDs in the API.
• In spite of this better naming scheme, it appears that TA's who use the SpeedGrader still don't know what groups they're correcting. It appears that this information is not being put into Canvas at all, but resides in the directory structure of the grading repository that TA's use. Is there any way to annotate a group with its section (e.g. 'Class Simon') in other ways than making it part of the group name?

@oleks, @nqpz, @athas: Opinions?

Use junctions instead of symlinks on Windows

In cli.groupsplit we should use junctions on Windows, as symlinks require admin privileges. See http://stackoverflow.com/a/32878688

Staffeli CLI 'staffeli clone' should let you pick destination directory

Instead of staffeli clone Advanced programming I want to be able to write staffeli clone "Advanced programming" ap17. Since this breaks current usage, I want to ask if people think this is a good idea.

Create proper installers

People might have a hard time modifying PATH and PYTHONPATH.

On Linux, it might be best to use AppImage, or similar.

Integration with Google Drive may be surprisingly easy

https://www.twilio.com/blog/2017/02/an-easy-way-to-read-and-write-to-a-google-spreadsheet-in-python.html

Automate canvas.yaml hierarchy setup

feedback.py is broken

Uploading via URL imposes some a limit (~50MB) on the amount of data that can be uploaded wrt. submissions. See also our Canvas Community post.

I suppose our best option is to once again try uploading via POST. Last we tried, Canvas didn't yield all the fields necessary for upload to their S3 store.

Require explicit command-line option for empty feedback

It's too easy to leave off -f or -m by accident.

Sections

It is time to make sections manageable. API can do this.

Sections allow a sane, section-based overview of the submissions in SpeedGrader™.

We should tune in to that.

Previously, we've abused groups to split up students based on TA's (having a TA group set). This is because there is a nice drag-and-drop UI to manage groups, and scripts could be written on the other side to internally give the illusion of "sections".

👍 if you would like this feature.

`staffeli fetch subs/N` creates 'new-download' when files match exactly.

This makes difficult the workflow of calling staffeli fetch subs/N multiple times in case of later submissions. Also, Hitting ^C in the process causes an exception that could give a graceful status of the progress before exiting, if not simply exit gracefully because what was printed beforehand was informative enough.

Consider using CanvasAPI

The is another Python wrapper of the Canvas API maybe we could use that, rather than talking directly to the REST API.

Set up a test suite

This will have to be a bit manual as at it requires a dummy course and some dummy students doing dummy work. Other than that, it is probably best to go with Hypothesis.

Getting KU login from name, and reverse

In some situations it would be really nice to be able to get the name of a student from the KU login, or to get the KU login from a (partial) name of the student.

The later seems to be almost possible with the user find command.