diku-dk/staffeli

DIKU Support Tools for Canvas LMS

staffeli — DIKU Support Tools for Canvas LMS

These tools leverage the Canvas LMS REST API to create a more pleasant environment for working with Absalon.

"Staffeli" is Danish for "easel" — a support frame for holding up a canvas.

Contents

• 1   Purpose
• 2   Status
• 3   Installation
• 4   Getting Started
  • 4.1   Cloning a Course
  • 4.2   Fetch Submissions for a New Assignment
  • 4.3   Fetch Groups
  • 4.4   Grade a Submission
  • 4.5   Split Submissions among TAs
• 5   Documentation
• 6   Contributing
• 7   Testing
  • 7.1   Static Testing Framework
  • 7.2   Dynamic Testing Framework
  • 7.3   Dynamic Test Coverage

1   Purpose

The purpose of Staffeli is two-fold:

1. Leverage the Canvas LMS REST API to get things done better, faster, stronger.
2. Quick prototyping of new features for Canvas LMS.

Initially, Staffeli is not intended for managing course content, merely to snapshot course data (e.g., enrolled students, groups, sections, submissions), and to get grading done efficiently.

Although Staffeli is written in Python 3, it is not intent on forcing you to manage your course using Python, or to have to get intimate with the Staffeli API to get things done. Staffeli extensively uses YAML files for storage, enabling the easy use of both command-line utilities and the programming language of your choice, to get things done quickly, and efficiently.

2   Status

Staffeli is maturing. It is being transitioned to be annotated with type hints, with the types checked statically with mypy, and a test-suite has been set up, but full-blown continuous integration remains to be set up.

We are still covering a fairly small subset of the API. Brace yourself. Lend a hand.

3   Installation

These are instructions for installing Staffeli on your system.

1. Download this repository or git clone https://github.com/DIKU-EDU/staffeli.git
2. Enter directory and run pip3 install -e .

These are instructions for installing Staffeli in a user directory.

1. Check that you have Python 3 and pip installed: which python3 pip3
2. Install virtualenv globally, e.g. with sudo pip3 install virtualenv
3. Download this repository or git clone https://github.com/DIKU-EDU/staffeli.git
4. Enter directory and create virtualenv with virtualenv .
5. Activate virtualenv with source bin/activate
6. Install Staffeli in virtualenv, pip3 install -e .

4   Getting Started

With Staffeli, we work with local course clones. We aim to keep these clones compatible with git.

We recommend that you create a local directory canvas, absalon, or similar, for all of you Canvas-related local course clones. Staffeli needs some initial help to be able to login with your credentials. You need to generate a token for Staffeli to use, and save it as .token, token, or token.txt in this high-level directory.

NB! This is your personal token so do not share it with others, else they can easily impersonate you using a tool like Staffeli. Unfortunately, to the best of our knowledge, Canvas has no means to segregate or specialize tokens, so this is really "all or nothing".

4.1   Cloning a Course

To clone a course:

$ staffeli clone '<course name>' [dir]

The 'course name' is any case-insensitive substring of the course name as it appears on your dashboard. Use quotes in case the substring contains spaces. If there are multiple conflicting names matching the substring, Staffeli will complain and let you try again.

The [dir] is an optional destination directory for the local working area, in case it should not be named <course string>. For example:

$ staffeli clone 'Advanced programming' ap17
Cloning '5100-B1-1E17;Advanced programming' into 'ap17'...

4.2   Fetch Submissions for a New Assignment

Use staffeli fetch. For instance, to fetch all submissions for "A3":

$ staffeli fetch subs/A3

To fetch just the metadata for all submissions, but not the submissions themselves:

$ staffeli fetch subs

4.3   Fetch Groups

This is a good idea to make sure you are up-to-date with canvas.

$ staffeli fetch groups

4.4   Grade a Submission

Assuming you are in the submission directory, you can use staffeli grade to grade the submission:

staffeli grade GRADE [-m COMMENT] [FILEPATH]...

Where
    GRADE           pass, fail, or an int.
    [-m COMMENT]    An optional comment to write.
    [-f FILEPATH]   Upload the contents of a file as a comment.
    [FILEPATH]...   Optional files to upload alongside.

4.5   Split Submissions among TAs

See the groups contrib.

5   Documentation

It is up-and-coming on [ReadTheDocs](http://staffeli.readthedocs.io/en/latest/). The source files for that page are under [docs/source](docs/source), and they are, much like this README, written in reStructured Text. It is also suggested to follow the Python documentation style guide.

6   Contributing

First, take a look at our design guide and style guide.

Contact us at dikunix at dikumail dot dk.

Take a look at our on-going issues.

7   Testing

Currently, Travis CI will only check that you roughly conform to the PEP 8 Python Style Guide (using flake8), and perform static type-checking with mypy, all only for selected Python files in this repository. See (and run?) `static_tests.py <static_tests.py>`__ for further details.

Before you do that however, you might want to do this:

$ pip3 install -r test-requirements.txt

This will also install what you need to run the dynamic tests we have in store under tests, except for Docker: `start_local_canvas.py <start_local_canvas.py>`__ will fire up a Docker image with a local Canvas instance for use with our tests. You will also find it in your browser under the address localhost:3000. The user is [email protected] and the password is canvas.

The static and dynamic tests are also part of the `pre-commit <hooks/pre-commit>`__ and `pre-push <hooks/pre-push>`__ hooks, respectively. Install these hooks by executing `hooks/install.sh <hooks/install.sh>`__. Unfortunately, neither these hooks, nor the hooks installer will work on Windows.

7.1   Static Testing Framework

We use flake8 for style-checking and mypy for static type-checking.

Assuming you have these tools installed, you can do this:

$ ./static_tests.py

This is also part of the `pre-commit <hooks/pre-commit>`__ hook.

7.2   Dynamic Testing Framework

We use pytest together with hypothesis.

Assuming you have these tools installed, you can do this:

$ pytest

This is also part of the pre-push hook.

7.3   Dynamic Test Coverage

Run pytest with the option --cov=staffeli to get an idea of the test coverage of Staffeli proper.

It is pretty lousy ATM. As of 2017-05-24, the numbers were:

Name                          Stmts   Miss  Cover
-------------------------------------------------
staffeli/assignment.py           28     28     0%
staffeli/cachable.py             22     13    41%
staffeli/canvas.py              326    326     0%
staffeli/cli.py                 295    295     0%
staffeli/course.py               43      4    91%
staffeli/files.py                57     41    28%
staffeli/gcat.py                 29      5    83%
staffeli/group.py                23     23     0%
staffeli/listed.py               31      8    74%
staffeli/names.py                 3      1    67%
staffeli/resubmissions.py       121    121     0%
staffeli/speedgrader_url.py       9      9     0%
staffeli/submission.py           22     22     0%
staffeli/typed_canvas.py        112      4    96%
staffeli/upload.py               17     17     0%
-------------------------------------------------
TOTAL                          1138    917    19%

staffeli's Issues

Moving

I am transferring to University in Oslo, but I intend to continue contributing/maintaining Staffeli, not least because they also use Canvas at UiO. Staffeli is mostly a DIKU project, so maybe it is best suited under the flag of the diku-dk organization. If you agree, how about a team like "eduhackers" for those involved in Staffeli development under diku-dk?

Login using OAuth2

This would greatly reduce the entry curve to staffeli.

More informative name than "Assignment 2 Group 071"

When grading through the SpeedGrader (I don't, but others might) and assignment groups have been created for an assignment, the dropdown list contains group names that are 1) hard to distinguish or remember, and 2) hard to know if they belong to me or not, since I can no longer filter this list based on my class.

• A more informative name might be "Group 2-071 (Adolf, Eva)". Would this naming scheme pose any problems wrt. automation? I suspect not, since groups associated with a group category can be traced back to their group category through IDs in the API.
• In spite of this better naming scheme, it appears that TA's who use the SpeedGrader still don't know what groups they're correcting. It appears that this information is not being put into Canvas at all, but resides in the directory structure of the grading repository that TA's use. Is there any way to annotate a group with its section (e.g. 'Class Simon') in other ways than making it part of the group name?

@oleks, @nqpz, @athas: Opinions?

typing module is required by package but not in setup.py

Require explicit command-line option for empty feedback

It's too easy to leave off -f or -m by accident.

Sections

It is time to make sections manageable. API can do this.

Sections allow a sane, section-based overview of the submissions in SpeedGrader™.

We should tune in to that.

Previously, we've abused groups to split up students based on TA's (having a TA group set). This is because there is a nice drag-and-drop UI to manage groups, and scripts could be written on the other side to internally give the illusion of "sections".

👍 if you would like this feature.

Staffeli should not silently use the message 'See attached files' if I pass neither -m nor -f when correcting

Instead it should not upload a comment at all, or (if required), the comment 'No further comment' or similar.

A documented workflow for handling resubmissions

We need a workflow for handling resubmissions. This may not involve writing new code, but merely documenting what's already there. There is a resubmissions.py script, but I do not understand what it does.

I think a good approach would be based on the existing logic for fetching submissions. The assumption is that at some point in the past, the original submissions were fetched, and this directory is still intact. When next fetching submissions for the same assignment, the tool will check for each submission whether it is identical to the previous one, and if so discard it. A special file should be created in directories for submissions that are only present in the "resubmission", and were not originally submitted.

Add a setup.py

Manual PATH/PYTHONPATH manipulation is bad Python, I hear.

Also, people like less tedious installation processes.

Consider using CanvasAPI

The is another Python wrapper of the Canvas API maybe we could use that, rather than talking directly to the REST API.

Fix ReadTheDocs

It be broken, and I don't understand why.

Make contrib/groups/* safer and more well-documented

• The documentation for contrib/groups/create-groups.py isn't accurate wrt. number of arguments.

• check-groups.py should assert that the group.txts it finds are actually plaintext files and not RTF documents or pictures of kittens.

• create-groups.py fails when two individuals both upload a group.txt that put them in the same group:

    (staffeli) ~/Courses/ap17/ap-e2017-grading $ create-groups.py subs/1/ 'Assignment 1 Groups' subs/1/groups.txt    
    Traceback (most recent call last):
      File "/home/simon/Courses/ap17/staffeli/contrib/groups/create-groups.py", line 62, in <module>
        sys.exit(create(*sys.argv[1:]))
      File "/home/simon/Courses/ap17/staffeli/contrib/groups/create-groups.py", line 51, in create
        assert len(subpaths) == 1
    AssertionError
  

  Either the documentation should be clear about what to when check-groups.py fails in various ways, or the script should handle the situation gracefully, e.g. by detecting that groups (abc123 def456) and (def456 abc123) are the same and only create one group for them.

• It would be neat if the script had a "dry run" parameter that doesn't create groups but fetches the ones that are created and performs the filesystem operations with them. Currently this isn't possible because canvas.create_group() (used here) both creates a group on Canvas and returns the group's properties. If there were a canvas.get_groups(), that'd be neat.

Typecheck the Python

See http://mypy-lang.org/

Use junctions instead of symlinks on Windows

In cli.groupsplit we should use junctions on Windows, as symlinks require admin privileges. See http://stackoverflow.com/a/32878688

Set up a test suite

This will have to be a bit manual as at it requires a dummy course and some dummy students doing dummy work. Other than that, it is probably best to go with Hypothesis.

Missing submissions

When I use the command staffeli --metadata fetch subs/<assignment> I get a lot of empty folders named by KUID_, but no submissions.

Improve StudentList class

• Use part before '@' as KUID instead of first six characters. This seems safer.
• Deprecate use of sis_login_id since this is actually not available Absalon's JSON.
• Define __contains__ for StudentList so student_id in students works when student_id is either a Canvas user_id or a KUID.
• Refer to StudentList instead of StudentList(...).mapping
• Use StudentList instead of canvas.list_users() when it makes sense.

Add a license declaration

Create proper installers

People might have a hard time modifying PATH and PYTHONPATH.

On Linux, it might be best to use AppImage, or similar.

Staffeli example

The guide at http://staffeli.readthedocs.io/en/latest/examples.html#list-students-who-did-not-submit :

• cat students/.staffeli.yml | grep sis_login_id
  change_to
  cat students/.staffeli.yml | grep login_id
  (changes in canvas ?)

Make a demo video

Add feedback.py to `bin/`

Staffeli CLI 'staffeli clone' should let you pick destination directory

Instead of staffeli clone Advanced programming I want to be able to write staffeli clone "Advanced programming" ap17. Since this breaks current usage, I want to ask if people think this is a good idea.

feedback.py is broken

Uploading via URL imposes some a limit (~50MB) on the amount of data that can be uploaded wrt. submissions. See also our Canvas Community post.

I suppose our best option is to once again try uploading via POST. Last we tried, Canvas didn't yield all the fields necessary for upload to their S3 store.

Treating paths as strings is a bad assumption.

Currently Staffeli assumes specific string representations of paths. This leads to an incompatibility with e.g. PowerShell as it prepends .\ to all relative paths.

Integration with Google Drive may be surprisingly easy

https://www.twilio.com/blog/2017/02/an-easy-way-to-read-and-write-to-a-google-spreadsheet-in-python.html

Actually download submissions via script

I now know how to do this.

`staffeli fetch subs/N` creates 'new-download' when files match exactly.

This makes difficult the workflow of calling staffeli fetch subs/N multiple times in case of later submissions. Also, Hitting ^C in the process causes an exception that could give a graceful status of the progress before exiting, if not simply exit gracefully because what was printed beforehand was informative enough.

Automate canvas.yaml hierarchy setup

Clarify/redo design

There are many layers in the code, and it is not clear to me how they work together.

For example, I'm perpetually confused by the Cachable* Python classes (though I kind of understand their purpose).

Also, when I had to add someone to a group on Absalon the other day, staffeli insisted on first downloading all groups before doing any changes. I suppose the idea here is to always have an up-to-date local copy of the Absalon groups (among other things), but I question whether this is necessary.

Canvas API functions for submissions

API functions for getting submissions for one or more assignments are only available as methods in the Assignment class that does filesystem walks and for which the constructor does heavy work.

• Move this to the Canvas class and make list_submissions alike list_assignments.

For some reason, the Canvas API supports listing submissions in a bunch of ways:

• List assignment submissions (the one used in Assignment):
  GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions

• Get a single submission (the one used in submission_history):
  GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions/:user_id, or
  GET /api/v1/sections/:section_id/assignments/:assignment_id/submissions/:user_id

• Add API functions for listing submissions for multiple assignments.

• List submissions for multiple assignments:
  GET /api/v1/courses/:course_id/students/submissions, or
  GET /api/v1/sections/:section_id/students/submissions

Python vs. Golang vs. Rust — I think we're sticking with Python

Staffeli is written in Python 3 for the following reason: "Python 3, is deemed[1] a popular, modern, cross-platform language. Python 3 is the primary choice for every part of Staffeli. Cross-platform Python 3 code SHOULD be preferred over non-cross-platform code. For instance, when handling file-system paths." — DESIGN.md.

I have on occasion discussed rewriting Staffeli in Golang or Rust.

The Python implementation has grown to use mypy for static type checking, flake8 for style conformity, as well as pytest and hypothesis for (quick-check style) testing.

I believe that Staffeli should move on to have a simple, safe core (statically typed and tested quick-check style), but allow people to hack around it in the interest of deadlines, without being subject to both a pedantic type-checker and having to write thorough tests. (For instance, you might implement a Staffeli extension, write a simple test for it, send it through a continuous integration pipeline, and go on to the real work. Then, you can come back later, when you have more time, type your code and write more exhaustive tests.)

The benefit of Golang or Rust is that their type-systems are much more mature than mypy. The benefit of Python is that dynamic types are nice for when you want to just hack around Staffeli. That however, could be achieved with a mere Python language binding around a Golang/Rust core.

The quick-check sittuation is also much nicer in Python and Rust than it is in Golang, but Rust IMO is harder for newcomers to learn than Golang.

Also mentioning @kfl @athas @sshine.

__
[1]: By @oleks, @nqpz, @orkeren

Accept files for input the feedback messages when grading

Installation instructions with virtualenv

Getting KU login from name, and reverse

In some situations it would be really nice to be able to get the name of a student from the KU login, or to get the KU login from a (partial) name of the student.

The later seems to be almost possible with the user find command.

resubmissions.py is wonky

• Check that grade was not given much later than the latest comment. This is likely to be due to the UI peculiarities of Canvas LMS rather than being intentional. If this is the case, download the resubmission as if not yet graded.

Submissions

Use the Submissions API to download submissions.

Python API should be something like this:

• Assignment object gets a submissions() function.
• This yields some Submission objects.
• A Submission object allows to fetch attempts and attachments one at a time.

Command-line API should be something like this:

staffeli clone <course name>
cd <course name>
staffeli subs <assignment>
cd subs/<assignment>
staffeli fetch # Fetch all the current submissions and resubmissions (attempts).

Upload to pip repo

`staffeli grade` does not parse attachments correctly

I would expect staffeli grade pass --no-feedback feedback.txt to upload feedback.txt as an attachment, per the usage message. Yet it does not.