cca/libraries_course_lists2

process Workday Student data into VAULT taxonomies

Libraries Course Lists 2.0

Brand new, redesigned course list handling for VAULT courtesy of Workday Student. The app creates an ORM of sorts around the openEQUELLA APIs, with objects for Groups, Taxonomies, and Taxonomy Terms, that lets us manipulate data easily.

Setup

1. Install the Google Cloud CLI (needed for gsutil)
2. Set up a python virtual environment & install dependencies pipenv install
3. Obtain a VAULT OAuth token for API use, cp example.config.py config.py, add the token to it. You can also change the logging configuration here if you so choose.
4. Obtain access to CCA Integrations data in Google Cloud (contact Integration Engineer). There should be JSON files present for employees, students, and courses for current semesters. With the integrations project active, running gcloud auth application-default login once will authenticate this project.

Usage

The main app should work now but has yet to be used to create taxonomies in VAULT. Thus far only unit tests have been performed.

usage: app.py [-h] [-c] [--course-lists] [-d] file

Create VAULT taxonomies from JSON course data.

positional arguments:
  file                 course list JSON file

optional arguments:
  -h, --help           show this help message and exit
  -c, --clear          only clear the given semester taxonomy term, do not
                       create new terms
  --course-lists       only create terms in course list taxonomies, ignore
                       others
  -d, --downloadtaxos  download fresh taxonomies from VAULT (do not use JSON
                       list in /data dir)

Logging information is sent both to stdout and to a dated log file in the "data" directory.

The taxonomies JSON is stored in data/taxonomies.json (not all their terms, just taxonomy names and identifiers). If you create a new taxonomy related to course lists or course information, e.g. if a new academic program is created, you'll need to rerun python app.py --downloadtaxos to refresh the JSON.

python faculty_groups.py data/data.json creates many text file lists of faculty usernames in the "data" directory. Each file is named after the LDAP group that the accounts belong to.

python make_informer_csv.py downloads the Workday JSON course data and transforms it into an "_informer.csv" spreadsheet. This can then be used in the previous "libraries_course_lists" project.

Testing

> pipenv run cover # run tests
> pipenv run report # coverage report
> coverage html # generate HTML in htmlcov/ that shows which lines aren't tested
> python -m unittest test.test_course # run a specific test suite

Add tests to the "test" folder and name them like "test_FILENAME" where FILENAME is roughly the name of the module that's being tested. This ensures unittest can discover them and makes it easier to see what tests still need to be written. You may need to create fixtures in both VAULT and the local filesystem to write some tests. Prefer using a fake, created datum to testing against production data that naturally changes.

LICENSE

ECL Version 2.0

libraries_course_lists2's Issues

fix gcloud auth for make_informer_csv

The make informer csv script no longer works on my new laptop, it must be because of a difference in the gcloud python library authentication, but I can't figure out what it is. Might be worth creating a service account for the script and saving the credentials in the project.

reverse the sorted courses before adding them

By adding, for instance, WRLIT last it actually floats to the top of the taxonomy with this new method, causing things to come out reversed. I just need to reverse the sorted courses before adding to fix this.

tests

https://docs.python.org/3/library/unittest.html

logging

https://docs.python.org/3/library/logging.html

speed up term addition, perhaps return to SOAP taxonomy wrapper

Nice as my new method is, it's incredibly slow: it took a little over two hours to sync only summer course lists, while the old process could do a full term in well under an hour. I may need to revisit my uptaxo script and find a way to utilize that again but with a stronger backbone than the series of shell scripts in the original course lists repository.