jnguyen1098/wyvern

University of Guelph website scraper that exports courses as .CSV

Wyvern

October 2021 Update: I have been busy and unable to update this repository—there are some tidbits in the other branch though where I've experimented a little bit with reading the new DOM. Assuming the old calendar links are still online, Wyvern should continue to work.

Scrapes the courselist of the University of Guelph website and writes it all to a .CSV file. I used BeautifulSoup to traverse the DOM.

usage: wyvern.py [-h] [-v] [-u URL] outfile

positional arguments:
  outfile              output file name

  optional arguments:
    -h, --help         show this help message and exit
    -v, --verbose      verbose debug output
    -u URL, --url URL  specify calendar URL

You'd only need to specify the [calendarurl] if the default URL becomes outdated. Please make an issue if there are any changes to the UofG website that break the scraper.

If you don't have the time to scrape an updated copy, there is a pre-made courses.csv file in this repo (though it may become outdated).

The .CSV file represents each course (row) with the following columns:

• Common Name
• Faculty
• Number
• Course Title
• Schedule
• Lecture Hours
• Lab Hours
• Weight
• Description
• Prerequisites
• Corequisites
• Restrictions
• Equates
• Departments
• Offerings
• External Course Codes

wyvern's Issues

simplified complex columns

Currently it does a naive regex matching of the entire prerequisites text and the restrictions text. I want these columns to remain, but I also want to make an easy-to-parse simplified restrictions and prerequisites column — that way if I were to scale the data to a web app, users can click on the nodes and receive the full, complex text, while only seeing accurate arrows

better documentation

self-explanatory

verbose mode

I'll probably add a verbose mode so everything being done is outputted to the screen in great detail. Useful for when the uoguelph site breaks my scraper

test and support different calendars

• Diploma calendar

• Undergraduate calendar

• Guelph-Humber calendar

support individual faculty export

Currently it scrapes all faculties and puts it into a giant .csv. I will probably make it:

• Export separate files, and

• Allow linking of an individual faculty and only export that