coursetable/ferry

A crawler for Yale courses and evaluation data. Integrates with Coursetable.

Coursetable

Coursetable is made of two big parts:

1. Website: The site you see when you go to coursetable.com. The code for this – the front-end site as well as the back-end server that handle user actions – is contained within this repository.
2. Crawler: The scripts behind the scenes that actually get all the data from Yale’s websites. The code for this is in our ferry repository.

Repository Layout

The various functions of the website are compartmentalized as follows:

• /api: Source code for API server with Docker Compose configuration for backend logic.
• /frontend: The current face of the site, built with React.

How to develop

Check out our contributing guide.

How to deploy

Deployments are automatically handled via GitHub Actions workflows. If necessary, you can also manually deploy. For all instructions relevant to deploying our code, see docs/deployment.md.

ferry's Issues

Evaluations for Summer 2020 courses are not retrieved

Despite OCE showing several summer courses with evaluations, running /ferry/crawler/fetch_ratings.py -s 202002 does not find any valid evaluations.

Add subject field to computed_course_info view

Support typing with mypy

Reduce memory footprint

transform.py uses nearly 12GB of RAM to construct the tables in Pandas. This can be reduced by dropping unused columns, reducing deep copies, and specifying more efficient column datatypes.

Look into using pg_stat_statements

Staging tables fails to rollback

Traceback (most recent call last):
  File "./ferry/stage.py", line 316, in <module>
    copy_from_stringio(raw_conn, professors, "professors_staged")
  File "/home/app/ferry/ferry/includes/importer.py", line 716, in copy_from_stringio
    cursor.rollback()
AttributeError: 'psycopg2.extensions.cursor' object has no attribute 'rollback'

Support sophomore seminars filtering

Indicate college seminars

Parse and import registrar notes

Certain courses have registrar notes associated with them. An example is CHNS 110, which has "Enrollment for this course is managed through Preference Selection". These are available in the regnotes field from the raw JSON responses.

Associate school with listing and not course

e.g. AFAM 118's school is Yale College but AFAM 518 is not

There might be some other fields that require this change as well.

Cache CI/CD dependencies

Tiebreaking on last enrollment numbers

There's a couple scenarios where we'll want to modify the last_enrollment field algorithms:

• When there's multiple sections with different professors (e.g. MATH 120, where Sudesh tends to have larger classes than other profs)
• When a previous professor comes back to teach a class that they previously taught, but didn't in the most recent instance (e.g. Stan with CPSC 323)

[Follow up on #37, cc @kevinhu]

Add "same overall" and "overall all" class ratings in ferry

Precedes coursetable/coursetable#229.

Remove full-text search logic

Outdated since we've moved it to the client-side

Store similar courses in database

• Add one of the course embedding algorithms to the import pipeline
• Store linked courses: perhaps a junction table?
  

  ferry/ferry/database/models.py

  Lines 40 to 51 in 3399f14

  	# Course-Professor association/junction table.
  	course_professors = Table(
  	"course_professors",
  	Base.metadata,
  	Column("course_id", ForeignKey("courses.course_id"), primary_key=True, index=True),
  	Column(
  	"professor_id",
  	ForeignKey("professors.professor_id"),
  	primary_key=True,
  	index=True,
  	),
  	)

Track enrollment last offered

Add computed fields in courses table tracking the evaluation_statistics.enrolled of the last time the course was offered (resolve by course codes) as well as the season and course_id of this last-enrolled date.

Planned fields:

• courses.last_enrolled
• courses.last_enrolled_season
• courses.last_enrolled_course

Track different sections of a course

Check out Hasura v2

https://www.producthunt.com/posts/hasura-graphql-engine-2-0

Static file generation

• Generate JSON files for all courses per season

Integrate evaluation sentiment estimates

• Create column(s) in evaluation_narratives with sentiment values.

Using issues in this repo

Due to the nature of this repo (being a "peripheral" service of CourseTable), every feature implemented here needs to eventually have a corresponding change in the website, and most changes here are motivated by features on the website. Therefore, in order for the team to better track work, all requests for new data structures should be directed to https://github.com/coursetable/coursetable instead, with the label https://github.com/coursetable/coursetable/labels/New%20data%20structure. Issues in this repo should be limited to the following purposes:

• Internal refactors
• Code bugs
• [Others; to be added later if they come up]

Address remaining DeepSource warnings

See https://deepsource.io/gh/coursetable/ferry/

Generate indexes and constraints on staged tables

follow up for #22

Extract and import Yale College distributional tags

• On the legacy site, these are the tags under the "Notes" section in the modal.
• These tags also specify if a course is a first-year seminar, which can be used as a secondary attribute.

Missed some first year seminars

e.g. AMST 034 and MUSI 012

Ferry should not redump the entire database

Every time Ferry recrawls, it basically recreates the DB. This makes it hard to tell what things have changed unless we inspect the ferry-data diff. We should make Ferry generate a diff and write data incrementally.

Different classes reusing the same course code

A very niche issue, brought to our attention by Professor Bensinger. We've only found this issue to affect one course code (CSTC 300). A fix could perhaps cross-validate classes using both the course code and the extended class name; currently we only group course codes together with the assumption that Yale does not reuse course codes.

CSTC 300 01 - https://coursetable.com/Table/201903/course/CSTC_300_1
2019 Fall - Leadership as Behavior
2014 Spring - Captivity & Law World History

Note if a course has been offered before

• Track if a course is being offered for the first time
• Detect based on title, codes, description

Split stage into csv + stage

Replace ujson.dumps with ujson.dump

Current method for exporting to JSON uses file.write(ujson.dumps(object)), which stringifies the JSON before writing. Instead, we can use usjon.dump(object, file) to write directly, which will reduce memory and should be a bit faster.

Check if discussion sections can be parsed from PDFs

Locations: https://www.sis.yale.edu/buildings/Discussion_locations.pdf
Demand: https://www.sis.yale.edu/buildings/ocs_ds_stats.pdf

Question codes have divergent texts

The Fall 2020 course evaluations appear to have mismatches in some question texts:

Traceback (most recent call last):
  File "./ferry/transform.py", line 155, in <module>
    ) = import_evaluations(
  File "/home/app/ferry/ferry/includes/transform_import.py", line 735, in import_evaluations
    raise database.InvariantError(
ferry.database.database_utilities.InvariantError: Error: question codes YC401, YC401-YCWR, YC403, YC403-YCWR, YC409, YC409-YCWR have divergent texts

Reorganize documentation

Rewrite and rearrange current docs, probably in the same format as coursetable/coursetable for clarity and consistency.

Analyze Hasura performance

• Search queries currently have a 600-800ms delay
• Try the 'Analyze' tool in Hasura console's GraphQL explorer

Fetch and import first-year seminar status

• Our responses don't include a status attribute for if a course is a first-year seminar.
• However, Yale Course Search allows us to filter for first-year seminars under the "Yale College Attributes" section.

Flush and regenerate all constraints when staging

• Staging tables should first delete all constraints on the previous ones (if they exist)
• Created staged tables should reflect all constraints in the SQLAlchemy model except foreign keys