mechami/slo_scraper

Scraper for Student/Program Learning Objectives

slo_scraper's Issues

Fetchall causes an error in unit tests

In PLODB_tests using fetchmany or fetchall on a database resource causes tests to hang after an error.

Missing noncredit certifications

Scraper isn't pulling noncredit certifications.

Inconsistent department

Department name pulled from site is inconstant with department drop down menu from http://butte.edu/academicprograms/.

Alternative courses need formatting

When scraping courses related to a program, some courses are alternatives and prefixed with 'or' eg. 'ART 2' and 'or ART 2H' in the Graphic design program.

Possible fix:
1. Text processing to normalize the format of alternative courses eg. 'or ART 2H' becomes 'ART 2H'.
2. Create a self referential relationship on the programs_courses table. Each programs_courses entry is a required course in the program, a required course can have many alternative courses.

Courses Dictionary causes an error

In PLODB, adding the courses list as a list of dictionaries in the plo_data dictionary
causes 'TypeError: sequence item 0: expected str instance, dict found' for any SQL query

Workaround:
The dictionaries are separated before insertion.

Programs with identical outcomes

Identical CERT programs or alternate degrees exist for many programs, which contain the same outcomes (Certificate of Achievement in Early Childhood Education, AS-T Degree in Early Childhood Education). Currently the CERT outcomes are stored separately from the degree outcomes, but if they are always identical and assessed at the same time then this could lead to data anomalies. Some programs also share an outcome, but have other differing outcomes (AS Degree in Agricultural Business, AS Degree in Agriculture Business).

Possible fix:
The relationship between programs and outcomes could become many to many.

Programs can share outcomes, but each degree requires different courses, so a separate   program entry is required for each degree.

A program can have many outcomes; an outcome can be in many programs. A new entry will be created in a join table for each of a program's outcomes.

ex.
-
Early Childhood Education (AS-T) is inserted into the database

When Early Childhood Education (CERT) is inserted into the database, 
1. Insert Early Childhood Education (CERT) into the program table
2. Insert related outcomes into the outcome table, since they were already inserted along with the 
	AS-T, the duplicates will be ignored. Any new outcomes will be inserted.
3. Create a join table entry between the program and each outcome

-

To create a solution, it will need to be determined if outcomes are assessed for individual programs or for all programs that contain that outcome.

Program name with '( 'or ')' characters are not inserted into DB

the program "Certificate in Emergency Medical Responder (EMR)" is not being inserted into DB but is being scraped.

Conserving Manual Database Additions

The items in the database can be deleted, edited, and/or created by select users. The issue arises is that theses changes are not conserved when the slo-scraper executes again. Re-acquiring information via scraper will not keep any manual changes or additions to the database. May need to add a special tag that differentiates modified scraped items and manually created new items so they are not deleted from the database or they can be removed, but then re-added after re-scraping via execution.