middlebury / coursecatalog Goto Github PK

This Course-Catalog is a project to develop a web front-end for searching and browsing course information stored in Banner.

Home Page: https://github.com/middlebury/coursecatalog/wiki

License: GNU General Public License v3.0

PHP 91.82% HTML 5.86% CSS 0.71% JavaScript 1.61%

ews

coursecatalog's People

Contributors

Stargazers

Watchers

Forkers

guniorobot

coursecatalog's Issues

Revision diffs should be link-able

Currently the diff viewer uses a POST form with what looks like the full JSON data. Unfortunately, this means that it isn't possible to link to a particular diff in email or other communications. It would be preferable for the diff selection form to send the user to a URL like /admin/export/1/revisiondiff/3/5 where the last two parameters are the starting and ending revisions chosen. This would also avoid posting big JSON blobs.

Add routes for export URLs

This is related to #6 , but the URL should use the numeric ID for the config so that the URL won't break if the title contains slashes or other special characters.

Also, a route should be added to application/library/routes.php so that the URL is more clean:

Example:

/admin/export would be the main screen with a selector for configs
/admin/export/1 is the detail screen.
/admin/export/1/history can be the history screen
/admin/export/1/json/71 would be the current viewJson action for a revision 71.
etc

You can still use $request->getParam('configId') in the controller, it will just be mapped from the positional location to a named parameter by the router.

Export Config: Simplify group styling

The current styling uses alternating colors for each group and a very heavy font, making the UI kind of heavy and a bit hard to scan/focus.

Use a single background color for groups.
Use a lighter or smaller font styling for the titles to make it easier to scan the list and find a particular department.
(Maybe) Shift the insert links to the right (either a little or fully) to reduce visual clutter when scanning.

Chef cookbook includes old archive_config.ini settings

Given new archive export UI, the default archive_config.ini file has a lot of unneeded settings. These should be removed from the cookbook.

Exports will get stuck if they don't complete.

If an export fails, there is no mechanism to check-for and remove the failed job and start a new one.

The archive.generate action should insert the current process-id (PID) into the database and check that the process is still running when called a second time. If the PID no longer exists, then the failed entry should be dropped and a new job started.
https://github.com/middlebury/coursecatalog/blob/master/application/controllers/ArchiveController.php#L360-L376

Export Jobs should allow selecting of unavailable terms, but with message.

Currently, creating a job for upcoming terms that are not yet enable results in an error that is hard to decipher:

The work-flow of the Registrar and Academic Affairs office is that Banner data gets copied from the previous year and then edited to make it reflect the new year's courses. They don't "enable" upcoming terms until data has been mostly cleaned up and validated.

It would be nice if they could create a new job for upcoming terms that are not yet enabled, but maybe not activate it until the terms are active. If the terms entered aren't active, then there should be a highlight and message indicating that one or more of the terms isn't active and that the job will not run until all are active.

Archive exports need to handle D9 requrirements pages

Archive exports are currently looking at D7 dept/requirements/feed pages for content which don't exist in D9.

Review the Course Catalog server logs for errors, bugs, and performance issues

Check the Course Catalog logs and enter as many GitHub issues based on them as you can in a day.

Move the markup testing tool to the admin screen

The markup testing tool is currently installed at
http://hammer.middlebury.edu/~afranco/catalog_markup/index.php

It should be made into an admin screen in the catalog proper to live under /admin so that it is more easily available for those doing data entry. It should be updated to strip out any javascript and because no data is saved, can be safely exposed without special authorization.

Requirements pages aren't being fetched on nightly catalog rebuild

Kathleen noticed today that updates to requirements in the website weren't being incorporated. Checking the main site's Varnish logs in Kibana confirms that requests were not made from Junco's IP address to fetch those requirements. There must be some caching or other issue preventing the lookup from even happening.

Saving Archive config has Javascript error in Firefox

Saves works fine in Chrome, but are failing in Firefox.

I added a new courslist section to Arabic when I noticed this error.

Web UI not successfully exporting archives.

I'm seeing errors like this in the logs when triggering an archive export. I'm not seeing these same errors when exporting from the command line.

Mar 20 10:28:48 sealion catalog.middlebury.edu: [Fri Mar 20 10:28:48 2020] [error] [log-id XnTS5HeS@IPOHOlpfG@@7QAAAAM] [pid 58854] [client 140.233.106.176
:51884] [x-forwarded-for 140.233.106.176] [request GET catalog.middlebury.edu/archive/exportjob] AH01102: error reading status line from remote server loca
lhost:8080
Mar 20 10:28:48 sealion catalog.middlebury.edu: [Fri Mar 20 10:28:48 2020] [error] [log-id XnTS5HeS@IPOHOlpfG@@7QAAAAM] [pid 58854] [client 140.233.106.176
:51884] [x-forwarded-for 140.233.106.176] [request GET catalog.middlebury.edu/archive/exportjob] AH00898: Error reading from remote server returned by /arc
hive/exportjob


PHP Warning:  DOMDocument::loadHTML(): ID Old already defined in Entity, line: 1 in /var/www/coursecatalog/application/controllers/ArchiveController.php on
 line 621
PHP Warning:  Module 'iconv' already loaded in Unknown on line 0
PHP Warning:  Module 'json' already loaded in Unknown on line 0
Another job with PID 59039 is already running and has status: Printed section 51 of 163
Error running command:

        /var/www/coursecatalog/bin/zfcli.php  -a archive.generate -p 'config_id=1&term%5B0%5D=term%2F202090&term%5B1%5D=term%2F202120&revision_id=latest&ve
rbose=0' > /tmp/MCUG-2020-21_snapshot-2020-03-20.html

Upgrade from ZendFramework 1

The CourseCatalog currently uses an out-of-date version of ZendFramework. Either upgrade to a new version of ZF or migrate to Symfony.

Schedule view not rendering -- javascript errors

https://webhelpdesk.middlebury.edu/helpdesk/WebObjects/Helpdesk.woa/wa/TicketActions/view?ticket=550077

Clicking on a schedule causes the browser to generate Javascript errors and open the raw JSON data rather than displaying the rendered schedule in an overlay:

Similarly, after clicking the "Print" button, the rendered weekly schedule doesn't display, likely due to the same issue.

Catalog export for 2024-2025 failing

https://support.gmhec.org/TDNext/Apps/49/Tickets/TicketDet.aspx?TicketID=145681

Running the export for 2024-2025 through https://catalog.middlebury.edu/admin/schedule is failing.

Clicking on the link to the export produces this error:

[08-Mar-2024 14:49:49 America/New_York] PHP Warning:  filesize(): stat failed for /var/www/archive-tmp/MCUG-2024-2025_snapshot-2024-03-08.html in /var/www/coursecatalog/application/controllers/helper/ExportJob.php on line 93
[08-Mar-2024 14:49:49 America/New_York] PHP Warning:  unlink(/var/www/archive-tmp/MCUG-2024-2025_snapshot-2024-03-08.html): No such file or directory in /var/www/coursecatalog/application/controllers/helper/ExportJob.php on line 95

The path /var/www/archive/MCUG/2024-2025/html is empty as is /var/www/archive-tmp/

Unable to e-mail schedule

Clicking on the e-mail link in My Schedule Planner results in a 503 error.

Review the Course Catalog server logs for errors, bugs, and performance issues

Check the Course Catalog logs and enter as many GitHub issues based on them as you can in a day.

Update OCI sync for OCI8 & PHP8

I just remembered that the Course Catalog already has it's own OCI support for copying tables directly from Banner, but we stopped using that in favor of the script on Zengarden because Dave had so much trouble custom-compiling PHP with OCI support in older versions of PHP. This should be pretty trivial to update to OCI8 if the oracle libraries get installed on the catalog server. I'll follow up with Dave about that next week and maybe we can drop the scripts on Zengarden and not need to rewrite them elsewhere.

Text encoding causing sync issues

Currently https://catalog.middlebury.edu/offerings/view/catalog/catalog%2FMIIS/offering/section%2F202298%2F91790 has some Chinese characters coming through corrupted.

Fixing the character encoding of the source data is then resulting in errors inserting this section's description into MariaDB:

Updating SSBDESC        Insert Error: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: 
'\xE7\x99\xBD\xE5\xBB\xBA...' for column `afranco_catalog`.`SSBDESC`.`SSBDESC_TEXT_NARRATIVE` at row 1. 

Query: 
INSERT INTO SSBDESC (SSBDESC_TERM_CODE, SSBDESC_CRN, SSBDESC_TEXT_NARRATIVE, SSBDESC_ACTIVITY_DATE, SSBDESC_USER_ID) 
VALUES (:SSBDESC_TERM_CODE, :SSBDESC_CRN, :SSBDESC_TEXT_NARRATIVE, :SSBDESC_ACTIVITY_DATE, :SSBDESC_USER_ID); 

SQL: [225] 
INSERT INTO SSBDESC (SSBDESC_TERM_CODE, SSBDESC_CRN, SSBDESC_TEXT_NARRATIVE, SSBDESC_ACTIVITY_DATE, SSBDESC_USER_ID) 
VALUES (:SSBDESC_TERM_CODE, :SSBDESC_CRN, :SSBDESC_TEXT_NARRATIVE, :SSBDESC_ACTIVITY_DATE, :SSBDESC_USER_ID); 

Sent SQL: [2340] 
INSERT INTO SSBDESC (SSBDESC_TERM_CODE, SSBDESC_CRN, SSBDESC_TEXT_NARRATIVE, SSBDESC_ACTIVITY_DATE, SSBDESC_USER_ID) 
VALUES ('202298', '91790', 'Chinese Grammar Pedagogy is an introduction to Chinese grammar pedagogy that focuses on structured grammar input, pedagogical methods in instructing Chinese grammar, discourse-based approach, types of grammar elicitation designs, and interaction among grammar, context, and pragmatics. It is designed to integrate theories into hands-on practice. The primary goals of this course include gaining insights from readings of books, Chinese pedagogical grammar articles, discussing grammar teachability issues from elementary to advanced levels, writing reaction journals and reflection, training grammar elicitation techniques, and compiling Chinese grammar from authentic content-based materials. The course will also integrate technology, be it Moodle, AdobeConnect, blogging, etc., into the classroom and invite K-16 Chinese linguists and pedagogues in this field through virtual Elluminate web-conferencing tool and recorded presentation, e.g. Professor Jianhua Bai 白建華, Professor Paul Jia, etc. The course will embrace the Monterey Way, be conducted using content-based instruction, be taught in Chinese and use reading materials in English and Chinese. All the assignments except grammar explanation should be written in Chinese. Students have to choose textbooks or authentic materials for their semester project, and are expected to write reaction journals (one article of your choice from the week’s reading), reflection entries (based on what you have learned with your teacher, peers, and online lecturers, etc.), and conduct a 15-20 min rehearsal and real-class teaching demos. The class is a 3+1 combo, which means three-hour classroom time plus one-hour online portion. Meetings during every Thursday evenings 6-8:50 p.m. and make-up classes on some Friday evenings 5:00-7:50 p.m. before Week 8, and you start to choose or develop your own approach to design your lesson plan during week 8 to week 10. We will meet online again in week 11, and resume our classes during week 13. Our teaching demos in week 13, final presentation is in week 15, and your semester project is due on December 10th, 2:00 p.m. (Submit both Paper and e-copy).<p>', '2022-04-01 20:07:02', 'PVENTURA'); 

Params:  5; Key: Name: [18] :SSBDESC_TERM_CODE; paramno=-1; name=[18] ":SSBDESC_TERM_CODE"; is_param=1; param_type=2; Key: Name: [12] :SSBDESC_CRN; paramno=-1; name=[12] ":SSBDESC_CRN"; is_param=1; param_type=2; Key: Name: [23] :SSBDESC_TEXT_NARRATIVE; paramno=-1; name=[23] ":SSBDESC_TEXT_NARRATIVE"; is_param=1; param_type=2; Key: Name: [22] :SSBDESC_ACTIVITY_DATE; paramno=-1; name=[22] ":SSBDESC_ACTIVITY_DATE"; is_param=1; param_type=2; Key: Name: [16] :SSBDESC_USER_ID; paramno=-1; name=[16] ":SSBDESC_USER_ID"; is_param=1; param_type=2;
#0 /home/afranco/private_html/CourseCatalog/catalog/application/library/CatalogSync/Syncer/Abstract.php(645): CatalogSync_Database_Statement_Insert_Pdo->insertAll(Object(CatalogSync_Database_Statement_Select_PdoMysql))
#1 /home/afranco/private_html/CourseCatalog/catalog/application/library/CatalogSync/Director.php(52): CatalogSync_Syncer_Abstract->copy()
#2 /home/afranco/private_html/CourseCatalog/catalog/bin/update-from-banner.php(30): CatalogSync_Director->sync()
#3 {main}

The part likely causing the issue:

e.g. Professor Jianhua Bai 白建華, Professor Paul Jia, etc

This is likely an issue with either the destination column encoding, the connection encoding, or both.

Export Config: Separate H1 into two distinct fields

Currently the H1 (and H2) has a single entry field with a delimiter for the two pieces of data, the full title and the table-of-contents (TOC) title:

This means that formatting instructions are necessary and the Group shows the full string, both adding to visual clutter in the UI.

Separate the data into two text fields, full title first and TOC text second.
Drop the formatting instructions
Title field can have placeholder text like "Full title for section heading".
TOC text can have placeholder text like "Short title for table-of-contents".
Change the group title to use only the TOC text, not the full title, for easier scanning of the full config.
Above changes for H2 as well.

Create New Job should have some help text

These can be on multiple lines if that helps.

The Terms field should either have placeholder text or a note saying that it is a comma-delimited list of term-codes with an example.
Since we know the Catalog ID, we can prompt with the last portion of that as a prefix for the path, even if the user might erase it.

Limit facets to just those for selected term

The Language Schools search form includes campuses (e.g. Mexico) that are no longer in use. When a term is selected, show just those options for the selected term.

Export Config History - Back link should go to previous page

Currently the bread-crumb link goes back to the /admin/export:

It would be better if it went back to the screen the user just came from, which is the details of a particular configuration like: /admin/export?config=....

Data-truncation on sync in SSBXLST_XLST_GROUP

The following errors occurred during database synchronization:

SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'SSBXLST_XLST_GROUP' at row 1

#0 /var/www/coursecatalog/application/library/CatalogSync/Database/Statement/Insert/Pdo.php(55): PDOStatement->execute()
#1 /var/www/coursecatalog/application/library/CatalogSync/Syncer/Abstract.php(581): CatalogSync_Database_Statement_Insert_Pdo->insertAll(Object(CatalogSync_Database_Statement_Select_PdoMysql))
#2 /var/www/coursecatalog/application/library/CatalogSync/Director.php(52): CatalogSync_Syncer_Abstract->copy()
#3 /var/www/coursecatalog/bin/update-from-banner.php(30): CatalogSync_Director->sync()
#4 {main}

Most likely the Banner database has had a column width expanded over the years and we need to expand the SSBXLST_XLST_GROUP table in our database to match.

Create README and install instructions

We need a README that describes the system and has install instructions.

Add per-section attributes to non-primary cross-lists

From WHD 534481:

The only thing that we noted in this round was that for cross-listed courses, if only some of the seats are CW seats, this shows correctly on the primary listing but not on the secondary listing. On the secondary listing, it says CW but does not indicate the number of seats. An example of this would be ECON/GSFS 0207. We think this happening because the number of CW seats is calculated by looking down into the section seats, and the secondary listing has no seats in it.

Our take on this is that if there is some super easy and obvious way to make it so that the total seats shows correctly on the secondary cross-list, then great. But if not, we will let it go and are happy to call the project done (!!!!!!!!!). You can just let us know what you think will work best.

PHP8: Requests containing multiple values for the same query parameter parsed by Zend framework as array.

Curated Course Lists in Drupal send a request to the Catalog with each course as a separate value of the "id" parameter, like this:

https://catalog.middlebury.edu/courses/byidxml/catalog/catalog%2FMCUG/id/course%2FECON0211/id/course%2FECON0222/id/course%2FECON0224/id/course%2FECON0225/id/course%2FECON0228/id/course%2FECON0229/id/course%2FECON0232/id/course%2FECON0234/id/course%2FECON0265/id/course%2FECON0328/id/course%2FECON0329/id/course%2FECON0344/id/course%2FECON0352/id/course%2FECON0365/id/course%2FECON0401/id/course%2FECON0405/id/course%2FECON0415/id/course%2FECON0418/id/course%2FECON0420/id/course%2FECON0425/id/course%2FECON0427/id/course%2FECON0428/id/course%2FECON0429/id/course%2FECON0430/id/course%2FECON0431/id/course%2FECON0444/id/course%2FECON0445/id/course%2FECON0448/id/course%2FECON0453/id/course%2FECON0454/id/course%2FECON0465/id/course%2FECON0466/id/course%2FECON0475/id/course%2FECON0480/id/course%2FECON1023/id/course%2FECON1028/id/course%2FINTD1208

This generates the following PHP error:

TypeError (0) with message urlencode(): Argument #1 ($string) must be of type string, array given

And stack trace:

#	File	Line	Call
0	Url.php	74
1	Url.php	74	urlencode(Array: 37 elements);
2	Url.php	116	Zend_Controller_Action_Helper_Url->simple(String: "courses", String: "byidxml", String: "default", Array: 3 elements);
3	HelperBroker.php	328	Zend_Controller_Action_Helper_Url->direct(String: "courses", String: "byidxml", NULL, Array: 3 elements);
4	CoursesController.php	389	Zend_Controller_Action_HelperBroker->__call(String: "url", Array: 4 elements);
5	Action.php	513	CoursesController->byidxmlAction();
6	Standard.php	295	Zend_Controller_Action->dispatch(String: "byidxmlAction");
7	Front.php	954	Zend_Controller_Dispatcher_Standard->dispatch(Object: Zend_Controller_Request_Http, Object: Zend_Controller_Response_Http);
8	Front.php	212	Zend_Controller_Front->dispatch();
9	index.php	57	Zend_Controller_Front::run(String: "/home/imcbride/private_html/CourseCatalog/catalog/application/controllers");

The referenced line of Url.php does this:

            foreach ($params as $key => $value) {
				$paramPairs[] = urlencode($key) . '/' . (is_null($value) ? '' : urlencode($value));
            }

Dumping $params gives:

array(3) {
  ["catalog"]=>
  string(12) "catalog/MCUG"
  ["id"]=>
  array(37) {
    [0]=>
    string(15) "course/ECON0211"
    [1]=>
    string(15) "course/ECON0222"
    [2]=>
    string(15) "course/ECON0224"
    [3]=>
    string(15) "course/ECON0225"
    [4]=>
    string(15) "course/ECON0228"
    [5]=>
    string(15) "course/ECON0229"
    [6]=>
    string(15) "course/ECON0232"
    [7]=>
    string(15) "course/ECON0234"
    [8]=>
    string(15) "course/ECON0265"
    [9]=>
    string(15) "course/ECON0328"
    [10]=>
    string(15) "course/ECON0329"
    [11]=>
    string(15) "course/ECON0344"
    [12]=>
    string(15) "course/ECON0352"
    [13]=>
    string(15) "course/ECON0365"
    [14]=>
    string(15) "course/ECON0401"
    [15]=>
    string(15) "course/ECON0405"
    [16]=>
    string(15) "course/ECON0415"
    [17]=>
    string(15) "course/ECON0418"
    [18]=>
    string(15) "course/ECON0420"
    [19]=>
    string(15) "course/ECON0425"
    [20]=>
    string(15) "course/ECON0427"
    [21]=>
    string(15) "course/ECON0428"
    [22]=>
    string(15) "course/ECON0429"
    [23]=>
    string(15) "course/ECON0430"
    [24]=>
    string(15) "course/ECON0431"
    [25]=>
    string(15) "course/ECON0444"
    [26]=>
    string(15) "course/ECON0445"
    [27]=>
    string(15) "course/ECON0448"
    [28]=>
    string(15) "course/ECON0453"
    [29]=>
    string(15) "course/ECON0454"
    [30]=>
    string(15) "course/ECON0465"
    [31]=>
    string(15) "course/ECON0466"
    [32]=>
    string(15) "course/ECON0475"
    [33]=>
    string(15) "course/ECON0480"
    [34]=>
    string(15) "course/ECON1023"
    [35]=>
    string(15) "course/ECON1028"
    [36]=>
    string(15) "course/INTD1208"
  }
  ["cuttoff"]=>
  NULL
}

Since $value is an array while $key is id, this generates the error in passage to urlencode().

I've worked around this by sending multiple requests to the catalog, one per course, in https://github.com/middlebury/drupal8/commit/0b516f691953a16544452e68217580f8a9049005, but it would be better to figure out how to have the Catalog accept these requests as it used to do. However, changing the behavior of CoursesController.php before it passes the URL to the Zend framework to interpret has the chance to break other things that use this API so caution is warranted.

Review the Course Catalog server logs for errors, bugs, and performance issues

Check the Course Catalog logs and enter as many GitHub issues based on them as you can in a day.

Upgrade to PHP 7.1

Fix apc_*() calls
Deploy

PHP8: Session cookies being set on anonymous requests

In PHP 7.4 we are successfully preventing PHPSESSID cookies from being set when there is no attempt at login.

[afranco@toboggan catalog (master)]$ curl -I https://saw.middlebury.edu/~afranco/catalog/
HTTP/1.1 200 OK
Date: Thu, 01 Dec 2022 15:29:02 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.4.33
X-Powered-By: PHP/7.4.33
Expires: Thu, 01 Dec 2022 16:29:02 GMT
Cache-Control: public
Pragma: 
X-Runtime: 0.323407
Cache-Control: max-age=3600
Vary: Cookie,Accept-Encoding
X-Runtime: 0.327753
Content-Type: text/html; charset=UTF-8

In PHP8 we're seeing sessions being started even in the first page load, which will prevent Varnish from doing any useful caching.

[afranco@toboggan catalog (master)]$ curl -I https://toboggan.middlebury.edu/~afranco/catalog/
HTTP/1.1 200 OK
Date: Thu, 01 Dec 2022 15:30:39 GMT
Server: Apache/2.4.37 (AlmaLinux) OpenSSL/1.1.1k
X-Powered-By: PHP/8.1.13
X-Runtime: 0.022981, 0.024046
Expires: Thu, 01 Dec 2022 16:30:39 GMT
Cache-Control: public, max-age=3600
Pragma: 
Vary: Cookie,Accept-Encoding
Set-Cookie: PHPSESSID=6dda8da1ad5254c9e5e1f8d25e49c1c3; path=/
Content-Type: text/html; charset=UTF-8

Looking at that actual session data it seems that this might be caused by phpCAS creating an session key with an empty array:

[root@toboggan afranco]# cat /var/lib/php/session/sess_6dda8da1ad5254c9e5e1f8d25e49c1c3 
phpCAS|a:0:{}

Clean messy HTML in course descriptions

The course catalog currently doesn't touch HTML tags in descriptions, but these may be problematic for downstream data consumers that expect valid markup.

Here is an example of a course description that is causing problems:

<BR><B>Security and Development in the Francophone World</B><p>
In this course, we will examine two major inter-connected themes of urgency and relevance in the contemporary Francophone world: security and development.
We’ll research and discuss complex security concerns in a rapidly changing political environment – both within France and in the wider geographical area of French influence. We will also use selected case studies to address issues of development throughout the Francophone world - including debt, sustainable economy, trade, science/technology, and education.Finally, we will explore connections between these two focus areas. <p>Students will improve all key language skills through activities such as discussion, use of authentic audiovisual and social media sources, creative projects, debates, guest speaker interactions and oral presentations. <p><B> Key Topics: Security</B> <p>

Border Control: Issues of Humanitarian Relief and National Security (illegal immigration, refugees, arms smuggling, human trafficking, etc.)<p>
Islamophobia and Security after 9/11<p>
French response to international terrorism, defense policies and collaboration within the European Union<p>
Food, land and water security in an era of climate change
<p><B> Key Topics: Development</B> <p>
Roles of the International Monetary Fund, NGOs, private and corporate sectors and World Bank in national/regional economies<p>
Economic initiatives such as microfinance and start-ups<p>
Women’s empowerment through education, training and family planning<p>
Fair trade and sustainable agriculture<p>
Science/technology <p>

convert <b> to <strong>
convert <i> to <em>
close paragraphs before starting new paragraphs
ensure that all tags are closed.

FULLTEXT searches cause excessively slow queries

When using the Keyword field on the search form a query is built that includes a MATCH (SSBSECT_fulltext) AGAINST ('<search keyword>' IN BOOLEAN MODE) AS relevence,. While this is valid syntax, these queries are taking up to 15-30 seconds in FULLTEXT initialization state and heavy use can subsequently impact database load and overall system performance.

Example query searching for biology:
https://catalog.middlebury.edu/offerings/search/catalog/catalog%2FMCUG?search=Search&term=term%2F201920&department=&keywords=biology&time_start=0&time_end=86400&type%5B%5D=genera%3Aoffering%2FLCT&type%5B%5D=genera%3Aoffering%2FSEM

SELECT
SSBSECT_TERM_CODE,
SSBSECT_CRN,
SSBSECT_SUBJ_CODE,
SSBSECT_CRSE_NUMB,
SSBSECT_SEQ_NUMB,
SSBSECT_PTRM_CODE,
SSBSECT_CRSE_TITLE,
SSBSECT_MAX_ENRL,
SSBSECT_ENRL,
SSBSECT_SEATS_AVAIL,
SSBSECT_LINK_IDENT,
SSBDESC_TEXT_NARRATIVE,
SCBCRSE_TITLE,
SCBDESC_TEXT_NARRATIVE,
SSBSECT_CAMP_CODE,
term_display_label,
STVTERM_START_DATE,
STVSCHD_CODE,
STVSCHD_DESC,
SSRMEET_BLDG_CODE,
SSRMEET_ROOM_CODE,
SSRMEET_BEGIN_TIME,
SSRMEET_END_TIME,
SSRMEET_SUN_DAY,
SSRMEET_MON_DAY,
SSRMEET_TUE_DAY,
SSRMEET_WED_DAY,
SSRMEET_THU_DAY,
SSRMEET_FRI_DAY,
SSRMEET_SAT_DAY,
SSRMEET_START_DATE,
SSRMEET_END_DATE,
COUNT(SSRMEET_TERM_CODE) as num_meet,
STVBLDG_DESC,
STVCAMP_DESC,
SCBCRSE_EFF_TERM ,
SCBCRSE_DEPT_CODE,
SCBCRSE_DIVS_CODE,
MATCH (SSBSECT_fulltext) AGAINST ('biology' IN BOOLEAN MODE) AS relevence,
SSRXLST_XLST_GROUP
FROM
ssbsect_scbcrse_scbdesc
INNER JOIN catalog_term ON SSBSECT_TERM_CODE = catalog_term.term_code
INNER JOIN course_catalog_college ON course_catalog_college.coll_code = SCBCRSE_COLL_CODE
INNER JOIN course_catalog ON course_catalog_college.catalog_id = course_catalog.catalog_id
LEFT JOIN STVTERM ON SSBSECT_TERM_CODE = STVTERM_CODE
LEFT JOIN SSBDESC ON (SSBSECT_TERM_CODE = SSBDESC_TERM_CODE AND SSBSECT_CRN = SSBDESC_CRN)
LEFT JOIN SSRMEET ON (SSBSECT_TERM_CODE = SSRMEET_TERM_CODE AND SSBSECT_CRN = SSRMEET_CRN)
LEFT JOIN STVBLDG ON SSRMEET_BLDG_CODE = STVBLDG_CODE
LEFT JOIN STVSCHD ON SSBSECT_SCHD_CODE = STVSCHD_CODE
LEFT JOIN STVCAMP ON SSBSECT_CAMP_CODE = STVCAMP_CODE
LEFT JOIN SSRXLST ON (SSBSECT_TERM_CODE = SSRXLST_TERM_CODE AND SSBSECT_CRN = SSRXLST_CRN)

WHERE
(SSBSECT_TERM_CODE = '201920')
AND (SSBSECT_SCHD_CODE = 'LCT'
OR SSBSECT_SCHD_CODE = 'SEM')
AND (MATCH (SSBSECT_fulltext) AGAINST ('biology' IN BOOLEAN MODE))
AND SSBSECT_TERM_CODE IN (
SELECT
term_code
FROM
catalog_term
WHERE
catalog_id = 'MCUG'
)
AND SCBCRSE_COLL_CODE IN (
SELECT
coll_code
FROM
course_catalog_college
WHERE
catalog_id = 'MCUG'
)
AND SSBSECT_SSTS_CODE = 'A'
AND (course_catalog.prnt_ind_to_exclude IS NULL OR SSBSECT_PRNT_IND != course_catalog.prnt_ind_to_exclude)

GROUP BY SSBSECT_TERM_CODE, SSBSECT_CRN
ORDER BY relevence DESC

Hopefully queries relying on keyword searching can be reworked to perform better. A few ideas:

Move the SSBSECT_fulltext column to its own table (this may allow the full-text search to run using its own index and then allow it to be joined using shared keys to our SSBSECT and maybe be more efficient).
Investigate optimizing this index.

Fix symlink issue resulting in broken catalog export

Somehow, MCUG/2018-2019/MCUG-2018-2019_latest.html became a symlink to MCUG/2018-2019/html/MCUG-2018-2019_snapshot-2018-04-20.html, which itself
was a snapshot to MCUG/2018-2019/html/html/MCUG-2018-2019_snapshot-2018-04-20.html.

Not only was the filepath invalid, but that second symlink should not have been created in the first place. Rather, it should have been an actual file.

Archive config has wrong path for drag/drop images

This is the image path currently used:
https://catalog.middlebury.edu/admin/images/arrow_cross.png
Should be
https://catalog.middlebury.edu/images/arrow_cross.png

Review the Course Catalog server logs for errors, bugs, and performance issues

Check the Course Catalog logs and enter as many GitHub issues based on them as you can in a day.

Update unit tests to support phpunit 10

Tests seem to be written for phpunit < 6 which no longer works on new versions of PHP. See: https://phpunit.de/supported-versions.html

Updating unit tests to work with new versions of phpunit will help ensure that nothing breaks as code gets refactored.

Progress bar for exporting through the UI

Add archive-job admin screen

This is a subsequent feature following on to Kanban#4650, where we have the configuration-sets stored in the database. Even with #4650, we still have to the export cron jobs on the server via chef.

A workflow improvement would be to add an admin UI that allows the registrar team to define these export jobs and whether or not they are currently active (exporting nightly). The cron job would then export all active jobs each night

Here is a mock-up of such a screen:

Configuration components for each job:

Is active? -- if true, the job will be run each night. Otherwise it will be skipped. Rather than trying to configure start/end dates, it is probably more straight-forward to just manually change this value. The big reason to turn off a job after the semester is over is to do so before changing configs or other data, which can all be done together.
Export path -- A directory to export into. This should allow subdirectories of the archive base to be defined, but prevent .. or directory traversal.
Configuration Set -- which of our defined configuration sets to use for this job. Should supply a drop-down list. Because the configuration set defines which catalog to use, changing this would potentially invalidate the terms available.
Configuration Version -- Defaults to "Latest". By allowing a particular configuration version to be chosen, exports for a current year can still be run while new configuration is changed and edited for an upcoming year (e.g. new program added or removed for the upcoming year).
Terms -- Which terms to use for this export job. These will be terms associated with the catalog defined in the configuration set. One or more are possible, so the UI will need to allow some mechanism for multi-select. If the Catalog associated with the configuration set (or the config set) changes, then the terms should show an error (at least on activation) until they are updated.

Additionally, it would be useful to provide a button to manually trigger a particular export from the UI. This would allow testing of configuration during its setup. As the export may take many minutes, a progress bar should be shown by dividing the number of sections by 100 to get a percentage. Partial content (the status ticker) will need to be flushed periodically to prevent time-outs on the long operation. This could be displayed in a JS overlay or it could be a full-screen ticker that just shows the response as it comes in. The export should also clear the varnish-cache for the particular archive-path only subsequent to the regeneration.

Review the Course Catalog server logs for errors, bugs, and performance issues

Check the Course Catalog logs and enter as many GitHub issues based on them as you can in a day.

Poor query performance in Instructor section lists

Similar to #27, requests that load all of an instructor's sections for all terms result in generally poor performance with the creation of on-disk temporary tables. These generally return in 10-20 seconds when the database is not particularly loaded.

SELECT
SSBSECT_TERM_CODE,
SSBSECT_CRN,
SSBSECT_SUBJ_CODE,
SSBSECT_CRSE_NUMB,
SSBSECT_SEQ_NUMB,
SSBSECT_PTRM_CODE,
SSBSECT_CRSE_TITLE,
SSBSECT_MAX_ENRL,
SSBSECT_ENRL,
SSBSECT_SEATS_AVAIL,
SSBSECT_LINK_IDENT,
SSBDESC_TEXT_NARRATIVE,
SCBCRSE_TITLE,
SCBDESC_TEXT_NARRATIVE,
SSBSECT_CAMP_CODE,
term_display_label,
STVTERM_START_DATE,
STVSCHD_CODE,
STVSCHD_DESC,
SSRMEET_BLDG_CODE,
SSRMEET_ROOM_CODE,
SSRMEET_BEGIN_TIME,
SSRMEET_END_TIME,
SSRMEET_SUN_DAY,
SSRMEET_MON_DAY,
SSRMEET_TUE_DAY,
SSRMEET_WED_DAY,
SSRMEET_THU_DAY,
SSRMEET_FRI_DAY,
SSRMEET_SAT_DAY,
SSRMEET_START_DATE,
SSRMEET_END_DATE,
COUNT(SSRMEET_TERM_CODE) as num_meet,
STVBLDG_DESC,
STVCAMP_DESC,
SCBCRSE_EFF_TERM ,
SCBCRSE_DEPT_CODE,
SCBCRSE_DIVS_CODE,
SSRXLST_XLST_GROUP
FROM
ssbsect_scbcrse_scbdesc
INNER JOIN catalog_term ON SSBSECT_TERM_CODE = catalog_term.term_code
INNER JOIN course_catalog_college ON course_catalog_college.coll_code = SCBCRSE_COLL_CODE
INNER JOIN course_catalog ON course_catalog_college.catalog_id = course_catalog.catalog_id
LEFT JOIN STVTERM ON SSBSECT_TERM_CODE = STVTERM_CODE
LEFT JOIN SSBDESC ON (SSBSECT_TERM_CODE = SSBDESC_TERM_CODE AND SSBSECT_CRN = SSBDESC_CRN)
LEFT JOIN SSRMEET ON (SSBSECT_TERM_CODE = SSRMEET_TERM_CODE AND SSBSECT_CRN = SSRMEET_CRN)
LEFT JOIN STVBLDG ON SSRMEET_BLDG_CODE = STVBLDG_CODE
LEFT JOIN STVSCHD ON SSBSECT_SCHD_CODE = STVSCHD_CODE
LEFT JOIN STVCAMP ON SSBSECT_CAMP_CODE = STVCAMP_CODE
LEFT JOIN SSRXLST ON (SSBSECT_TERM_CODE = SSRXLST_TERM_CODE AND SSBSECT_CRN = SSRXLST_CRN)
LEFT JOIN SYVINST ON (SYVINST_TERM_CODE = SSBSECT_TERM_CODE AND SYVINST_CRN = SSBSECT_CRN)
WHERE
(WEB_ID = 'E2450BA8979164270943B3DF04520BB1')
AND SSBSECT_TERM_CODE IN (
SELECT
term_code
FROM
catalog_term
WHERE
TRUE
)
AND SCBCRSE_COLL_CODE IN (
SELECT
coll_code
FROM
course_catalog_college
WHERE
TRUE
)
AND SSBSECT_SSTS_CODE = 'A'
AND (course_catalog.prnt_ind_to_exclude IS NULL OR SSBSECT_PRNT_IND != course_catalog.prnt_ind_to_exclude)

GROUP BY SSBSECT_TERM_CODE, SSBSECT_CRN
ORDER BY SSBSECT_TERM_CODE DESC, SSBSECT_SUBJ_CODE, SSBSECT_CRSE_NUMB, SSBSECT_PTRM_CODE, SSBSECT_SEQ_NUMB

Profiling these queries indicates that there is file-sort and no index used on the catalog_term table, but adding additional indices didn't seem to help in initial testing, possibly due to the way subqueries are built.

A few initial ideas:

maybe rework subqueries as joins.
add/change indices to match query usage.
possibly require/provide a limited range of terms to prevent quite as large data-sets.

Provide listing of courses by itemized ids

In support of middlebury/drupal8#1783

Needs the same format as /courses/topicXml (including which terms taught).

Instead of a search selection, an itemized list of course ids will be passed.

Should support the cutoff parameter for the period of time to include results from.

Improve time-based search form options

Via TDX 167734

Fix time check-boxes

In the searchable catalog
https://catalog.middlebury.edu/catalogs/view/catalog/catalog%2FMCUG/term/term%2F202490
the search boxes for "Time" have 30-minute increments, but 12:00 am and 12:30 am are listed twice (midnight and also between 11am and 1pm). There is no listing for 12:00 pm and 12:30 pm.

I know this sounds so picky, could this be fixed...?

This came up for me because I'm looking for FYSE courses that conflict with our intro physics courses, so I wanted to be able to figure out possible conflicts.

Add note about OR search of days of week

Also, in the section for selecting days of the week, could a note be added saying that selecting only one limits the search to courses on that day exclusively? I think I noticed that 1 day is a stricter limit than I expected (i.e. Select W only --> results are limited to W only, excluding MW or MWF), while selecting 2+ days yields results that include either and/or both (i.e. Select M,W –> results include M only, W only, MW two sessions weekly, and MWF three sessions weekly).

@afranco a hub UX question for you - a prof was concerned about the presentation of course schedule info on her hub dashboard. See: https://courses.middlebury.edu/hub/miis/202028/icco/8535a

The schedule presents meeting days and times in reverse order, and also seems to favor the day of the week over month and day. Not sure what the sort order is, but imagine it could be confusing to some who might rely on hub for this info, at start of classes.

In Banner it looks like this:

I can file a ticket, she also indicates that the data for ICCO is correct, but the cross-listed site for the same course is in-correct: https://courses.middlebury.edu/hub/miis/202028/educ/9535a

So, there's a combination of issues. UX and correct data. Banner info for EDUC 9535 is correct.

Adam Franco 2:22 PM

Thanks for reporting this, Bob. That is certainly a complex case. That string is being generated here:
https://github.com/middlebury/coursecatalog/blob/master/application/library/banner/course/CourseOffering.php#L480-L526
And here for loading the entries: https://github.com/middlebury/coursecatalog/blob/master/application/library/banner/course/CourseOffering/AbstractSession.php#L268-L310
Right now these aren’t ordered, but that can change.

Archive: CW attributes not showing for some courses

I looked through the catalog and compared to the course schedule and the new process didn't seem to work for all. I see it did work for DANC/ENVS 0277, but found it did not work for a few: ENAM 0373, CHNS 0331, ITAL 0490, HIST 0222. There were others but I am hoping this is a good sampling for you to look at.
Kathleen

Nightly archive exports are hanging and not completing

https://support.gmhec.org/TDNext/Apps/49/Tickets/TicketDet.aspx?TicketID=31773

middlebury / coursecatalog Goto Github PK

coursecatalog's People

Contributors

Stargazers

Watchers

Forkers

coursecatalog's Issues

Recommend Projects

Recommend Topics

Recommend Org