ap7t/lecture-scraper

Lecture Scraper

lecture-scraper

A program to download and organise lecture notes

Setup

• Download/clone this repository
• Ensure Google Chrome is installed
• Download chromedriver https://chromedriver.chromium.org/downloads
  • Unzip and move the chromedriver file the the lecture-scraper-master directory

  • lecture-scraper-master
    ├── README.md
    ├── chromedriver
    ├── lecture_scraper.py
    └── requirements.txt
    

• Open the terminal/command line and cd into the lecture-scraper-master directory and run the following command to install dependencies

  • pip install -r requirements.txt
    

  • if pip not installed https://pip.pypa.io/en/stable/installing/

Usage

python3 lecture_scraper.py

lecture-scraper's Issues

Deal with dead-ends faster

In lecture-scraper_physics, one often lands in dead-ends (where there are no download links). It takes about 12 seconds for the program to move onto the next screen in those cases.

When there is a download button, the program can move on within .5 seconds.

Can the dead-end case be made faster?

Download report

I propose that the user gets a report at the end: any files / pages that were skipped are specified in the end, and files that were sucessfully downloaded are also specified. This helps the user check which downloads were successful or not. Useful in the case where there are panopto videos that cannot be downloaded: it makes it very clear which videos were not downloaded.

User configurations

Perhaps allow users to save module configurations, so rather than having to edit the source code, their preferences are stored in an external JSON file?