Giter Site home page Giter Site logo

paljsingh / impartus-downloader Goto Github PK

View Code? Open in Web Editor NEW
22.0 4.0 8.0 27.5 MB

Download Impartus lectures, convert to mkv for offline viewing.

License: GNU General Public License v3.0

Python 100.00%
impartus mkv lecture ilc bits ffmpeg impartus-downloader lecture-slides scraper pilani

impartus-downloader's Introduction

Impartus Downloader (Desktop App)

For Android version - click https://github.com/paljsingh/Impartus-Downloader-Android


Downloader for impartus streaming videos.

  • Tested on Mac OSX, Linux (ubuntu) and Windows 10.

Feature list

  • Convert impartus lectures to mkv files.
  • Regular and flipped lecture download supported.
  • Supports multi-track files. Tested with video lectures having up to 4 tracks.
  • Backpack slides download supported.
  • Lecture chats overlayed as closed captions for the video.
  • A command line interface.

UI

Videos Impartus Downloader - Video Tab

Documents Impartus Downloader - Documents Tab

Demo video

[TBA]

Prerequisites

python >= v3.7 ffmpeg >= v4.3

Installation / Run

OSX (10.14+)

# setup virtualenv (optional)
$ virtualenv venv
$ source venv/bin/activate 

# clone repo
$ git clone https://github.com/paljsingh/impartus-downloader.git
$ cd impartus-downloader

# install dependencies
$ pip3 install -r requirements.txt
$ brew install ffmpeg

# Ensure ffmpeg version is >= 4.3.0
$ ffmpeg -version 

# Run application
$ python3 App.py

Windows 10

The following steps shall work on Windows CMD prompt.

# clone repo
git.exe clone https://github.com/paljsingh/impartus-downloader.git
cd impartus-downloader

# setup virtualenv (optional, but recommended if you have Anaconda installed on the system)
pip.exe install virtualenv
virtualenv venv
call .\venv\Scripts\activate.bat

# install dependencies
pip.exe install -r requirements.txt

Download win64-gpl zip from the following link, extract and copy ffmpeg.exe to
impartus-downloader folder. The ffmpeg.exe statically bundles the dependent libs and
should be about 90 MB in size.
https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-win64-gpl.zip

# Ensure ffmpeg version is >= 4.3.0
ffmpeg.exe -version 

# Run application
python.exe App.py

Linux (Ubuntu 20+)

# setup virtualenv (optional)
$ virtualenv venv
$ source venv/bin/activate 

# clone repo
$ git clone https://github.com/paljsingh/impartus-downloader.git
$ cd impartus-downloader

# install dependencies
$ pip3 install -r requirements.txt
$ sudo apt-get install ffmpeg

# Ensure ffmpeg version is >= 4.3.0
$ ffmpeg -version 

# Run application
$ python3 App.py

Command Line


export IMPARTUS_USER='user@domain'
export IMPARTUS_PASS='password'

# login to impartus
$ python3 app-cli.py login

$ export IMPARTUS_TOKEN='token.from.login-command'

# save json info of subscribed subjects under json/subjects/
$ python3 app-cli.py subjects -o json/subjects/

# save lectures info for subject-1 under json/subject-1/lectures/
$ python3 app-cli.py lectures -j json/subjects/subject-1.json -o json/subject-1/lectures/

# save backpack documents info for subject-1 under json/subject-1/documents/
$ python3 app-cli.py documents -j json/subjects/subject-1.json -o json/subject-1/documents/

# download video for subject-1 / lecture-1, save it under data/videos/
$ python3 app-cli.py download video -j json/subject-1/lectures/lecture-1.json -o data/videos/

# download lecture chats for subject-1 / lecture-1, save it under data/videos/
$ python3 app-cli.py download chat -j json/subject-1/lectures/lecture-1.json -o data/videos/

# download backpack document for subject-1 / document-1, save it under data/documents/
$ python3 app-cli.py download document -j json/subject-1/documents/document-1.json -o data/documents/

Configuration

see etc/impartus.conf

Unit Tests

FIXME:

$ python3 -m pytest -v test


Known Issues

Hard to read closed captions on white background

With VLC, you can set the background opacity value to 255.

Go to Preferences.

Select 'All' settings

Navigate to Video > Subtitle / OSD > Text renderer

Change 'Background Opacity' to 255

Save and restart VLC.

Setting background opacity in VLC

Below is a sample output of the opacity change.

Background Opacity


System theme settings not picked up on Linux

On linux, PySide2 must be installed as system-wide package to access the currently selected theme settings. A pip based installation falls back to Fusion theme, and won't pick up the run time changes in the theme settings.

Steps needed in order to use system theme settings on linux.

remove pip Pyside2 installation:

$ pip3 uninstall PySide2

also remove any system wide Pyside2 pip install

$ sudo pip3 uninstall PySide2

Qt5ct is not needed, so just ensure the current envrionment does NOT have the Qt5ct variable

$ env | grep QT_QPA_PLATFORMTHEME

The above command should show empty result.

Install PySide2 and dependent components as system wide package.

$ sudo apt-get install python3-pyside2.qtuitools python3-pyside2.qtwidgets libpyside2-py3-5.14

Ensure to use PySide2 api, in case there also exists a Qt5/Qt4 installation,

$ echo 'export QT_API=PySide2' >> ~/.bashrc 
$ source ~/.bashrc
$ python3 App.py

alternatively, one can run the app as:

$ QT_API=PySide2 python3 App.py

Ref: PySimpleGUI/PySimpleGUI#2437


Error on application launch - "No module named Crypto"

The error may appear if 'crypto' package is installed on the system. crpyto is deperecated and no longer recommended due to various security issues.
Impartus application uses 'pycryptodome' that may conflict with 'crypto' package as they both provide the functions under 'Crypto' namespace.

To resolve the issue, uninstall crypto, then reinstall pycryptodome:

$ pip3 uninstall crypto
$ pip3 uninstall pycryptodome
$ pip3 install pycryptodome


Error on application launch - "ImportError: DLL load failed: The specified procedure could not be found."

On windows, this error may appear due to incompatibility between the PySide2 and python versions. Upgrading to python version 3.7 or above should resolve the issue.

You may also need to upgrade your pip version, in case pip3 list output shows PySide2 version other than 5.15.2

$ pip3 install --upgrade pip

If you still see the issue, you may try upgrading python version, at present the following python / windows version seem to have compatibility issues with PySide2 v5.15.2

Python v3.7.3, Windows 8.1 - https://bugreports.qt.io/browse/PYSIDE-1002

Python v3.9.0, Windows PE - https://bugreports.qt.io/browse/PYSIDE-1479


Drop a mail to [email protected] in case of any issues/errors.

impartus-downloader's People

Contributors

paljsingh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

impartus-downloader's Issues

[Bug]: Invalid format error while getting the epoch value

Describe the bug
strftime('%s') is deprecated and it throws the invalid format error on python 3.7.10.
To get the epochtime we can use timestamp() from datetimetime object.
https://github.com/paljsingh/impartus-downloader/blob/master/ui/content.py#L502

To Reproduce
Steps to reproduce the behavior:

  1. Run the App

Expected behavior
A clear and concise description of what you expected to happen.

Error Stack
Exception in thread Thread-7:
Traceback (most recent call last):
File "C:\Users<>\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users<>\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Users<>\Documents\impartus_downloader\idm\ui\content.py", line 533, in _download_video
self.save_captions_if_needed(video_metadata, root_url, captions_path)
File "C:\Users<>\Documents\impartus_downloader\idm\ui\content.py", line 502, in save_captions_if_needed
start_epoch = int(datetime.strptime(video_metadata['startTime'], date_format).strftime('%s'))
ValueError: Invalid format string

Desktop (please complete the following information):

  • OS: Windows 10
  • Python Version 3.7.10

Additional context
Add any other context about the problem here.

Use a better default content_font for windows / linux.

Tahoma font on windows makes the progress bar characters choppy and uneven, while it looks pretty good on OSX.
configuration file should have option to specify default content_font for each of the supported platforms, keeping the UI aesthetics in mind.

[Bug]: On move/rename, app only removes leaf level empty dir.

Describe the bug
It should also check the parent directory, and remove it if it is empty, do it recursively.

To Reproduce
Steps to reproduce the behavior:
1.
2.
3.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. Windows 10]
  • Python Version [e.g. 3.7]

Additional context
Add any other context about the problem here.

[Enhancement]: Option to open all the slides, when more than one present for a lecture.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Slides button on exe attachments throws error

Error response on console:

ERROR:Impartus:[4081252]: Error fetching slides from url: https://a.impartus.com/download1/installer/impartus/w64/Impartus-1.2.8-setup.exe and https://a.impartus.com/download1/installer/impartus/w32/Impartus-1.2.8-setup.exe
ERROR:Impartus:[4081252]: Http response code: 404, response body:

<title>Error</title>
Cannot GET /https://a.impartus.com/download1/installer/impartus/w64/Impartus-1.2.8-setup.exe%20and%20%20https://a.impartus.com/download1/installer/impartus/w32/Impartus-1.2.8-setup.exe

[Bug]: tksheet version issue

Describe the bug
new version of tksheet published on 11th June is breaking the module.

To Reproduce
Steps to reproduce the behavior:

  1. install dependencies
  2. Run the app

Screenshot
image

Error Stack
Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\<>\AppData\Local\Programs\Python\Python37\lib\tkinter\__init__.py", line 1705, in __call__ return self.func(*args) File "C:\Users\<>\Documents\impartus_downloader\idm\ui\content.py", line 900, in show_video_callback self.set_display_widgets() File "C:\Users\<>\Documents\impartus_downloader\idm\ui\content.py", line 145, in set_display_widgets self.fill_content() File "C:\Users\<>\Documents\impartus_downloader\idm\ui\content.py", line 460, in fill_content self._init_content() File "C:\Users\<>\Documents\impartus_downloader\idm\ui\content.py", line 98, in _init_content self.set_display_columns() File "C:\Users\<>\Documents\impartus_downloader\idm\ui\content.py", line 735, in set_display_columns self.sheet.display_columns(indexes=column_states, enable=True, redraw=False) File "C:\Users\<>\AppData\Local\Programs\Python\Python37\lib\site-packages\tksheet\_tksheet.py", line 2156, in display_columns return self.displayed_columns(indexes = indexes, enable = enable, reset_col_positions = reset_col_positions, set_col_positions = set_col_positions, refresh = refresh, redraw = redraw, deselect_all = deselect_all) File "C:\Users\<>\AppData\Local\Programs\Python\Python37\lib\site-packages\tksheet\_tksheet.py", line 2147, in displayed_columns deselect_all = deselect_all) File "C:\Users\<>\AppData\Local\Programs\Python\Python37\lib\site-packages\tksheet\_tksheet_main_table.py", line 3333, in display_columns if enable and list(range(len(max(self.data_ref, key = len)))) != self.displayed_columns: ValueError: max() arg is an empty sequence

Desktop (please complete the following information):

  • OS: Windows 10
  • Python Version : 3.7.10

Additional context
Add any other context about the problem here.

App shows video not downloaded, when the topic is changed

A video downloaded at the time when the topic was unset ends up with a file path as
$HOME/wilp/subject-profname/NN-No-Topic-Entered-yyyy-mm-dd.mkv

If at a later point the topic is updated, the application on a next relaunch, expects the downloaded video at the path:
$HOME/wilp/subject-prof-name/NN-New-Topic-Description-yyyy-mm-dd.mkv
Failing to find a file here, it shows the download status as 0%.

A similar case happens, if the user adds a new rule to map video being marked under Misc category to a new subject category,
or
if the video_path format is changed.
or
the video is manually moved to a new folder, or renamed.

[Bug]: ':' character not filtered in the file paths, on mac Finder.app it shows as '/' character.

OSX UI apps for historical reasons use ':' as file separator (since HFS days),
The posix based cli tools allow ':' character in the file name, but the same is shown in Finder.app as '/' character.

https://apple.stackexchange.com/questions/173529/when-did-the-colon-character-become-an-allowed-character-in-the-filesystem

Filter out ':' character in the filepath structure to keep the discrepancy away.

[enhancement] CLI for automated use cases.

A cli that supports options like:

  • all available and not downloaded videos (and slides)
  • date range to download the videos (and slides)
  • download videos (and slides) by subject (or subject code), professor (or professor code)

[Bug]: video size is 1Kb - empty video

Describe the bug
If the track is more than 1, then the video is empty, 1KB

To Reproduce
download any video with track more than 1, check the video size

Expected behavior
Video is expected to be full

Screenshots

image

image

Desktop (please complete the following information):

  • OS: mac os 10.15.7
  • Python Version 3.9.0

Try using Treeview for the table content.

Treeview though only supports label widgets, and buttons/progressbar cannot be a part of it,
may be adding a sub-frame can help here.
If not, the label widgets can be rendered via Treeview, which may also improve the usability and performance issues with the current table.

[Enhancement]: Add unit tests

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Unable to run successfully on windows

OS.path and OS.xxx functions do not seem to retrieve the required path type. There is no HOME present explicitly in my windows profile, however when I created one, the script throwed an error:
expected escape sequence of 8 hexdecimal numbers, but found 's'
in "", line 2, column 18:
target_dir: "C:\Users\XYZ/wilp"
^

After changing this to expected pattern of "C:/Users/XYZ", the script ran, and opened firefox and everything, but failed after downloading in firefox:

c:\sw\impartus-downloader>python impartus.py
Files will be saved at: C:/Users/XYZ/wilp


Traceback (most recent call last):
File "c:\sw\impartus-downloader\impartus.py", line 129, in
impartus.process_videos()
File "c:\sw\impartus-downloader\impartus.py", line 68, in process_videos
media_files = self.browser.get_media_files(ttid)
File "c:\sw\impartus-downloader\browser\firefox.py", line 159, in get_media_files
conn = sqlite3.connect(self.indexed_db())
File "c:\sw\impartus-downloader\browser\firefox.py", line 171, in indexed_db
db = os.path.join(os.path.dirname(media_dir), os.path.basename(media_dir).split('.')[0] + ".sqlite")
File "C:\sw\python3\lib\ntpath.py", line 223, in dirname
return split(p)[0]
File "C:\sw\python3\lib\ntpath.py", line 185, in split
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

[enhancement] Option to save credentials

The username can be added to the conf file at present, which shows up on the UI.
It may be good to have a checkbox to save both username and password and save those to config file.

Improve table sort performance.

At present it takes more than a second to show the sorted table content, the entire table goes blank during this time.
Not a very good user experience.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.