Giter Site home page Giter Site logo

pyfluminus's Introduction

pyFluminus

Coverage Status Python port of the excellent fluminus.

Automatically download recently uploaded files from Luminus. I personally use pyFluminus to download files into Dropbox so that I can view all my modules' files on any device.

PRs are welcome

Features (at parity with fluminus)

  • Authentication via ADFS (vafs.nus.edu.sg)
  • Get name of student
  • Get list of modules
    • Taking/Teaching
    • Only this semester's modules
  • Get announcements
  • Get listing of workbin files and download them
  • Get listing of webcasts and download them
  • Get listing of weekly lesson plans and their associated files

CLI Usage

To download your files:

mkdir /tmp/fluminus
python pyfluminus_cli.py -username="e123456" -password="passw0rd"  --download-to=/tmp/luminus

If you are uncomfortable with having to pass your credentials in as plaintext you can instead save it as environment variables: LUMINUS_USERNAME, LUMINUS_PASSWORD and pass in the --env flag.

This downloads files of all your modules to the directory specified. To download files again simply do:

python pyfluminus_cli.py -username="e123456" -password="passw0rd"  --download-to=/tmp/luminus

More information can be found in the help page:

usage: pyfluminus_cli.py [-h] [-username USERNAME] [-password PASSWORD]
                         [--env] [--download_to DOWNLOAD_TO] [--ignore IGNORE]

CLI wrapper to pyfluminus

optional arguments:
  -h, --help            show this help message and exit
  -username USERNAME    NUSNET username, e.g. e01234
  -password PASSWORD    NUSNET password
  --env                 Get username and password from environment variables
  --download_to DOWNLOAD_TO
                        Download destination
  --ignore IGNORE       Comma separated list of modules to ignore (e.g.
                        CS1231,CS4321)
  --announcements       Display announcements

Extending pyFluminus

The original goal for this project was a barebones CLI interface for downloading LumiNUS files. I can imagine that a GUI wrapper would be a strong use case and a useful tool for users not comfortable with CLI tools. You can see a working example here.

To assist with that, there is the pyfluminus.fluminus.get_links_for_module utility function that returns all of the download links of a module.

Example use case:

from pyfluminus.authorization import vafs_jwt
from pyfluminus import fluminus
from pyfluminus.structs import Module
modules_res = api.modules(auth)
if not modules_res.ok:
    print("Error: ", modules_res.error_msg)
modules = modules_res.data
for module in modules:
    if module is None:
        continue
    print("{} {}".format(module.code, module.name))
    data = fluminus.get_links_for_module(auth, module)
    print(data)

Example output:

{   
    'children': [   {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'test2.pdf',
                                            'type': 'file'},
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'Sample test 2.pdf',
                                            'type': 'file'}],
                        'name': 'Tests',
                        'type': 'folder'},
                    {   'children': [   {   'children': [],
                                            'name': 'Revised Project '
                                                    'Objectives',
                                            'type': 'folder'},
                                        {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                                                'name': 'NG YI '
                                                                        'CHONG '
                                                                        'RAYNOLD '
                                                                        '- '
                                                                        'Project '
                                                                        'Objectives.pdf',
                                                                'type': 'file'}],
                                            'name': 'Project Objectives',
                                            'type': 'folder'},
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'CS4215 Project List '
                                                    'AY2019_20 Semester 2.pdf',
                                            'type': 'file'},
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'Project Objectives '
                                                    'Template.docx',
                                            'type': 'file'}],
                        'name': 'Projects',
                        'type': 'folder'},
                    {   'children': [   {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                                                'name': 'NG YI '
                                                                        'CHONG '
                                                                        'RAYNOLD '
                                                                        '- '
                                                                        'qn4.newlayout.js',
                                                                'type': 'file'}],
                                            'name': 'week6_mark_sweep',
                                            'type': 'folder'},
                                        ...
                                        {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                                                'name': 'NG YI '
                                                                        'CHONG '
                                                                        'RAYNOLD '
                                                                        '- '
                                                                        'week2.txt',
                                                                'type': 'file'}],
                                            'name': 'week2_submission',
                                            'type': 'folder'},
                                        ...
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'week2_template.txt',
                                            'type': 'file'}],
                        'name': 'Lab task submission',
                        'type': 'folder'},
                    {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'slides_11.color.pdf',
                                            'type': 'file'},
                                        ...
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'slides_01_corrected_2020_01_13.bw.pdf',
                                            'type': 'file'}],
                        'name': 'Slides',
                        'type': 'folder'},
                    {   'children': [   {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'notes_11.pdf',
                                            'type': 'file'},
                                        ...
                                        
                                        {   'link': 'https://luminus.nus.edu.sg/v2/api/files/download/....',
                                            'name': 'notes_01.pdf',
                                            'type': 'file'}],
                        'name': 'Notes',
                        'type': 'folder'}],
    'name': 'CS4215',
    'type': 'folder'
}

Which is a useful data source to build a GUI around.

pyfluminus's People

Contributors

indocomsoft avatar raynoldng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

pyfluminus's Issues

File/Folder names with invalid characters break the script

Thanks for creating this script, it's really useful!

However, I seem to be experiencing issues on Windows when the file/folder names contain invalid characters such as colons.
https://docs.microsoft.com/en-sg/windows/win32/fileio/naming-a-file#naming-conventions

- C:\Users\paksh\Documents\luminustest\SP1541-ES1541\Submission Folder:  Pre-course Writing Task\SP1541_AY2122Sem2_PreCourseWritingTask_SubmissionTemplate.docx
Traceback (most recent call last):
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus_cli.py", line 109, in <module>
    download_files(module_file, auth, args.download_to, True)
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus_cli.py", line 36, in download_files
    download_files(child, auth, download_path, verbose)
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus_cli.py", line 36, in download_files
    download_files(child, auth, download_path, verbose)
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus_cli.py", line 27, in download_files
    file.download(auth, download_path, verbose)
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus\structs.py", line 314, in download
    return utils.download(url, destination, verbose)
  File "C:\Users\paksh\Documents\GitHub\pyfluminus\pyfluminus\utils.py", line 26, in download
    os.makedirs(dir_path, exist_ok=True)
  File "C:\Users\paksh\AppData\Local\Programs\Python\Python39\lib\os.py", line 225, in makedirs
    mkdir(name, mode)
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\paksh\\Documents\\luminustest\\SP1541-ES1541\\Submission Folder:  Pre-course Writing Task'

It works fine on Linux systems though, I discovered this issue because I run this script on my linux server and sync the files to my Windows laptop, but the sync wasn't working for certain folders with naming issues.

ERROR: unable to get local issuer certificate (_ssl.c:1123)

using Ubuntu 20.03 with NUS_STU wifi, latest python 3.8.5, did pip3 with requirements.txt

tried to python3 pyfluminus_cli.py -username="e123456" -password="password" (obviously not real credentials)

got the following error:
Traceback (most recent call last):
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in validate_conn
conn.connect()
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/util/ssl
.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/home/naws/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='luminus.nus.edu.sg', port=443): Max retries exceeded with url: /auth/callback?code=itFF1arP8UGo8DZsRZZDGg.IavbUaiz2AjgZhCQENfk4bNurmI.CSB1siK9zkWoL9RGFndyd_CyMp-UIvyYSlS-WIH_dLsCV_ynMNINlRHQ2uv-IYOpA5D1JDxth7IO47546yNvnPNEqCM_ZzudsuTY5ptO4kfmI2uy1OEyuzgyhkrlEPzRTnjv0eYlc-bkxJE79fTZqWX0q8bL6FR0igYKs1qjlmDlB29FyKvCcc9KbEMZ7ca7S3ekRf9ZedN7-flaX-tbQfGrRPVuPNrBwU8-p7dWusLdAtKqLEOVQPU5gVcQRWNRdeyHvHgFPOvsFxdy8PzjI24FVn6JeSCdwJAQEdlw6ekvqwCM-ewVt6MzkcgQfjTMQ8vrWayIXNxXlheA9m_yZg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "pyfluminus_cli.py", line 47, in
auth = vafs_jwt("nusstu\" + username, password)
File "/home/naws/pyfluminus/pyfluminus/authorization.py", line 33, in vafs_jwt
response = requests.post(url, headers=auth_headers, data=auth_data)
File "/usr/lib/python3/dist-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 668, in
history = [resp for resp in gen] if allow_redirects else []
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 239, in resolve_redirects
resp = self.send(
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='luminus.nus.edu.sg', port=443): Max retries exceeded with url: /auth/callback?code=itFF1arP8UGo8DZsRZZDGg.IavbUaiz2AjgZhCQENfk4bNurmI.CSB1siK9zkWoL9RGFndyd_CyMp-UIvyYSlS-WIH_dLsCV_ynMNINlRHQ2uv-IYOpA5D1JDxth7IO47546yNvnPNEqCM_ZzudsuTY5ptO4kfmI2uy1OEyuzgyhkrlEPzRTnjv0eYlc-bkxJE79fTZqWX0q8bL6FR0igYKs1qjlmDlB29FyKvCcc9KbEMZ7ca7S3ekRf9ZedN7-flaX-tbQfGrRPVuPNrBwU8-p7dWusLdAtKqLEOVQPU5gVcQRWNRdeyHvHgFPOvsFxdy8PzjI24FVn6JeSCdwJAQEdlw6ekvqwCM-ewVt6MzkcgQfjTMQ8vrWayIXNxXlheA9m_yZg (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.