kirankotari/weblinks

Weblinks, It get all links from a given website and we can apply filters on top of it to get desired results, when you are good you can start downloading them with -d flag. In the library we are supporting plain webpages, authentication based webpages, proxy, authentication on proxy, etc. We also support storing the config in either local/global configuration, for best practise we suggest not to store your password, you get a prompt on runtime where you can provide it.

Weblinks

• Introduction
• Pre-requisites
• Installation and Downloads
• Commands
• Docs
• Bug Tracker and Support
• License and Copyright
• Author and Thanks

Introduction

Weblinks, It get all links from a given website and we can apply filters on top of it to get desired results, when you are good you can start downloading them with -d flag.

In the library we are supporting plain webpages, authentication based webpages, proxy, authentication on proxy, etc.

We also support storing the config in either local/global configuration, for best practise we suggest not to store your password, you get a prompt on runtime wher you can provide it.

Pre-requisites

A system need to support curl commands and python3

Installation and Downloads

pip install weblinks

Commands

weblinks --help

usage: weblinks [-h] [-w WEB] [-s SUBSTRING] [-e EXT] [-d] [-u USERNAME]
                [-p PASSWORD] [-g] [-l] [-v] [--proxy PROXY]
                [--proxy-username PROXY_USERNAME]
                [--proxy-password PROXY_PASSWORD] [--version]

optional arguments:
  -h, --help            show this help message and exit
  -w WEB, --web WEB     the website
  -s SUBSTRING, --substring SUBSTRING
                        the sub-string in the links
  -e EXT, --ext EXT     file extention
  -d, --download        download links
  -u USERNAME, --username USERNAME
                        web login username
  -p PASSWORD, --password PASSWORD
                        web login password
  -g, --global          global configuration
  -l, --local           local configuration
  -v, --verbosity
  --proxy PROXY         proxy address
  --proxy-username PROXY_USERNAME
                        proxy username
  --proxy-password PROXY_PASSWORD
                        proxy password
  --version             weblinks version

Docs

Weblinks usage

To see current lib. version

weblinks --version
# weblinks version: 2.0

To see python file from given url

weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python
# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found
# Python-3.8.13.tar.xz
# Python-3.8.13.tar.xz.asc
# Python-3.8.13.tgz
# Python-3.8.13.tgz.asc

Still wanted to filter add file extention

weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz
# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found
# Python-3.8.13.tgz

Start download, listed links are good

weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz -d
# INFO     | 2022-07-23 16:25:34,807 | run     :117  | links found
# Python-3.8.13.tgz
# INFO     | 2022-07-23 16:25:34,807 | run     :124  | start download: Python-3.8.13.tgz
# INFO     | 2022-07-23 16:25:34,807 | utils   :58   | downloading: Python-3.8.13.tgz
# INFO     | 2022-07-23 16:25:36,849 | run     :126  | completed: Python-3.8.13.tgz

For authentication

weblinks --web <url> --substring <sub> --username <kirankotari> --password <xxxxxx> --ext .tgz
# Note: don't add --password, it will ask dynamically

For verbose add -v

weblinks --web <url> --substring <sub> --username <kirankotari> --ext .tgz -v

To store config add --local or --global respectively

weblinks --local --web <url> --username <kirankotari> --ext .tgz

For proxy

weblinks --proxy <ip>:<port> --web <url> --substring <sub>

For proxy with authentication

weblinks --proxy <ip>:<port> --web <url> --substring <sub> --proxy-username <proxy user> --proxy-password <proxy password>

Adding web authentication on top of proxy auth.

weblinks --proxy <ip>:<port> --web <url> --substring <sub> --proxy-username <proxy user> --proxy-password <proxy password> --username <user>

Bug Tracker and Support

• Please report any suggestions, bug reports, or annoyances with weblinks through the Github bug tracker. If you're having problems with general python issues, consider searching for a solution on Stack Overflow.
• If you can't find a solution for your problem or need more help, you can ask a question.

License and Copyright

• weblinks is licensed Apache 2.0 2022

Author and Thanks

Weblinks was developed by Kiran Kumar Kotari

weblinks's Issues

feature: to added local and global configs

weblinks/weblinks/weblinks.py

Line 91 in 44adc3f

parser = argparse.ArgumentParser()

parser = argparse.ArgumentParser(
    prog='weblinks', 
    add_help=False
)
parser.add_argument("substring", help="sub-string filter")
parser.add_argument("-w", "--web", default=None, help="the website")
parser.add_argument("-u", "--username", default=None, help="web login username")
parser.add_argument("-p", "--password", default=None, help="web login password")
parser.add_argument("-e", "--ext", default=None, help="file extention")
parser.add_argument('-d', '--download', action='store_true', help="download links")
parser.add_argument("-v", "--verbosity", action="count", default=0)

parent_config_parser = argparse.ArgumentParser(
    prog='weblinks', 
    add_help=False
)
parent_config_parser.add_argument('-w', '--web', nargs="?", default=None, help='web site url')
parent_config_parser.add_argument('-u', '--username', nargs="?", default=None, help='web site username')
parent_config_parser.add_argument("-e", "--ext", nargs="?", default=None, help="file extention")
parent_config_parser.add_argument('-d', '--download', action='store_true', help="download links")
parent_config_parser.add_argument('-s', '--show', default=False, action='store_true', help="display configuration")

config_main_parser = argparse.ArgumentParser(
    prog='weblinks'
)
config_subparsers = config_main_parser.add_subparsers(title="configuration", dest="config")
config_parser = config_subparsers.add_parser("substring", help="the sub-string in the links", parents=[parser])
config_parser = config_subparsers.add_parser("local", help="weblinks local configuration", parents=[parent_config_parser])
config_parser = config_subparsers.add_parser("global", help="weblinks global configuration", parents=[parent_config_parser])
args = config_main_parser.parse_args()
print(args)

feature: Add loggers and add Trace log

weblinks/weblinks/weblinks.py

Line 127 in 68f8c79

# TODO: need to update to obj.log.info("")

def _create_trace_loglevel(logging):
    "Add TRACE log level and Logger.trace() method."

    logging.TRACE = 5
    logging.addLevelName(logging.TRACE, "TRACE")

    def _trace(logger, message, *args, **kwargs):
        if logger.isEnabledFor(logging.TRACE):
            logger._log(logging.TRACE, message, args, **kwargs)

    logging.Logger.trace = _trace