malloydelacroix/downloaderforreddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.

Downloader For Reddit

Downloader for Reddit is a GUI application for downloading content, self text, and comments submitted to reddit.
It has many advanced features and customizable settings to extract only content that you want and avoid duplicate downloads. Everything that is extracted is stored in an sqlite database. A built-in database view window allows for viewing and filtering all the stored data. The database can also be accessed with any software capable of reading an sqlite database, so the data you extract can be used and analysed in limitless ways.

Users and subreddits are kept in two separate list displays. You can have an unlimited number of lists containing an unlimited number of users/subreddits. The lists can be downloaded as a whole, or the individual users/subreddits can be downloaded independently. User downloads can also be constrained to posts made only to the subreddits in the current subreddit list.

The output display can be customized to what level of information you would like to have shown. Each output level can be customized to display in what ever color you would like it shown.

Most download settings can be applied to the entire application, an entire user/subreddit list, or individual users/subreddits. This allows you to fine tune what is downloaded from where on a large or small scale.

The settings dialog is robust and lets you fine tune many aspects of the application such as download parameters, display settings, and notifications. Through this dialog you can also schedule downloads to run at a particular time interval or run at certain times on certain days.

The database view can be adjusted to show you only the data that you want to see. Each object view can be toggled on or off and is dependent on the model view before it. For instance: as shown above, if a subreddit is selected, and the post view is toggled off, all the content for that subreddit will be displayed. If posts are toggled on, only posts for the selected subreddit will be displayed, and only content for the selected post will be displayed.

In addition to this filtering, there is a robust specific filtering system in which every model can be filtered by almost every parameter in an easy to use interface.

Requirements:

Imgur Posts:

Before any content can be downloaded from Imgur.com you will need to have an imgur client-id and client secret. To obtain this go to https://api.imgur.com/oauth2/addclient.
The following settings are recommended:

• Application name: Downloader For Reddit
• Authorization type: Anonymous usage without user authorization
• Authorization callback URL: Any valid url can be used here (ex. https://google.com)
• Application website: https://github.com/MalloyDelacroix/DownloaderForReddit
• Email: This is the email the client-id and secret will be sent to

This client-id and client secret will be entered on the Imgur category of the settings menu.

Imgur will allow you 12,500 downloads per day and 500 downloads per hour. Your remaining Imgur credits and reset time can be viewed by clicking 'Imgur Credits' in the help menu.

Reddit Video Posts:

Due to the way they are hosted by reddit, video's that are downloaded from v.redd.it will be saved as two files, one video file and one audio file. In order for the application to merge the two files into one playable video after download, FFmpeg must be installed and on the system path.

Please see this wikiHow article for more information on how to install FFmpeg on a Windows system.

Installing The Downloader For Reddit

By far the easiest way to install and use The Downloader for Reddit is to download one of the pre-packaged executables from the releases section (at the moment stand-alone executables are only available for Windows).

To run the program, extract the folder (DownloaderForReddit) from the zip file and scroll down to and click the executable file (DownloaderForReddit.exe). Everything is self-contained in the folder, there is nothing to install.
To run the application from source, please refer to the wiki entry Setting Up A Development Environment.

downloaderforreddit's Issues

Performance Slowdowns

I generally notice with these new commits the general performance slowdowns when downloading, takes much longer to extract than usual.Would like to see some speedups in the future.Perhaps SQL-lite backend and more efficient processing on the part of the extractors. from commit 55e8789 to 3ec6f1e is a noticable difference

Windows - reports old version, crashes when trying to remove list

Hi -

Windows 7 (DFR version unkown, ZIP named 2.0.2 but the Help>About reports 2.0)

Also, I can't remove lists as the app will crash (Windows appcrash dialog).

Let me know if you need additional information.

Thanks!

Suggestions

I have a few suggestions for this great program.

• Automatically save changes to lists (crashes wiped out my lists the first few times before I realised I had to manually save)
• Option to Backup/restore lists
• Selecting "import save file" then cancelling results in an error
• Include a pause/resume button
• Option to exclude subreddit from downloading without removing it from list
• Option to rename subreddits (if a typo is made when entering. have to remove and readd at the moment)
• Allow adding multiple subreddits to lists by pasting from the clipboard instead of one at a time.

Maybe in the future you could make a version which works with 4chan or similar.

A couple more bugs/issues

• It's possible to open more than one instance of the program. (make this an option?)
• Clicking [download] places a * in the titlebar as if settings need to be saved again?
• Lists menu - "Rmove Subreddit List"
• Some child windows appear in random locations (annoying on multimonitor). maybe centre to parent?

Program crash if domain name doesn't exist or if server doesn't respond

Hi,

I don't have any logs right now, but this one should be easy enough to replicate.
If you crawl a subreddit, and domain the program is trying to download an image from has expired/doesn't exist, the program will crash.

The same thing seems to happen if the domain exist, but is offline (doesn't respond).
Would it be possible to make the program just skip these entries after a few retries, instead of completely shutting down?

Crashing during validation with InvalidTokenException

Downloader suddenly crashes during validation phase. The log reports a Critical level entry caused by: prawcore.exceptions.InvalidToken: received 401 HTTP response.

Wont load due to missing attribute

Upgrading from ver: 2.3.0-master commit: 55e8789
to
commit: 3ec6f1e
App will no load due to missing attribute error
commit: e63a253 seems to be the suspect

Log:
https://imgur.com/a/8giX6vi

rolling back to last working commit

Reset Date in User Pages causes crash

Steps to reproduce
1- open a user
2- right click "restrict by date" option
3- choose any 2 reset options in menu
4- crash

Version- 2.3.0 -master
commit- 49a38be

python- 3.7.0-windows stable

Error Log:
https://imgur.com/a/hqaPLX2

Failed to retrieve json data from link

Noticed about a week ago that this error comes up when it hits a gfy link.

Happens in both Win10 and Linux.

Portable, not installed on C, save_file.dat...

Could it be possible to determine location where to save save_file.dat?

Some stuff I don't want to keep on Windows partition as Windows gets clean installed few times a year. So it's better to not have files there - easier & faster & safer.

Imgurpython library is depreicated

The imgurpython library is no longer being maintained. The imgur extractor currently makes heavy use of this library and will need to be updated to access the imgur api without the library.

The library is still working at the moment, but there has been an update in imgur's api documentation, so it is likely only a matter of time before it becomes non-operational.

Not getting past validity check

Using my two wallpaper/s groups as before, the app says both are valid, then sits at 50% seemingly indefinitely. Logs below.

{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/16/2018 09:11:31 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/16/2018 09:11:36 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/16/2018 09:11:46 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/16/2018 09:12:14 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/16/2018 09:12:14 AM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/16/2018 09:13:02 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 357, in increment\n File "lib\site-packages\urllib3\packages\six.py", line 685, in reraise\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 390, in run_extraction\n File "DownloaderForReddit\Core\DownloadRunner.py", line 414, in extract\n File "DownloaderForReddit\Extractors\Extractor.py", line 52, in run\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 490, in send\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))"}
{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.SettingsManager", "filename": "SettingsManager.py", "module": "SettingsManager", "funcName": "check_first_run", "lineno": 60, "message": "First run of new version", "asctime": "03/17/2018 08:42:16 AM", "new_version": "v2.2.1", "old_version": "v2.2.0"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/17/2018 08:42:17 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/17/2018 08:42:23 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/17/2018 08:42:28 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/17/2018 08:42:32 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/17/2018 08:42:32 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/17/2018 08:43:18 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/17/2018 08:43:30 AM"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Extractors.Extractor", "filename": "Extractor.py", "module": "Extractor", "funcName": "extract", "lineno": 82, "message": "Failed to extract content: Unknown error", "asctime": "03/17/2018 09:10:46 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connection.py", line 141, in _new_conn\n File "lib\site-packages\urllib3\util\connection.py", line 83, in create_connection\n File "lib\site-packages\urllib3\util\connection.py", line 73, in create_connection\nTimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn\n File "lib\site-packages\urllib3\connection.py", line 284, in connect\n File "lib\site-packages\urllib3\connection.py", line 150, in _new_conn\nurllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 388, in increment\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 508, in send\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x032383F0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))", "url": "https://i.imgur.com/wjk5QuI.jpg", "reddit_object": {"name": "wallpaper", "object_type": "SUBREDDIT", "version": "v2.2.1", "save_path": "U:/Downloads/Reddit Downloader/", "post_limit": 1000, "avoid_duplicates": true, "download_videos": true, "download_images": true, "nsfw_filter": "INCLUDE", "added_on": 1521240191.041695, "do_not_edit": false, "new_submission_count": 925, "saved_submission_count": 0, "previous_download_count": 4076, "date_limit": 1521235400.0, "custom_date_limit": null, "content_count": 4076, "failed_extract_count": 1, "saved_content_count": 0}}
{"levelname": "INFO", "name": "DownloaderForReddit.Core.DownloadRunner", "filename": "DownloadRunner.py", "module": "DownloadRunner", "funcName": "stop_download", "lineno": 347, "message": "Downloader stopped", "asctime": "03/17/2018 09:11:18 AM", "run_time": "28 mins, 0 secs"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:10 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 122, in save_pickled_state\n File "shelve.py", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "03/17/2018 09:13:14 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 122, in save_pickled_state\n File "shelve.py", line 124, in setitem\nTypeError: can't pickle Content objects", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}

Crash during download

not related to other crash in issue #21
happened during downloading
log: https://imgur.com/a/N8pKTFB
version: 2.3.1?-master
commit: 3ec6f1e
python: 3.7.0-windows stable

Ver 2.0.2 shows (about screen) Ver 2.0.0

And offers and update... (if clicked).

Migrating to sqlite support?

I know this is on your todo list. I've put some thought into this too and I'm wondering how I can help. I don't really have time right now, but maybe if we can split it up into small chunks it'll be easier to do.

To ORM or not to ORM?

Should we use the stdlib sqlite3 package or an ORM?

• https://docs.python.org/3/library/sqlite3.html
• Peewee ORM http://docs.peewee-orm.com/en/latest/
• PonyORM https://ponyorm.com/
• SQLAlchemy https://www.sqlalchemy.org/

Initial migration path

1. Hard migrations – we have a one time migration. One and done. This would mean everything has to get converted to sqlite
2. Gradual migration – migrate one model at a time. RedditObjects, Redditors, Redditor preferences, Subreddits, Subreddit preferences, system preferences, etc.

Maintenance

Deleting old data? Vacuums? Schema migrations?

Crash when viewing imgur credits after a user download (ver-2.3.0 python-3.7.0-windows)

1.) click a user
2.) click download this user
3.) after process completes i click save
4.) click help> view imgur credits
5.) crash

edit: also crashes while trying view credit count DURING downloading

Program crashing when saving the settings

So, i've tried pretty much every version of the software. The program keeps crashing for me when i try to edit my settings.
Prior 2.1.0 it used to freeze then give me the option to end the process... now after the "save fix" it only crashes instantly.
I've tried deleting the save file, re-import the save and everything i could think of but nothing seems to work.

The only thing i can think of is reinstalling the .net framework? would that help? thanks.

REQ: Import usernames/subreddits from a text file?

Import usernames/subreddits from a text file?

A list that I'm following is on at text file, I'd like to import that to the app...
Manually wayyy slow by adding one at a time :/

OFF TOPIC: Quick question.. about Twitter

Quick question.. about Twitter

Does anyone know a an app similar to this but for Twitter?

Keep Using Credits For Downloaded Content

I like the app but it 's getting annoying when my hourly user credits get burned each time I download users with avoid duplicates on.Thought the purpose for it was to remember what content it downloaded and save it in a database this way next time I download it knows better to make a call for it.Literally when my hourly credits reset I am left with just 50 credits (from 500) just to download 6 imgur pics.I notice credits burn and nothing is downloaded.

Unable to save user list

I am having a lot of trouble saving the user list.
If I add users then immediately save, it's all good.
If I add users and then start the download, then one of 2 things happens:

1. the final download to complete is a success (an image on a host supported by the app) -> then all is good, the user list and progress can be saved.

2. if the final download to process is a failure (unsupported type) -> unable to save, any users added if not saved before starting the download are lost. The modification date on the save file is not changed.

When scenario 2 happens, the "download" button remains stuck, clicking it to stop does nothing. Attempting to save via the file menu gives a failure message.

The biggest issue with this is not only are users added to the list discarded if not saved before starting the download, but the same content is downloaded over and over again until a scenario # 1 happens and a save can be done properly.

the log shows this when attempting to save, also URL to screen capture of pop up error.

{"levelname": "ERROR", "name": "DownloaderForReddit.DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "save_pickled_state", "lineno": 129, "message": "Unable to save to save_file", "asctime": "12/07/2018 04:32:02 PM", "exc_info": "Traceback (most recent call last):\n File \"DownloaderForReddit\\Persistence\\ObjectStateHandler.py\", line 121, in save_pickled_state\n File \"shelve.py\", line 124, in __setitem__\nTypeError: can't pickle Content objects", "save_file_location": "C:\\Users\\Administrator\\AppData\\Roaming\\SomeGuySoftware\\DownloaderForReddit\\save_file"}

url

Downloading direct links instead of content

I'm not quite sure it's the right place but I've been looking for a way to extract the video and images direct link rather than the content in itself. Is there a way to do that?

Any plans on adding multireddit support?

DownloaderForReddit works great when downloading from a handful of subreddits at a time but there doesn't seem to be any support for multireddits, when i tried to work around this and add each subreddit manually the program started to crash when a list became greater than ~85 subreddits long. Being able to download relevant images from multiple subreddits in one batch could certainly be quite handy.

Crash before downloading

Thanks for the update. Unfortunately, I can't get it to work at all now.

I completely removed the app and installed it again, including roaming data.
I have added one list "wallpaper" and added the groups "wallpaper" and "wallpapers".
It says both are valid, gets to 50% then crashes. Checking my download folder, nothing is there, not even the subfolders.

Here is my log, thanks.

{"levelname": "ERROR", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 87, "message": "Failed to load from save file", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "shelve.py", line 111, in getitem\nKeyError: 'user_list_models'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Persistence\ObjectStateHandler.py", line 62, in load_pickled_state\n File "shelve.py", line 113, in getitem\n File "dbm\dumb.py", line 148, in getitem\nKeyError: b'user_list_models'", "save_file_location": "C:\Users\Craig\AppData\Roaming\SomeGuySoftware\DownloaderForReddit\save_file"}
{"levelname": "ERROR", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1329, "message": "Load state error: No save file found", "asctime": "03/16/2018 09:11:25 AM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\GUI\DownloaderForRedditGUI.py", line 1312, in load_state\nTypeError: 'NoneType' object is not subscriptable"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "init", "lineno": 47, "message": "Settings dialog opened", "asctime": "03/16/2018 09:11:31 AM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "set_imgur_client", "lineno": 201, "message": "Imgur client_id and client_secret set", "asctime": "03/16/2018 09:11:36 AM", "valid_client_id": true, "valid_client_secret": true}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditSettingsGUI", "filename": "DownloaderForRedditSettingsGUI.py", "module": "DownloaderForRedditSettingsGUI", "funcName": "save_settings", "lineno": 307, "message": "Settings saved", "asctime": "03/16/2018 09:11:46 AM", "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_user_settings", "lineno": 1056, "message": "User settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_users": 0, "total_users": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "update_subreddit_settings", "lineno": 1070, "message": "Subreddit settings updated", "asctime": "03/16/2018 09:11:46 AM", "updated_subreddits": 0, "total_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/16/2018 09:12:14 AM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/16/2018 09:12:14 AM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/16/2018 09:13:02 AM", "exc_info": "Traceback (most recent call last):\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "lib\site-packages\requests\adapters.py", line 440, in send\n File "lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen\n File "lib\site-packages\urllib3\util\retry.py", line 357, in increment\n File "lib\site-packages\urllib3\packages\six.py", line 685, in reraise\n File "lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen\n File "lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request\n File "lib\site-packages\urllib3\connectionpool.py", line 850, in validate_conn\n File "lib\site-packages\urllib3\connection.py", line 326, in connect\n File "lib\site-packages\urllib3\util\ssl.py", line 329, in ssl_wrap_socket\n File "ssl.py", line 401, in wrap_socket\n File "ssl.py", line 808, in init\n File "ssl.py", line 1061, in do_handshake\n File "ssl.py", line 683, in do_handshake\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 390, in run_extraction\n File "DownloaderForReddit\Core\DownloadRunner.py", line 414, in extract\n File "DownloaderForReddit\Extractors\Extractor.py", line 52, in run\n File "DownloaderForReddit\Extractors\Extractor.py", line 62, in extract\n File "DownloaderForReddit\Extractors\ImgurExtractor.py", line 56, in init\n File "lib\site-packages\imgurpython\client.py", line 85, in init\n File "lib\site-packages\imgurpython\client.py", line 94, in get_credits\n File "lib\site-packages\imgurpython\client.py", line 131, in make_request\n File "lib\site-packages\requests\api.py", line 72, in get\n File "lib\site-packages\requests\api.py", line 58, in request\n File "lib\site-packages\requests\sessions.py", line 508, in request\n File "lib\site-packages\requests\sessions.py", line 618, in send\n File "lib\site-packages\requests\adapters.py", line 490, in send\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))"}

right click -> paste when adding users/subs missing

Mouse - right click -> paste username/subredditname into adding field - latest version right click option >>> ### PASTE <<<< missing.

Cannot import save file

I'm trying all I can... I just cant get the app to read/save imported save file :/

Crash During Downloading

Happens sometimes randomly, happens more if you have more than 1 download thread set.
Steps:
1-click download
2-while downloading crash sometimes occurs randomly

Version: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

Screenshot:
https://imgur.com/a/w1S4bC2

"A save file could not be created..." error when loading old config

I have been receiving errors after updating from DFR 2.22 to 2.3x - I have tried 2.31 and 2.32. OS Win10. To find the tidiest way to duplicate this problem here's what I did:

• deleted %appdata%\Roaming\SomeGuySoftware\DownloaderForReddit
• started DFR
• asked it to import save file from an old copy I had on my desktop
• confirmed overwrite

After overwrite this error immediately appears, and it also appears if the app is closed and reopened:

If the config directory is purged completely and a new config is created, the app works fine.

See attached log:
DownloaderForReddit.log

Not properly carrying over imgur downloads

idk best way to title this but here the issue,app I know is designed for any reason it cannot download an imgur imgage because of credits,over capacity or etc what happens(or should happen) is that link get saved for next download run.So when you download again it should get the image it could not get for some reason last time.In this version I believe that is not properly working .Imgur images that fail to download do not download next run, but prob is that it triggers a crash as described in issue #21 and #27 .Crash triggers right away making the user undownloadable. Only way to fix it is to remove user then readd user and redownload that user..

Steps to reproduce
1- start with very low imgur credits
2- add a user (that has imgur images more than your credits)
3- open user
4- download
5- crash

Version- 2.3.0 -master
commit- 49a38be

python- 3.7.0-windows stable

Imgur Credits Draining on their own

I noticed this just now, with nothing going on credits go down on their own

https://imgur.com/a/CTyT6VP

ver- 2.3.0-master
commit- 9f1f509
python- 3.7.0-windows stable

Crashes on save settings

Just trying to save settings without changing anything and it crashes lmao. Also why is the default save path C:\Users\<user>/Downloads/. Windows 7 btw

Scheduling a download

More of an enhancement idea than an issue.

Is there a way or plans to schedule a download at regularly set intervals? I find that some users are deleting posts after they have been up for less than a day and I would like to capture it if I am not at the computer.

.gifv files...?

Some mp4 / webm files comes as .gifv files to local...?

What to do? Try to re-DL?

Allow us to disable video sites

A window to allow us to enable which sites we want videos to be downloaded from and which we want disabled since some sites do not work or if we do not want certain sites to download.
The new video downloader does not work 100% as if I try download a user and if for example they have p0rnhub link it does not download thus going into an infinite page download.But what if I want to disable that site all together? Removing said site from supported_video_sites .txt does not work as it just say cannot find extractor but it will still try to download it anyway.
something like:

-----------------------------------|III|-----------------------------
     sites enabled                 |III|          availables sites           
-----------------------------------|III| ----------------------------
Youtube                            |III|  P0rnhub
Vimeo                              |III|  AMC
etc                                |III|  etc

Constant crashes

I have a subreddit list with two groups, "wallpaper" and "wallpapers". Although it has download a combined few thousand files, it keeps crashing whenever I start the download. Log file below ...

{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 83, "message": "Object lists loaded from save file", "asctime": "03/03/2018 05:39:15 PM", "total_users": 0, "total_subreddits": 32, "updated_users": 0, "updated_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1317, "message": "Save_file successfully loaded into gui", "asctime": "03/03/2018 05:39:15 PM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/03/2018 05:39:18 PM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/03/2018 05:39:18 PM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/03/2018 05:39:43 PM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 385, in extract\n File "DownloaderForReddit\Core\RedditObjects.py", line 195, in load_unfinished_downloads\nTypeError: init() missing 1 required positional argument: 'display_only'"}
{"levelname": "INFO", "name": "DownloaderForReddit.Persistence.ObjectStateHandler", "filename": "ObjectStateHandler.py", "module": "ObjectStateHandler", "funcName": "load_pickled_state", "lineno": 83, "message": "Object lists loaded from save file", "asctime": "03/03/2018 05:41:26 PM", "total_users": 0, "total_subreddits": 32, "updated_users": 0, "updated_subreddits": 0}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "load_state", "lineno": 1317, "message": "Save_file successfully loaded into gui", "asctime": "03/03/2018 05:41:26 PM"}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "display_imgur_client_information", "lineno": 1180, "message": "Imgur client info calculated", "asctime": "03/03/2018 05:41:32 PM", "remaining_app_credits": 5567, "remaining_user_credits": 500}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "run_subreddit", "lineno": 474, "message": "Subreddit download initiated", "asctime": "03/03/2018 05:41:57 PM", "list_size": 2, "settings": {"imgur_client_valid": true, "restrict_by_score": false, "score_limit_operator": "GREATER", "score_limit": 3000, "subreddit_sort_method": "NEW", "subreddit_sort_top_method": "DAY", "post_limit": 1000, "restrict_by_date": false, "restrict_by_custom_date": false, "custom_date": 86400, "download_videos": true, "download_images": true, "avoid_duplicates": true, "nsfw_filter": "INCLUDE", "save_subreddits_by": "Subreddit Name", "name_downloads_by": "Image/Album Id", "save_directory": "U:/Downloads/Reddit Downloader/", "max_download_thread_count": 4, "save_undownloaded_content": true}}
{"levelname": "INFO", "name": "DownloaderForReddit.GUI.DownloaderForRedditGUI", "filename": "DownloaderForRedditGUI.py", "module": "DownloaderForRedditGUI", "funcName": "start_reddit_extractor_thread", "lineno": 536, "message": "Downloader thread started", "asctime": "03/03/2018 05:41:57 PM"}
{"levelname": "CRITICAL", "name": "DownloaderForReddit.main", "filename": "main.py", "module": "main", "funcName": "log_unhandled_exception", "lineno": 86, "message": "Unhandled exception", "asctime": "03/03/2018 05:42:23 PM", "exc_info": "Traceback (most recent call last):\n File "DownloaderForReddit\Core\DownloadRunner.py", line 385, in extract\n File "DownloaderForReddit\Core\RedditObjects.py", line 195, in load_unfinished_downloads\nTypeError: init() missing 1 required positional argument: 'display_only'"}

Audio issue

Reddit splits video/audio and this app couldnt grab audio from links, is there anyway to grab audio and merge it with video file on this app ?

Error on startup

version: 2.3.1?-master
commit: 842be61
python: 3.7.0-stable windows

https://imgur.com/a/z7pCizh

I changed the version.py like last time and it didn't work

What's the preferred requirements.txt style?

In my local branch, I updated the requirements and things seem to be working. In trying to figure out how to contribute that back, I wasn't sure which style of requirements.txt the project is using to match.

https://github.com/MalloyDelacroix/DownloaderForReddit/blob/master/requirements.txt

1. List everything -- it looks like it's listing everything because I see prawcore and six, but requests's requirements and some others are missing
2. Only list the top level requirements -- Then delete deps of deps like prawcore and six
3. Only list the top level requirements and deps of deps if there's a problem -- I would expect to see comments explaining why something had to be kept back

---

I've used pip-tools and pipenv and I prefer pip-tools. The current versions I get from that are:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
beautifulsoup4==4.6.3
certifi==2018.8.24        # via requests
chardet==3.0.4            # via requests
idna==2.7                 # via requests
imgurpython==1.1.7
praw==6.0.0
prawcore==1.0.0           # via praw
pyqt5-sip==4.19.12        # via pyqt5
pyqt5==5.11.2
python-json-logger==0.1.9
requests==2.19.1
update-checker==0.16      # via praw
urllib3==1.23             # via requests
youtube-dl==2018.8.28

3rd party Reddit backup pholder_dot_com

It caches tons of Reddit subreddits. urls are similar as on actual Reddit domain - like pholder.com/r/feminism AND user pages like: pholder.com/u/feminist

^ just an example ^ could it be possible for this app to check pholder_dot_com/* if it has /r/ or /u/ content cached (pholder_dot_com can go years back.. :D) and possibly DL if usual places are empty?

Or provide links of non-downloaded but available content on DFR's user page? The log can become overwhelming to read and copy paste links..

Just ideas!

Imgur Credits not refreshing

Credits not refreshing at the designated time unless you reload the app.

Ver 2.3.0-master
PY 3.7.0-windows stable
Edit: typo with python version

Windows Defender (up-to-date WIN10) detects "dfr_updater.exe" as a Trojan

In the latest provided release (zipped dfr 1.1.0 for windows).
Don't know if it is a false positive or not.

Download Videos setting ignored

download videos checkbox being ignored in user settings page with "do not overwrite override these settings" checked.
It still tries to download videos even if I set the user not to

Ver: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

Can't execute

I try to execute the app but I get an error message: "failed to execute script main". How to solve?

Windows 10.

Failed to extract post: Url domain not supported

Hi,

I was trying to use your GUI based downloader, and i wanted to download the top posts of all time from the subreddit www.reddit.com/r/learnprogramming

I added the subreddit into the subreddit list. Checked download subreddit. I have attached a screenshot of my settings as well.

After all that, i get the error : Failed to extract post: Url domain not supported.

Could you please help me out, let me know what im doing wrong. They are almost all text posts with some posts containing links. No images, no videos etc. Thank you so much for your help and for creating this program.

Scanning of previously DL'd data?

I'm not quite sure does that work, it seems that every time DFR dl's a bunch of duplicates... :/

Then again I had Reddit Data Ext before DFR and I didn't delete old content...

I'm using a tool to find exact duplicates after DL with DFR to sort out old duplicates compared to new.

Hmm I'm not sure ~ perhaps RDE has few moe settings to fine tune. That client either doesn't 'find' other clients data as something of previous.

Gifv Support

App is downloading Gifv instead of .webm/.mp4, gifv are not loadable they just contain metadata of a sort.

Error whilst trying to add a subreddit

"There are no user lists available. To add a user, please add a user list"

I've added 2 user lists but its not letting me add a subreddit.

Can't save settings

Using the last version but when I try to save settings, the program exists completely and doesn't save any settings!!!
Also tried the previous version and the same happens.
Is there a way to edit the settings, like a config file?? If yes, where is located?? Couldn't find it...

#jpg Being Appended

#jpg being appended to all pictures (jpgs) that is on i.redd.it

Ver: 2.3.0-master commit: 4a321df
Python: 3.7.0-windows

Need docs for how to run the tests

Some code assumes you're running from the DownloaderForReddit/ directory, but the tests are then in ../Tests. There's lots of other weird things that prevent the normal ways of getting python -m unittest from being able to discover and run the tests.

I can get tests to start running if I'm in the project root and do PYTHONPATH=DownloaderForReddit/ python -m unittest but then I'm in the wrong directory for open(supported_sites_file, 'r') to work.