Giter Site home page Giter Site logo

x89 / shreddit Goto Github PK

View Code? Open in Web Editor NEW
989.0 20.0 136.0 540 KB

Remove your comment history on Reddit as deleting an account does not do so.

License: Other

Python 100.00%
python praw reddit reddit-api oauth privacy privacy-protection privacy-tools reddit-application

shreddit's Introduction

Shreddit

Shreddit is a Python command line program which will take a user's post history on the website Reddit, and will systematically go through the user's history deleting one post/submission at a time until only those whitelisted remain. It allows you to maintain your normal reddit account while having your history scrubbed after a certain amount of time.

When it became known that post edits were not saved but post deletions were saved, code was added to edit your post prior to deletion. In fact you can actually turn off deletion all together and just have lorem ipsum (or a message about Shreddit) but this will increase how long it takes the script to run as it will be going over all of your messages every run.

Important New Changes (as of Dec 2016)

Due to deprecation of the PRAW 3.x library, Shreddit is using PRAW 4. This requires that OAuth be used to authenticate. Thankfully, however, it is much easier than in previous versions. If you are upgrading, please review the usage section to ensure that you have set up credentials correctly.

Pip Installation

pip install -U shreddit will install the package and its dependencies, and it will add a shreddit command line utility to your PATH. This is typically either run in a virtualenv or using administrative privileges for global installation.

Manual Installation

  1. Clone the shreddit repository to a directory.
  2. From the directory, run pip install -r requirements.txt
  3. Run python setup.py install to install the package and the shreddit command line utility. This is typically either run in a virtualenv or using administrative privileges for global installation.

Usage

After installing the shreddit command line utility, the first step is setting up the tool's configuration files. Simply typing shreddit -g will generate configs. After configuring credentials, running the tool with the shreddit command will begin the tool's operation.

Configuring Credentials

Running shreddit -g will generate a blank praw.ini file that looks like this:

# Credentials go here. Fill out default, or provide one or more names and call shreddit with the -u option to specify
# which set to use.
[default]
client_id=
client_secret=
username=
password=

You must provide values for each of these. As strange as it may seem to provide both a username/password pair and a client id/secret pair, that is how the Reddit API does "OAuth" script applications.

Username and password are simply your Reddit login credentials for the account that will be used. However, to obtain the client ID and secret, follow these steps (taken from PRAW documentation):

  1. Open your Reddit application preferences by clicking here.
  2. Add a new application. It doesn't matter what it's named, but calling it "shreddit" makes it easier to remember.
  3. Select "script".
  4. Redirect URL does not matter for script applications, so enter something like http://127.0.0.1:8080
  5. Once created, you should see the name of your application followed by 14 character string. Enter this 14 character string as your client_id.
  6. Copy the 27 character "secret" string into the client_secret field.

Finally, your praw.ini should look like this (with fake data provided here):

[default]
client_id=f3FaKeD4t40PsJ
client_secret=dfK3pfMoReFAkEDaTa123456789
username=testuser
password=123passwordgoeshere123

Keep your praw.ini either in the current directory when running shreddit, or in one of the config folders described here such as ~/.config in Linux or %APPDATA% in Windows.

To use more than one account, you can add multiple profiles instead of just [default] and use the -u option to shreddit to choose which one each time.

Automating

The easiest way to automate this tool after the first run is by using the cron utility. Run crontab -e to edit your user's crontab settings.

Examples:

The following examples require that the PRAW configuration file is located in the config directory. See this PRAW documentation for more information.

  • Run every hour on the hour 0 * * * * shreddit -c <full path to shreddit.yml>

  • Run at 3am every morning 0 3 * * * shreddit -c <full path to shreddit.yml>

  • Run once a month on the 1st of the month 0 0 1 * * shreddit -c <full path to shreddit.yml>

If virtualenv was used, be sure to add source /full/path/to/venv/bin/activate && before the command. For example:

0 * * * * source /full/path/to/venv/bin/activate && shreddit -c <full path to shreddit.yml>

Command Line Options

$ shreddit --help
usage: app.py [-h] [-c CONFIG] [-g] [-u USER]

Command-line frontend to the shreddit library.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Config file to use instead of the default shreddit.yml
  -g, --generate-configs
                        Write shreddit and praw config files to current
                        directory.
  -u USER, --user USER  User section from praw.ini if not default

For Windows users

  1. Make sure you have Python installed. Click here for the Python download page. - Note: Install either python 2.x or python 3.x, not both.
  2. Follow the pip installation instructions.
  3. Open a new command prompt and verify that the shreddit command works before moving on to the usage section.

Caveats

  • Certain limitations in the Reddit API and the PRAW library make it difficult to delete more than 1,000 comments. While deleting >1000 comments is planned, it is necessary right now to rerun the program until they are all deleted.

  • We are relying on Reddit admin words that they do not store edits, deleted posts are still stored in the database they are merely inaccessible to the public.

Donate

A few people have asked about donating so here's a Bitcoin address, any amount is appreciated, spread amongst recent contributors and if there's enough interest a web service may be on the horizon! 1PbeYK7FonjVmgWxf4ieKmvwtomZR1K1Qu

shreddit's People

Contributors

bbigras avatar crypticgeek avatar denizdogan avatar disser avatar drewcrawford avatar fhriley avatar jc5 avatar kllmanu avatar kmwhite avatar leonfedotov avatar markrawlingson avatar noflag avatar pteek avatar rtgibbons avatar scott-hand avatar secondspass avatar skath avatar smjd avatar x89 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shreddit's Issues

Not working

Upon executing the script, following message shows in Terminal:

INFO:main:Logged in as [username]

No other messages are shown and when I visit reddit I see that none of the comments or submission are deleted.

Running El-Capitan. Followed the instructions on the repository homepage and satisfied all the Python requirements.

verbose is set to True.

[feature] proxy support

to allow users extra level of privacy I would like to suggest that Shreddit have support for proxy server. This would easy allow users to run Shreddit over Tor

This could be control with new section in .cfg. e..g.

[proxy]
ip: 127.0.0.1
port: 9051

Upgrade to PRAW 4

Pinging @scott-hand.

Okay so I've made a praw4 branch but I can immediately see the difficulties. praw.errors I suppose should be replaced with praw.exceptions if at all, it may be that just catching the long-names praw.whatever.SomeException works as well.

As I think you alluded toโ€ฆ https://github.com/praw-dev/praw/tree/praw4#quickstart there's a big problem here with OAuth being required and to be readable before even initialising the Reddit object. That's going to be fun.

I propose using the praw4 branch at least for making shreddit working praw4, it won't be being merged until there's a user friendly easy way for OAuth to be set up for the user.

I'm going to keep digging around with my own client details and see what else is broken.

UnicodeDecodeError

Traceback (most recent call last):
File "shreddit.py", line 195, in
remove_things(get_things())
File "shreddit.py", line 159, in remove_things
content = str(content).encode('ascii', 'ignore')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 25: ordinal not in range(128)

I'm getting this error for every account I've tried. Does anyone have a solution?

Just a User

Hey. Almost got the program running, but it seems I need a bit of code knowledge that I don't have and I'm not finding it in a readme. When I run shreddit in the virtualenv, I get the error

File "shreddit.py", line 140, in
replacement_text = get_sentence<>
NameError:name 'get_sentence' is not defined

I'm using python 34 and have updated pip and I believe I successfully updated virtualenv. Not sure. I also seem to be getting lost at "Run the following command, you must run this every time you wish to run the script source ./bin/activate" What command? I try to run ./bin/activate with python, can't find it. Source isn't recognized as a valid command by Windows or if I put it after python, it can't find "source." If it is the next command, the requirements.txt, then I was able to install that using pip.

Anyways, thanks for any of the help. I'm almost there! Sorry if this isn't the best place to ask. Github is really confusing for me. I just want to delete comments! :p

No module named errors

Traceback (most recent call last):
  File "/usr/local/bin/shreddit", line 7, in <module>
    from shreddit.app import main
  File "/usr/local/lib/python2.7/dist-packages/shreddit/app.py", line 10, in <module>
    from shreddit.shredder import Shredder
  File "/usr/local/lib/python2.7/dist-packages/shreddit/shredder.py", line 12, in <module>
    from praw.errors import (InvalidUser, InvalidUserPass, RateLimitExceeded,
ImportError: No module named errors

Wordlist in shreddit.yml fails as boolean

4e785dc#diff-770fb4bc83d8c1913fa4cfcb8e330100R76

~/Development/Shreddit master* 27s
โฏ shred-reddit
~/Development/Shreddit ~/Development/Shreddit
DEBUG:__main__:Logged in with OAuth.
INFO:__main__:Logged in as XXX.
DEBUG:__main__:Deleting messages before 2016-09-28 12:04:09.370784.
DEBUG:__main__:Keeping messages from subreddits programming
Traceback (most recent call last):
  File "./shreddit.py", line 91, in <module>
    if os.name == 'posix' and os.path.isfile(os_wordlist):
  File "/Users/mike/Development/Shreddit/bin/../lib/python2.7/genericpath.py", line 37, in isfile
    st = os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

However, when I change the config line to: wordlist: '' then it works perfectly.

Deploy to Heroku

It'd be awesome to have something like https://github.com/christophercliff/tweet-lifetime for this. So a simple deploy to heroku, enter basic config, and then your are off to the races.

If there is interest in this let me know and I'll try to help. I'm not a web python dev so I'm not sure what all would be needed. My guess is we could create a basic web server with a timer built in that executes "runners" to this script. From the above repo, they are using a plugin to keep the dyno from sleeping, and providing the uptime, other then that it just executes the runner as needed.

Another option would be to look at https://elements.heroku.com/addons/scheduler

This doesn't work

There is no activate script. Also instructions seem to be outdated and don't work on a mac

Doesn't seem to do anything?

I ran this on my account and while it produced lots of output that would indicate it was editing and deleting posts, my posts are all still there. Here is my yaml.


---
# Login details for Reddit. Fill out if you don't wish
# to be prompted for a login every time you run Shreddit.
### NOTE: This may be deprecated as you can specify in praw.ini instead
username: null
password: null

# How many hours of comments you want to keep
#24 hours in a day,
#168 hours in a week,
#672 hours in two fortnights,
#720 hours in a month (30 days),
#8766 hours in a year (365.25 days)
hours: 168

# Max score, higher than this and it'll be kept.
max_score: 10

# Options: new, top, controversial, more?
sort: new

# Enables print statements to notify you of what's going on
verbose: True

# Removes your vote before deleting the item
clear_vote: False

# Options: comments, submitted, overview
# See: https://github.com/mellort/reddit_api/blob/master/reddit/objects.py#L359
# Overview: both submissions and comments. Comments / Submitted are as expected
item: overview

# Anything in this list won't be deleted, coma delimited
# spaces should work as .strip() is called after splitting
# on comma.
whitelist: [java, AskScience, TheCulture, redditdev, programming, charity, netsec]

# If you want any specific posts to be whitelisted stick 'em in here
whitelist_ids: []

# If you set this then no editing or deleting will be done
# but the output from the program will be shown as an example
trial_run: False

# Don't delete but *do* edit, could prove... interesting to see a comment
# with 5000 upvotes and it's just a lorem ipsum!
edit_only: False

# Ignore distinguished comments.
whitelist_distinguished: True

# Ignore gilded (gold) comments
whitelist_gilded: True

# Delete everything older that this date, **This ignores whitelists**.
# Can be used as a second deletion, as in "delete items older than 24 hours
# except on whitelisted subreddits but after 3 months delete everything.
nuke_hours: 2160

# Save a copy to disk of comments and posts before deleting them.
keep_a_copy: True
save_directory: /tmp

# Replacement text format
# Defines what you want to edit deleted content with pre-deletion (to ensure
# it's not saved in their database).
# Default: Random string. But this can be detected as spam in some cases.
# options: [random, dot, "user entered string"]
replacement_format: dot

# vim: syntax=yaml ts=2

When you edit a comment, can you not replace it with spam?

I just used your script to remove all my comments and posts and it works like a clock.

Thank you for creating it.

However, in the process, I have tripped the alarm of /r/conspiracy (and potentially other subreddits too) and I have been banned from there.

I had to message them and explain to them what was going on, they were understanding and removed the ban.

So is it possible to not replace the original comments with spam? Just replace them with . (a dot) or would that change how reddit treats comments and edits?

Option to ignore distinguished comments?

I'd like to use your script to scrub my reddit history. The problem is that I have a lot of official distinguished comments as a moderator and those comments need to stay up. Could you add an option to ignore distinguished comments?

Thanks

Breaks when Encountering Accented Character is Post Title

I haven't confirmed this but, if I could venture a guess. The post being deleted had an "รข" in the title. I deleted it manually and shreddit was able to resume.

Traceback (most recent call last):
File "shreddit.py", line 143, in
url=thing.url)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 59: ordinal not in range(128)

Depreciation warning when running

/usr/local/lib/python2.7/dist-packages/praw/decorators.py:88: DeprecationWarning: reddit intends to disable password-based authentication of API clients sometime in the near future. As a result this method will be removed in a future major version of PRAW.

For more information please see:

Pass disable_warning=True to login to disable this warning.
warn(msg, DeprecationWarning)

SSL doesn't work

I had to shut it off in the reddit browser app, but otherwise it works great.

I forgot to grab the python stackdump for you, but it topped out in urllib3. Let me know if you need it.

Also, otherwise, I'm on a bass-ackwards Slackware box, so I installed pip and added your libs manually from my root shell and everything worked swimmingly!

Non-programmers

Any chance you could create an executable file so that people can use this without needing terminal experience? If not, can you create an ELI5 walkthrough of how to use it for those who aren't familiar with the terminal?

Should ignore clear_vote option for archived submissions

When trying to remove archived submissions and the clear_vote option is True, shreddit will error out with a cryptic error message. Shreddit should either detect this error and inform the user or ignore the clear_vote option and continue deleting.

[feature] allow for multiple account

I would like to see a feature added where shreddit will allow for easier work with multiple accounts.

I could see it work in two way

  1. shreddit username

in this instance you could add multiple [user] group to the .cfg file, this would essentially replace the [main] section so you can have full control over each user.

  1. shreddit user.cfg

in this instance you would have a config file per user and then tell shreddit which config to use.

ImportError: No module named errors

Not really sure why this is happening. I have tried uninstalling, reinstalling, manually importing modules. Every time I run Shreddit windows or Mac OS X I get errors. Any suggestions?

Traceback (most recent call last):
  File "/usr/local/bin/shreddit", line 9, in <module>
    load_entry_point('shreddit==2.0.0', 'console_scripts', 'shreddit')()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2697, in load_entry_point
    return ep.load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load
    return self.resolve()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2376, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build/bdist.macosx-10.12-intel/egg/shreddit/app.py", line 8, in <module>
  File "build/bdist.macosx-10.12-intel/egg/shreddit/shredder.py", line 12, in <module>
ImportError: No module named errors
Traceback (most recent call last):
  File "shredder.py", line 12, in <module>
    from praw.errors import (InvalidUser, InvalidUserPass, RateLimitExceeded,
ImportError: No module named errors

I have tried using python 2.7 and 3.5 with fresh installs of everything.

It just can't find the module named "errors"

Thanks for your help.

Edits/deletes ~100 comments and then stops

I started the process and it runs fine but only gets through ~100 comments/posts each time and then stops. I then have to restart it and it continues where it left off. It seems like the call things = r.user.get_overview(limit=None, sort=sort) isn't actually getting all of the comments/submissions for some reason...it looks like an issue with https://praw.readthedocs.org/en/v2.1.21/pages/code_overview.html#praw.__init__.BaseReddit.get_content in which get_content() isn't actually doing "If limit is None, then fetch as many entries as possible (reddit returns at most 100 per request, however, PRAW will automatically make additional requests as necessary)."

Is there a testing module or dummy dataset to replicate the issue perchance?

shreddit not seeing some pip modules again

So I've updated to the latest shreddit dated Sept 16 and run with the following errors

$ python3 ./shreddit.py -c shreddit.cfg
Traceback (most recent call last):
  File "./shreddit.py", line 8, in <module>
    from simpleconfigparser import simpleconfigparser
ImportError: No module named 'simpleconfigparser'

but SimpleConfigParser is installed

$ pip list | grep -i simple
SimpleConfigParser (0.1.0)

and with python 2 as it looks like the python3 requirement has been removed

$ ./shreddit.py -c shreddit.cfg
Traceback (most recent call last):
  File "./shreddit.py", line 56, in <module>
    whitelist_distinguished = config.getboolean('main', 'whitelist_distinguished')
  File "/usr/lib/python2.7/ConfigParser.py", line 369, in getboolean
    if v.lower() not in self._boolean_states:
AttributeError: 'NoneType' object has no attribute 'lower'

AttributeError: can't set attribute

Bit of a puzzling traceback for me on Py3.4:

$ python3 shreddit.py
Version 2.1.16 of praw is outdated. Version 2.1.18 was released 2 days ago.
Username: [user]
Password for [user]:
Logged in as [user]
Deleting messages before 2014-08-08 01:26:43.205966.
Keeping messages from subreddits askscience, theculture, redditdev, programming, charity
Traceback (most recent call last):
  File "shreddit.py", line 104, in <module>
    for thing in things:
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 471, in get_content
    page_data = self.request_json(url, params=params)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/decorators.py", line 161, in wrapped
    return_value = function(reddit_session, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 516, in request_json
    data = json.loads(response, object_hook=hook)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 331, in loads
    return cls(**kw).decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 359, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/__init__.py", line 403, in _json_reddit_objecter
    return object_class.from_api_response(self, json_data['data'])
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/objects.py", line 58, in from_api_response
    return cls(reddit_session, json_dict=json_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/objects.py", line 514, in __init__
    underscore_names=['replies'])
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/objects.py", line 72, in __init__
    self.has_fetched = self._populate(json_dict, fetch)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/objects.py", line 141, in _populate
    setattr(self, name, value)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/praw/objects.py", line 102, in __setattr__
    object.__setattr__(self, name, value)
AttributeError: can't set attribute
sys:1: ResourceWarning: unclosed <socket object at 0x104c9cc48>
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/importlib/_bootstrap.py:2150: ImportWarning: sys.meta_path is empty

shreddit failing on "import praw"

Since after I install shreddit from the current shreddit-master.zip the program does not work.

If run ./shreddit.py from the Shreddit Master directory I get the error

$ ./shreddit.py
Traceback (most recent call last):
  File "./shreddit.py", line 3, in <module>
    import praw
 ImportError: No module named 'praw'

But as with all version of shreddit I usd before praw has always been there

$ pip list | grep praw
praw (2.1.17)


$ python3 --version
Python 3.4.0

Running on Ubuntu 14.04

Issue using/installing on windows.

So I followed the windows installation instructions installing python 3.5, but the command pip install -U shreddit-master.zip didn't work as it couldn't find pip in the path (even after a system reboot).

I did some searching and found it in the python/scripts folder. I navigated to there and used the command to avoid path issues, which seemed to do some stuff saying it'd installed shreddit & downloading some dependencies.

I then tried looking for the shreddit.yml.example to edit as per your instructions, but couldn't. So I tried editing it in the zip file and then installing again, no luck.

To get it finding the shreddit.yml I pasted a copy into the python/scripts folder and it found it, but then gave me the error of

yaml.parser.ParserError: expected '<document start>', but found '<block mapping
start>'
  in "shreddit.yml", line 14, column 1

So I'm not sure how to get over this hurdle. Any ideas?

Error

(Shreddit)[root@dev01 Shreddit]# ./run.sh
Requirement already up-to-date: praw in ./lib/python2.7/site-packages
Requirement already up-to-date: update-checker>=0.11 in ./lib/python2.7/site-packages (from praw)
Requirement already up-to-date: requests>=2.3.0 in ./lib/python2.7/site-packages (from praw)
Requirement already up-to-date: decorator>=3.4.2 in ./lib/python2.7/site-packages (from praw)
Requirement already up-to-date: six>=1.4 in ./lib/python2.7/site-packages (from praw)
/root/Shreddit/lib/python2.7/site-packages/praw/decorators.py:74: DeprecationWarning: reddit intends to disable password-based authentication of API clients sometime in the near future. As a result this method will be removed in a future major version of PRAW.

For more information please see:

Pass disable_warning=True to login to disable this warning.
warn(msg, DeprecationWarning)
Traceback (most recent call last):
File "./shreddit.py", line 95, in
login(user=_user, password=_pass)
File "./shreddit.py", line 84, in login
r.login(_user, _pass)
File "", line 2, in login
File "/root/Shreddit/lib/python2.7/site-packages/praw/decorators.py", line 75, in wrap
return function(_args, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/praw/init.py", line 1418, in login
self.request_json(self.config['login'], data=data)
File "", line 2, in request_json
File "/root/Shreddit/lib/python2.7/site-packages/praw/decorators.py", line 113, in raise_api_exceptions
return_value = function(_args, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/praw/init.py", line 612, in request_json
retry_on_error=retry_on_error)
File "/root/Shreddit/lib/python2.7/site-packages/praw/init.py", line 444, in _request
response = handle_redirect()
File "/root/Shreddit/lib/python2.7/site-packages/praw/init.py", line 425, in handle_redirect
verify=self.http.validate_certs, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/praw/handlers.py", line 148, in wrapped
result = function(cls, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/praw/handlers.py", line 57, in wrapped
return function(cls, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/praw/handlers.py", line 103, in request
allow_redirects=False, verify=verify)
File "/root/Shreddit/lib/python2.7/site-packages/requests/sessions.py", line 579, in send
r = adapter.send(request, *_kwargs)
File "/root/Shreddit/lib/python2.7/site-packages/requests/adapters.py", line 369, in send
timeout=timeout
File "/root/Shreddit/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/root/Shreddit/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib64/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/root/Shreddit/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 216, in sendall
data = memoryview(data)
TypeError: cannot make memory view because object does not have the buffer interface

Clearing upvote/downvote history on submissions?

It might be somewhat overkill or unnecessary, but the PRAW API let's you clear your voting history on reddit submissions (at least the last 6 months worth of voting, because anything after that is considered "archived" and can't have votes changed). I whipped up a short little method that can clear either only upvotes, only downvotes, or both.

Would this be something worth opening a PR for? And if so, do you have any style preferences or testing requirements?

OAuth get_secret.py Upgrade

Running get_secret.py gives a syntax error.
I might just be an idiot, but I can't seem to find a way around this. I'd like to upgrade to OAuth before it stops being supported by Reddit.

python get_secret.py 
Traceback (most recent call last):
File "get_secret.py", line 18, in <module>
    import asyncio
  File "/usr/local/lib/python2.7/dist-packages/asyncio/__init__.py", line 9, in <module>
    from . import selectors
  File "/usr/local/lib/python2.7/dist-packages/asyncio/selectors.py", line 39
    "{!r}".format(fileobj)) from None
                               ^
SyntaxError: invalid syntax

OAuth2 not working

I'm just missing something really obvious probably, but I've stared at this for 30 minutes now and I still haven't figured it out.
I did the setup process for OAuth2, but it still seems to be using the username and password in my config.yml file. If I leave those empty it will prompt me for my username and pw.

Notes:

  • If I specify in my config.yml everything works fine
  • My praw.ini setup went succesfully, I have a oauth_refresh_token in praw.ini

It just seems to me that I need to add a argument to make it use the praw.ini OAuth2 data instead of the config.yml username and password data.

What am I doing wrong?

shredditcfg.yml:


username:
password:
hours: 168
max_score: 300
sort: new
verbose: True
clear_vote: False
item: overview
whitelist: [AskScience, TheCulture, redditdev, programming, charity, netsec]
whitelist_ids: []
trial_run: False
edit_only: False
whitelist_distinguished: True
whitelist_gilded: True
nuke_hours: 720
keep_a_copy: True
save_directory: C:\Users\Rubeb\Documents\Reddit backup
replacement_format: "Leave no evidence"
vim: syntax=yaml ts=2

Setting edit_only=True seems to get you banned from certain subreddits

I have no reason to suspect edit_only=True is the culprit other than intuition, but i imagine it is tripping a bot detector. It is the only flag from the default yml that i changed. If that's not it, then shreddit could be problematic for some users.

I'm just experimenting on a garbage account, so it doesn't really matter, but so far i've been banned from /r/worldnews and /r/gadgets

And for what it's worth, here's the rest of the config yml:

hours: 0
max_score: 99999999
sort: new
verbose: True
clear_vote: False
item: overview
whitelist: [AskScience, TheCulture, redditdev, programming, charity, netsec]
whitelist_ids: []
multi_blacklist: []
multi_whitelist: []
trial_run: False
edit_only: True
whitelist_distinguished: True
whitelist_gilded: True
nuke_hours: 0
keep_a_copy: False
save_directory: /tmp
replacement_format: random
debug: DEBUG
wordlist: []
batch_cooldown: 10

Comments not saved in trial run

I am trying a trial run of shreddit. Used OAuth2/ created a script linked to my reddit account, and the warning messages about authentication went away. But during the trial run, it seems much too fast (like 15 seconds or less), and no comments are saved in the save directory I specified in my config. Is this because it doesn't access the web at all during a trial run? Or is something else not working?

I did save all my comments by hand, just in case, but I really would like getting a better save in .json format from the script.

Thanks!

How to get password-based authentication of API?

It's not clear based on the information provided how to remove the warning. The error says normal login will be disabled, but the reddit thread link states:

we will not be forcing the swap.

I understand why 2 factor is important, but I'm not sure what steps I need to take to not get the warning at all.

Secondly, I'm not sure how to disable the warning. Where do I pass the flag? I've tried in both my config file and praw.ini and I'm still getting the warning.

Interest in some changes I made to support AWS Lambda

I really like this project, and I wanted to find an easy way to schedule it to run, and so I created a version that can be called using AWS Lambda scheduled events. The best part about it is that it is almost certainly guaranteed to use well under the limit for the Lambda Free Tier, meaning that users will be able to run it for free without managing any infrastructure.

The catch is that to make it easy and clean to use with AWS's Python deployment approach, I restructured things to make it a more standard Python package layout with a shred module that can be called programmatically. This also made it easier to do things like allowing users to specify the praw.ini file explicitly (and either multiple praw.ini files or multiple site entries in the one file will allow for multiple users to be handled). It also creates a nice "shreddit" script in the virtualenv's bin/ folder.

Here's the fork: https://github.com/scott-hand/Shreddit

I know it's a lot of structural changes, so I just wanted to see if you'd be interested in integrating them. If so, I'll make sure things are cleaned up and submit a PR. Otherwise that's totally cool. I can always spin off my own and cite you and your license file as contributors, and that would be ideal if you didn't want all the structural change introduced.

Crashing after a while

Traceback (most recent call last): File "shreddit.py", line 198, in <module> count = remove_things(things) File "shreddit.py", line 187, in remove_things thing.edit(replacement_text) File "/usr/local/lib/python2.7/dist-packages/praw/decorators.py", line 348, in wrapped return function(cls, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/praw/objects.py", line 308, in edit response = self.reddit_session.request_json(url, data=data) File "/usr/local/lib/python2.7/dist-packages/praw/decorators.py", line 173, in wrapped return_value = function(reddit_session, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/praw/__init__.py", line 579, in request_json retry_on_error=retry_on_error) File "/usr/local/lib/python2.7/dist-packages/praw/__init__.py", line 424, in _request _raise_response_exceptions(response) File "/usr/local/lib/python2.7/dist-packages/praw/internal.py", line 196, in _raise_response_exceptions raise Forbidden(_raw=response) praw.errors.Forbidden

Error - "expected '<document start>', but found '<block mapping start>"

Getting this error when I try to run Shreddit -

yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>' in "settings.yml", line 14, column 1

Which is the "hours" line.

Any help would be appreciated!


# Login details for Reddit. Fill out if you don't wish
# to be prompted for a login every time you run Shreddit.
### NOTE: This may be deprecated as you can specify in praw.ini instead
username:<removed by me for this post>
password:<removed by me for this post>

# How many hours of comments you want to keep
#24 hours in a day,
#168 hours in a week,
#672 hours in two fortnights,
#720 hours in a month (30 days),
#8766 hours in a year (365.25 days)
hours: 24

# Max score, higher than this and it'll be kept.
max_score: 400

# Options: new, top, controversial, more?
sort: new

# Enables print statements to notify you of what's going on
verbose: True

# Removes your vote before deleting the item
clear_vote: False

# Options: comments, submitted, overview
# See: https://github.com/mellort/reddit_api/blob/master/reddit/objects.py#L359
# Overview: both submissions and comments. Comments / Submitted are as expected
item: overview

# Anything in this list won't be deleted, coma delimited
# spaces should work as .strip() is called after splitting
# on comma.
whitelist: [<removed by me for this post]

# If you want any specific posts to be whitelisted stick 'em in here
whitelist_ids: []

# If you set this then no editing or deleting will be done
# but the output from the program will be shown as an example
trial_run: False

# Don't delete but *do* edit, could prove... interesting to see a comment
# with 5000 upvotes and it's just a lorem ipsum!
edit_only: False

# Ignore distinguished comments.
whitelist_distinguished: True

# Ignore gilded (gold) comments
whitelist_gilded: True

# Delete everything older that this date, **This ignores whitelists**.
# Can be used as a second deletion, as in "delete items older than 24 hours
# except on whitelisted subreddits but after 3 months delete everything.
nuke_hours: 720

# Save a copy to disk of comments and posts before deleting them.
keep_a_copy: False
save_directory: /tmp

# Replacement text format
# Defines what you want to edit deleted content with pre-deletion (to ensure
# it's not saved in their database).
# Default: Random string. But this can be detected as spam in some cases.
# options: [random, dot, "user entered string"]
replacement_format: random

# vim: syntax=yaml ts=2


HTTPS Support

Just a heads up - this is more of a praw issue than Shreddit (Unless it has an option to specify https - I didn't check); however if you have HTTPS enabled in Reddit preferences, then praw will crap out and the script will fail when attempting to login.

missing module

$ python shreddit.py -c shreddit.cfg
Traceback (most recent call last):
  File "shreddit.py", line 8, in <module>
    from simpleconfigparser import simpleconfigparser
ImportError: No module named simpleconfigparser

I'm on a mac. Followed instructions (twice). Still get a missing module.

Argument of type NoneType is not iterable.

Hi!

Ran into this bug. No matter what I try. Here's my full output:

james@ubuntu:~$ cd shreddit/
james@ubuntu:~/shreddit$ virtualenv .
Running virtualenv with interpreter /usr/bin/python2
New python executable in ./bin/python2
Not overwriting existing python script ./bin/python (you must use ./bin/python2)
Installing setuptools, pip...done.
james@ubuntu:~/shreddit$ source ./bin/activate
(shreddit)james@ubuntu:~/shreddit$ pip install -r requirements.txt
Requirement already satisfied (use --upgrade to upgrade): praw==2.1.21 in ./lib/python2.7/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): requests==2.6.0 in ./lib/python2.7/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): SimpleConfigParser==0.1.0 in ./lib/python2.7/site-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./lib/python2.7/site-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): update-checker==0.11 in ./lib/python2.7/site-packages (from -r requirements.txt (line 5))
Cleaning up...
(shreddit)james@ubuntu:~/shreddit$ python shreddit.py -c shreddit.cfg
Logged in as StellaHunter
Deleting messages before 2015-04-09 05:03:14.650545.
Keeping messages from subreddits
Traceback (most recent call last):
  File "shreddit.py", line 150, in <module>
    if 'body_html' in d:
TypeError: argument of type 'NoneType' is not iterable

Failing over HTTPS

Heya,

I noticed this tool is failing when you force https to be on. I'm not the python hero this tool needs; could you look at this?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.