d-rickyy-b/pastepwn

Python framework to scrape Pastebin pastes and analyze them

pastepwn - Paste-Scraping Python Framework

Pastebin is a very helpful tool to store or rather share ascii encoded data online. In the world of OSINT, pastebin is being used by researchers all around the world to retrieve e.g. leaked account data, in order to find indicators about security breaches.

Pastepwn is a framework to scrape pastes and scan them for certain indicators. There are several analyzers and actions to be used out-of-the-box, but it is also easily extensible - you can create your own analyzers and actions on the fly.

Please note: This framework is not to be used for illegal actions. It can be used for querying public Pastebin pastes for e.g. your username or email address in order to increase your own security.

⚠️ Important note

In April 2020 Pastebin disabled access to their scraping API for a short period of time. At first people weren't able to access the scraping API in any way, but later on they re-enabled access to the API setup page. But since then it isn't possible to scrape "text" pastes. Only pastes with any kind of syntax set. That reduces the amount of pastes to a minimum, which reduced the usefulness of this tool.

Setting up pastepwn

To use the pastepwn framework you need to follow these simple steps:

1. Make sure to have a Pastebin premium account!
2. Install pastepwn via pip (pip3 install pastepwn)¹
3. Create a file (e.g. main.py) in your project root, where you put your code in²
4. Fill that file with content - add analyzers and actions. Check the example implementation.

¹ Note that pastepwn only works with python3.6 or above
² (If you want to store all pastes, make sure to set up a mongodb, mysql or sqlite instance)

Behind a proxy

There are 2 ways to use this tool behind a proxy:

• Define the following environment variables: HTTP_PROXY, HTTPS_PROXY, NO_PROXY.
• When initializing the PastePwn object, use the proxies argument. proxies is a dict as defined in requests' documentation.

Troubleshooting

If you are having troubles, check out the wiki pages first. If your question/issue is not resolved there, feel free to create an issue or contact me on Telegram.

Roadmap and ToDos

Check the bug tracker on GitHub to get an up-to-date status about features and ToDos.

• REST API for querying paste data (will be another project)
• Add a helpful wiki with instructions and examples

pastepwn's Issues

Twitter Action

Similar to the good old dumpmon on Twitter it would be lovely to have a Twitter action which sends found pastes to twitter.

To add a bit more context:

To solve this issue there needs to be a new action added in the actions directory. The action must follow the example of the other actions in this directory. Don't forget to add the action to the action package file.

The most simple example for such an action is the GenericAction which executes a passed function.

Tests are not necessary but highly appreciated. If there are questions, don't hesitate to contact me.

Credit Card data analyzer

A regex analyzer which matches on credit card data

Refactor templating for messages

Currently each action itself needs to implement the templating. This should be moved to a paste class. Best would be to put it into some kind of abstract base class.

I don't think that this issue is suitable for hacktoberfest. If you feel different, you can create pull requests for it. But I can't promise to merge them, because I have very exact conceptions about how this should work.

Blacklist for Wordanalyzer

Maybe it would be helpful to add words to a blacklist to specify the wordanalyzer better

Tags for pastes in the database

Implement WebhookAction

Email and password pair analyzer

There should be a new analyzer to find pastes which contain email/password pairs.

To solve this issue there needs to be a new analyzer added in the analyzers directory. The analyzer must follow the example of the other analyzers in this directory. Don't forget to add the analyzer to the analyzer package file.

The most simple example for such an analyzer is the AlwaysTrueAnalyzer which always returns True.

This analyzer should match email/password pairs. Check out this example paste. Best would be to subclass the regex analyzer.

Tests are not necessary but highly appreciated. If there are questions, don't hesitate to contact me.

API for accessing data from the server

Since the Pastebin API only allows for one IP to be entered there should be an pastepwn API which allows for searching through pastes or getting statistics. Also it would be great if the application can be controlled via the API (e.g. adding new analyzers on the fly).

Discord Action

"Request" class singleton

The request class should be made a singleton in order to be able to set proxy settings only once.

This must be done before merging #33

URL Analyzer

Additionally to that a subclass "PastebinURLAnalyzer" would be great to match pastebin urls inside pastes.

Add PastebinScraper by default, since it's currently the only scraper

When the scraping_handler.scrapers list is empty and the start method (see below) is being called, a new instance of PastebinScraper should be initialized and added via self.add_scraper(PastebinScraper) in line 97.

pastepwn/pastepwn/core/pastepwn.py

Lines 92 to 100 in 641afb3

	def start(self):
	"""Starts the pastepwn instance"""
	if self.__exception_event.is_set():
	self.logger.error("An exception occured. Aborting the start of PastePwn!")
	exit(1)
	self.scraping_handler.start()
	self.paste_dispatcher.start()
	self.action_handler.start()

Improve analyzer identifier

There needs to be some way to store an identifyer for a certain analyzer to know why a paste was matched.

Private Key Analyzer

Match on -----BEGIN PRIVATE KEY-----

Do not save paste if it was deleted already

If the paste's body is Error, we cannot find this paste., it means that the paste was already deleted, before we could download it.

In that case we should not store, nor process it!

Add possibility to force use IPv4 or IPv6

If there are IPv4 and v6 available, the user should be able to force use one of them.

Create unit tests

There are no tests yet. The code could have several bugs which are not found due to missing tests.

MISP Action

Would be nice to be able to send alerts to MISP.

Fix Action imports

As just done for the analyzers the actions should import the local files (from .basicAction import BasicAction) to prevent issues regarding import loops/conflicts.

Error / Exception handler

Users might want to execute an action when the application crashes. Add proper support for an exception handler.

Database Dump Analyzer

There should be a new analyzer to find pastes which contain a database dump.

To solve this issue there needs to be a new analyzer added in the analyzers directory. The analyzer must follow the example of the other analyzers in this directory. Don't forget to add the analyzer to the analyzer package file.

The most simple example for such an analyzer is the AlwaysTrueAnalyzer which always returns True.

This analyzer should match different kinds of database dumps. There are different formats for database dumps. It's up to you how many of those you want to implement. On haveibeenpwned.com there is a nice explanation on database dumps in pastes which should serve as a source for ideas.

Tests are not necessary but highly appreciated. If there are questions, don't hesitate to contact me.

Wrapper around various hash analyzers

Users might not want to calculate their password hashes beforehand and thus might want to be able to use a wrapper (analyzer) around various hash analyzers. They initialize that wrapper with their password in the clear. The wrapper generates the hashes on the fly and checks each paste against those hashes.

I am not sure if that's a good idea but I'll leave this here for now.

IBAN Analyzer

There should be a way to analyze for IBANs.

The analyzer should inherit from the BasicAnalyzer or RegexAnalyzer class and should implement a method match which returns True if the paste.body contains a IBAN.

The constructor of the IBANAnalyzer should contain a parameter validate=False which can be set in order to validate the found IBAN.

Pastes missing content

Description
Some pastes fetched from pastebin lack the paste's body - it will only contain the following text: File is not ready for scraping yet. Try again in 1 minute.

If this happens, the paste must be added back to the scraping queue and be scraped again.

E-Mail action

There should be a possibility to send emails on certain pastes.

Add Travis

Automatic tests and deployment are very nice

• Testing
• PyPi module build
• New GitHub Releases

Proxy settings for request

Currently it's not possible to run this framework behind a proxy

Improve telegram message by telegramAction

ActionHandler

There needs to be another thread handling the triggered actions from the analyzers

API method: Scrape specific paste

There should be an API method to request the scraping of a specific paste

API method: Fetch pastes in specific time frame

The API should be able to return a specific amount of pastes from within a specific time frame.

Example Call:
/pastes/findByDate

Parameters:

• from_time
• to_time

Add custom user agent to requests

Pass analyzer to action somehow

The action should know why/by what it was triggered

Slack Action

Send pastes to Slack channels

Database Connector analyzer

Match on database connection strings such as mongodb://ip

API method: switch on/off pastepwn

Switch on/off PastePwn

Yara Analyzer

Matching pastes with Yara rules could be interesting.

https://techanarchy.net/2017/09/hunting-pastebin-with-pastehunter/

Add license

There is no license on the code yet

On start method

There are users which would like to perform an action when pastepwn is fully initialized and running. There should be a way to register a handler for that.

Check paste insertion code of database adapters

Check if an error is thrown if a duplicate paste ID is inserted into the db (and catch it otherwise or refactor code to be an upsert).

Create docker-compose.yml file

Currently there is no (working) docker image available. The goal is to have a docker image + a docker-compose file which automatically starts a mongodb & pastepwn or a mysql & pastepwn or sqlite & pastepwn.

For that we need to read environment variables.

Syslog Action

Implement an action which sends matched pastes to syslog.

Hash Analyzer

Add a Hash Analyzer which matches certain password hashes. Can inherit from RegexAnalyzer

RegexAnalyzers should return what they find, instead of just indicating whether or not they found it.

I'm not sure whether this idea is compatible with your current idea of how this tool should be used, but it strikes me as odd that the RegexAnalyzers simply report whether or not a match was found, rather than returning all the data they were able to match.

For the PastebinURLAnalyzer I just made a pull request for, for example, I imagine it might be useful if you could feed it a number of pastes, and it could create some sort of dictionary which mapped the paste it had found a match on with a list of all URLs it was able to find.

That way, I could say "check out these 200 pastes and show me all the emails, pastebin urls, and bcrypt password hashes you find."

Just an idea, of course. Would probably require a bit of a redesign of the way the analyzers are used, but would require minimal changes to the actual analyzers.

Phone number analyzer

There should be a new analyzer to find pastes which contain phone numbers.

To solve this issue there needs to be a new analyzer added in the analyzers directory. The analyzer must follow the example of the other analyzers in this directory. Don't forget to add the analyzer to the analyzer package file.

The most simple example for such an analyzer is the AlwaysTrueAnalyzer which always returns True.

This analyzer should match phone numbers in the international format. Best would be to subclass the regex analyzer and create a regex to detect phone numbers. Make sure to also detect whitespace separated country codes with \s*. Also please check if the country codes and lengths are valid for a phone number.

Tests are not necessary but highly appreciated. If there are questions, don't hesitate to contact me.

Analyzers: Thread Pool

It's important to quickly analyze the pastes. Currently each paste is being processed in a single thread. It would be important to implement a thread pool which limits the amount of concurrent execution.

See https://www.metachris.com/2016/04/python-threadpool/

[MySQL] Auto-create database. if it doesn't exist yet.

If the specified database does not exist yet, create it for the user.

That seems to have several issues with SQLI... let's see.

Create pypi module

so that pastepwn can be installed via pip

Database imports

1. There are missing imports in the requirements.txt

2. Currently when using pastepwn a user needs to download ALL the db connector packages (mongo & mysql currently). Depending on the amount of db connectors which at some point will be supported by pastepwn, it might be stupid to have them all in the same package.

---

Possible solution(s):

1. try-except the error and return an error message. This is not elegant but allows everything to be bundeled in one package.
2. move the database connectors to a different package. That way it's way cleaner but users need to install multiple packages.

Pass exception event to the scrapers, analyzers and actions

This is currently not possible but should be.