derrezed-fez/trending-topic-weaponization

Monitor trending topics over a period of time (e.g., 2 months?) to analyze and characterize how the terms are weaponized to conduct malicious activities on the Internet.

To pull and install Python library requirements: pip3 install -r requirements.txt

This project is meant to be installed on a server so that it can be run on a continuous basis. The main.py file should be executed from a CRON job daily. It has been throttled to not exceed the limits of the Virus Total API.

To run the data collection manually, run python -m main.py Be sure to input your phone number into the Twilio entry points to receive text notifications of data collection.

Database Structure

MongoDB Commands

If you are using Windows or Mac, install a viewer like Studio 3T. If using Linux, the following commands may help.

1. To start the MongoDB instance: sudo service mongod start
2. To stop the MondoDB instance: sudo service mongod stop
3. To restart eh MongoDB instance: sudo service mongod restart
4. To enter the Mongo shell, simply enter: mongo
5. Once in Mongo shell, show DBs by entering: show dbs
6. To export a Mongo database or collection, enter: mongodump -d <database_name> -o <directory_backup>
7. To import a Mongo database or collection, enter: mongorestore -d <database_name> <directory_backup>
8. To export as JSON, enter: mongoexport --db DB_NAME --collection COLLECTION_name --fields Field_name(s) --type=[csv or JSON} --out=Name-Path-Output-File

To use SCP to copy a Mongo DB from the remote server to your local machine: scp -r [email protected]:/path/to/remote/source /path/to/local/destination

trending-topic-weaponization's Issues

Migrate Dashboard and Data Collection Scripts into one Repository

Put both components of the code in one repository for simplicity.

Display IP Address information Geographically on a World Map

Create a system to look up URL and correlate to an IP address, then correlate the IP address to a general location on the world map. Create circular vectors show magnitude of weaponized IPs.

Unit Test Data Collection Scripts

Unit testing requires that each functional block of code be testing for any conditions it may be subjected to. All logic should be evaluated for a good, a bad, and any case that requires execution of certain logic in any functions that exist within each file.

Here is a good reference:
https://smartbear.com/learn/automated-testing/what-is-unit-testing/#:~:text=A%20unit%20test%20is%20a,of%20the%20definition%20is%20important.

Create a UI Interface for Searching URLs on Bing

Use a library to interact with the Bing browser and verify URL status pulled directly from VirusTotal.

Pull Data from Database into the Dashboard view

Aggregate all of the data from the MongoDB and pull into application in JSON format so it can be used to display data in the application.