ecstatic-nobel/osweep

Don't Just Search OSINT. Sweep It.

[OSweep™]

Don't Just Search OSINT. Sweep It.

Description

If you work in IT security, then you most likely use OSINT to help you understand what it is that your SIEM alerted you on and what everyone else in the world understands about it. More than likely you are using more than one OSINT service because most of the time OSINT will only provide you with reports based on the last analysis of the IOC. For some, that's good enough. They create network and email blocks, create new rules for their IDS/IPS, update the content in the SIEM, create new alerts for monitors in Google Alerts and DomainTools, etc etc. For others, they deploy these same countermeasures based on provided reports from their third-party tools that the company is paying THOUSANDS of dollars for.

The problem with both of these is that the analyst needs to dig a little deeper (ex. FULLY deobfuscate a PowerShell command found in a malicious macro) to gather all of the IOCs. And what if the additional IOC(s) you are basing your analysis on has nothing to do with what is true about that site today? And then you get pwned? And then other questions from management arise...

See where this is headed? You're about to get a pink slip and walked out of the building so you can start looking for another job in a different line of work.

So why did you get pwned? You know that if you wasted time gathering all the IOCs for that one alert manually, it would have taken you half of your shift to complete and you would've got pwned regardless.

The fix? OSweep™.

Prerequisites

Before getting started, ensure you have the following:
Ubuntu 18.04+

• Python 2.7.14 ($SPLUNK_HOME/bin/python)
• Splunk 7.1.3+
• Deb Packages
  • gcc
  • python-pip

CentOS 7+

• Python 2.7.14 ($SPLUNK_HOME/bin/python)
• Splunk 7.1.3+
• Yum Packages
  • epel-release
  • gcc
  • python-pip

Optional Packages

• Git

Click HERE to get started.

Gallery

OSINT Sweep - Dashboard

Certificate Search - Dashboard

CyberCrime Tracker - Dashboard

GreyNoise - Dashboard

Hybrid-Analysis - Dashboard

MalShare - Dashboard

Phishing Catcher - Dashboard

Phishing Kit Tracker - Dashboard

Pastebin Dump - Dashboard

ThreatCrowd - Dashboard

Twitter - Dashboard

URLhaus - Dashboard

urlscan.io - Dashboard

Dashboards Coming Soon

• Alienvault
• Censys
• PulseDive

Please fork, create merge requests, and help make this better.

osweep's Issues

Module Suggestion - Criminal IP

Please add Criminal IP!

https://www.criminalip.io/ - Criminal IP is a specialized Cyber Threat Intelligence (CTI) search engine that allows users to search for various security-related information such as malicious IP addresses, domains, banners, etc. It can be widely integrated as it offers diverse features.

Greynoise Not working

Hi.

I configured this app on Splunk and been using it for long, its great! Thanks a lot for your wokr!

Although, I never got to work Greynoise section, it just looks like its not working in the config.

I have a full API developer key that is fully working (have a greynoise app in splunk that works with the same API Key), so im certain I comply with API query limits. Could you possibly check if the Greynoise configuration has any issues in your app?

Thanks again, Im willing to help you if needed!

Search is waiting for input...

hi @ecstatic-nobel i setup your project its really cool but got this error Search is waiting for input...
what should i need to do ?

Lookup tables are invalid.

Hi, i'm having a bit trouble with "osweep".
I'm running Splunk on MacOS as an administrator, followed osweep setup instructions.
I don't see any data on Splunk and i get "The lookup tale 'XXXXX' is invalid (see attached screen grub).
Any ideas?

Thanks in advance.

sudo pip install -r requirements.txt -t _tp_modules/

i have this problem when installing the requirements some help please

root@nulled-virtual-machine:/opt/splunk/etc/apps/OSweep/bin# sudo rm -rf _tp_modules/*
root@nulled-virtual-machine:/opt/splunk/etc/apps/OSweep/bin# sudo pip install -r requirements.txt -t _tp_modules/
Collecting bs4
Using cached bs4-0.0.1.tar.gz (1.1 kB)
ERROR: Could not find a version that satisfies the requirement entropy (from -r requirements.txt (line 2)) (from versions: none)
ERROR: No matching distribution found for entropy (from -r requirements.txt (line 2))
root@nulled-virtual-machine:/opt/splunk/etc/apps/OSweep/bin#

/OSweep/lookups/greynoise_feed.csv missing

First of all thanks for the great work.

Tested on ubuntu and on centos, both install cannot get GreyNoise working.

command="greynoise", Error: Traceback (most recent call last): File "/opt/splunk/etc/apps/OSweep/bin/commons.py", line 84, in return_results new_results = module.process_iocs(None) File "/opt/splunk/etc/apps/OSweep/bin/greynoise.py", line 127, in process_iocs open_file = open("{}/greynoise_feed.csv".format(lookup_path), "r") IOError: [Errno 2] No such file or directory: '/opt/splunk/etc/apps/OSweep/lookups/greynoise_feed.csv'

CentOS requirements

Although is not an issue itself, I apologize but I don't know other place to share this info for the people after do an intallation in a different Linux Platform.

Prerequites:
Also is required Python-devel package
If you are newbie on CentOS, keep in mind you need also to install "epel-release" and python-pip packages too

Considerations:
When you clone using git, is not required to move the folder from owseep-master to osweep as it is beign downloaded as osweep
After each update, installation or any change review and change the owner of all files on the splunk intallation (solves initial issues)
Most of the issues will come from the Splunk configuration rather from an app configuration or deployment, so it's useful to check the splunk log file on [splunk_installation_folder]/var/log/splunk/ (mainly splunkd.log)

entropy package missing

Hello,
I'm still using python 2.7, but it seems entropy package disappeared from pip repository.
Is there any other place where to download it?

thanks,
Fausto

Issue SPLUNK_HOME

Hi!
I have tried to occupy the application, but the application gives me an error. First it does not show me data and when I try to execute some script of the available modules, it gives me an error in the following environment variable:

Traceback (most recent call last):
File "./malshare.py", line 32, in
app_home = "{}/etc/apps/OSweep".format(os.environ['SPLUNK_HOME'])
File "/opt/splunk/lib/python2.7/UserDict.py", line 40, in getitem
raise KeyError(key)
KeyError: 'SPLUNK_HOME'

GreyNoise API configuration issue

Hi,

I've a GreyNoise API key and I want to test it on the platform, but as soon as I set it up on the config.py file, the APP does not work, is not able to download any feed. There is required any format, quotations or something like that?

PS: I would like to add also the twitter API ones, but firts I prefer to identify how to configure it

Thanks in advance