lockfale/osint-framework

OSINT Framework

OSINT Framework

http://osintframework.com

Notes

OSINT framework focused on gathering information from free tools or resources. The intention is to help people find free OSINT resources. Some of the sites included might require registration or offer more data for $$$, but you should be able to get at least a portion of the available information for no cost.

I originally created this framework with an information security point of view. Since then, the response from other fields and disciplines has been incredible. I would love to be able to include any other OSINT resources, especially from fields outside of infosec. Please let me know about anything that might be missing!

Please visit the framework at the link below and good hunting!

https://osintframework.com

Legend

(T) - Indicates a link to a tool that must be installed and run locally
(D) - Google Dork, for more information: Google Hacking
(R) - Requires registration
(M) - Indicates a URL that contains the search term and the URL itself must be edited manually

For Update Notifications

Follow me on Twitter: @jnordine - https://twitter.com/jnordine
Watch or star the project on Github: https://github.com/lockfale/osint-framework

Suggestions, Comments, Feedback

Feedback or new tool suggestions are extremely welcome! Please feel free to submit a pull request or open an issue on github or reach out on Twitter.

Contribute with a GitHub Pull Request

For new resources, please ensure that the site is available for public and free use.

1. Update the arf.json file in the format shown below. If this isn't the first entry for a folder, add a comma to the last closing brace of the previous entry.

{
  "name": "Example Name",
  "type": "url",
  "url": "http://example.com"
}

2. Submit pull request!

Thank you!

OSINT Framework Website

https://osintframework.com

Happy Hunting!

osint-framework's Issues

Add SameIP.org

SameIP.org - Reverse IP Lookup

IP Address -> Neighbor Domains:

            {
              "id": "<ID>",
              "name": "SameIP",
              "type": "url",
              "url": "https://sameip.org/"
            },

Another Resource - Historical DNS

Sent you a tweet on twitter as well. Thought you might look at including this site for historical DNS research.
https://securitytrails.com/

Consider adding a license

I know we already spoke about this but I wanted to make another issue to remind you and because I'm not using Github enough.

arf.json editor

Hi again

What is a good editor for json files?

I find it really hard to edit this file with notepad++

Thanks !!

Request to add Numbering Plans

Please add to a new International folder.
(Telephone Numbers) > (International) > (Numbering Plans)

{
  "id": "<ID>",
  "name": "Numbering Plans",
  "type": "url",
  "url": "https://www.numberingplans.com/?page=analysis&sub=phonenr"
},

more map services

transportation

• http://www.openseamap.org
• https://www.openrailwaymap.org/

network

• https://wigle.net/map (also a nice network tool)
• https://opencellid.org/

Forum Search Enginer "boardtracker" provides no service

It seems that http://www.boardtracker.com/ should be removed from the list as the site provides no service.

Forums -> Forum Search Engine -> BoardTracker

Add Kirby's FB Google Doc

https://docs.google.com/spreadsheets/d/1y1PeYh8VBHApgaKaO9DwZU3koiKAZeVCApCUJsrd3xI/edit#gid=0

HTTP Server

Why use python as the HTTP server when http-server could be used instead? One less dependency.

Add CloudScraper (T)

{
  "id": "<ID>",
  "name": "CloudScraper (T)",
  "type": "url",
  "url": "https://github.com/jordanpotti/cloudscraper"
},

Username -> Specific Sites

So the sites in the Username -> Specific sites section...what are the thoughts about including my https://github.com/WebBreacher/WhatsMyName project in there? The project has over 160 sites with user name enumeration but it is mostly used in Recon-ng, Spiderfoot and using the script in the project.

Do we put every site where you can do user name enumeration or add those to a list/project and note the project in that section?

usernamecheck.com sppears on the Google (un)safe browsing database

usernamecheck[.]com sppears on the Google (un)safe browsing database and if you power through and ignore the warning the site is the Apache "it works" page.

I suggest removing their entry.

Add LeakedSource for Breach data

I would have made a PR, but readme.md doesn't explain how "arf.json" is set, so i don't know the latest "id" and how to set the link's category.

URL: https://www.leakedsource.com/
Section: Email Address -> Breach data

NC State employee salary info

http://www.newsobserver.com/news/databases/state-pay/

A bit specific to NC but lots of good info here. You can search by name, etc.

Add scanless (T)

scanless - online port scan scraper

IP Address -> Host / Port Discovery:

            {
              "id": "<ID>",
              "name": "scanless (T)",
              "type": "url",
              "url": "https://github.com/vesche/scanless"
},

Use of python is not obvious, particularly python version

The README.md only casually mentions the use of python. The very brief statement actually assumes the use of python 2, because the used SimpleHTTPServer module was merged to the python core with python 3 and is now called http.server.

Apart from this documentation need, the dependency on python is probably superfluous, because a webserver could also be hosted with node itself (but that's for another issue).

some rf stuff

https://github.com/eldraco/Salamandra
https://github.com/xmikos/qspectrumanalyzer
https://github.com/ghostop14/sparrow-wifi
https://github.com/Oros42/IMSI-catcher

more https://github.com/cn0xroot/RFSec-ToolKit

Spokeo issue

I tried to use spokeo for people search but found no possibility to see any result for free. There are only paid plans.

Request to add DeHashed.com

Can conduct “password” searches.
(Email Address) > (Breach Data) > (DeHashed)

{
  "id": "<ID>",
  "name": "DeHashed",
  "type": "url",
  "url": "https://dehashed.com/"
},

Twitter tools unavailable

These tools are no longer available:

Social Networks -> Twitter ->

http://nearbytweets.com/
https://www.allmytweets.net/connect.php

haveibeenpwned.com

Would you consider adding https://haveibeenpwned.com to the Adv Recon Framework -> Email Address -> Breach Data -> haveibeenpwned.com?

lower level requirements missing

I'm installing this in a Lubuntu VM I use for forensics. I am not a regular user of node/npm/bower.

First, it wasn't clear that node needed to be installed, maybe say more about prerequisites for someone starting from scratch?

Then this happened:

./node_modules/bower/bin/bower install
bower invalid-meta for:/home/hunch/OSINT-Framework/bower.json
bower invalid-meta The "name" is recommended to be lowercase, can contain digits, dots, dashes

https://github.com/lockfale/ARF redirects to https://github.com/lockfale/OSINT-Framework, not sure if that's a problem, I manually fixed it in bower.json. After this I read bower help, check, and this is installed.

./node_modules/bower/bin/bower list
bower check-new Checking for new versions of the project dependencies...
arf /home/hunch/OSINT-Framework
└── d3#3.5.16 (latest is 4.7.1)

Executive Summary - this app could stand a careful documenting from start to finish on a fresh Ubuntu system not used by a knowledgeable JavaScript person.

Add certificate searching

Check out https://crt.sh/?cn=%25.nsa.gov!! Search for other domains too! apple.com?

Add Public buckets by grayhatwarfare

{
  "id": "<ID>",
  "name": "Public buckets",
  "type": "url",
  "url": "https://buckets.grayhatwarfare.com/"
},

Images -> Instagram services needs update

http://collec.to/
Instagram from 2th June 2016 has blocked the application that allows us to give you the best experience. We apologize for the inconvenience. We ask you to leave your email to receive news about the projec

http://www.picturegr.am/
There is no app configured at that hostname.
Perhaps the app owner has renamed it, or you mistyped the URL.

https://instagr.in/explore/
Not reachable

MrLooquer. IPv6 Scan

As Shodan, MrLooquer discover new devices on Internet. The difference is that MrLooquer is focus on IPv6 protocol and as Shodan we are pioneering on IPv6 scanning.

MrLooquer: IPv6 Intelligence.
https://mrlooquer.com

Finding the next unused ID is a PITA

I just created the PR #77 and it was a pain in the ass to find an ID thats not used.
Lets discuss ways to make it easier for people to submit new content as a PR instead of just creating an issue.

Broken link or service is no longer available

Link to a Keyword Analysis under Domain Names -> Analytics leads to a Page not found page.

Watcha (Instant Messaging/Whatsapp) Broken

Hey there,

It seems the link is broken.
Best,

Addition of PassiveTotal to the list

I am one of the co-founders of PassiveTotal, now owned by RiskIQ. Our platform is a great resource for passive DNS, WHOIS, SSL certificates, web crawling and more. RiskIQ focuses on surfacing digital risk by web crawlers and Internet scanning. The PassiveTotal platform is a great way to pivot through the data sets to identify reputation, context and correlate OSINT. Registration is free and all accounts have enough queries to be useful on a daily basis.

Several Issues

1. Search Engines -> Code Search -> Meanpath
   meanpath.com’s server DNS address could not be found
   Not sure whether it's my DNS problem or a global one

2. Search Engines -> Search Tools
   Google Alerts are listed twice

3. Email Address -> Email Search -> Find Any Email Checker
   default backend - 404 error

Add WhatWeb (T)

WhatWeb - Web application and web server fingerprinting tool.

Domain Name -> Analytics:

            {
              "id": "<ID>",
              "name": "WhatWeb (T)",
              "type": "url",
              "url": "https://github.com/urbanadventurer/WhatWeb"
},

Request to add tool: IntRec-Pack.

Hello,

Recently I released a utility written in Bash designed to download and install a number of high quality OSINT, Threat Intel and Recon tools. Not only does it perform an installation of these tools it also manages the installation of any dependencies required by them . Most of the tools available in my implementation are already featured in OSINT-Framework. The aim of my shell script however, is to automate the deployment of the tools it features, and to make setting up an intelligence gathering environment quick and easy.

Here is a quick summary of the programs available with IntRec-Pack

+-----------------------+-------------------------------------------+
| Tool                  | Utility type and feature summary          |
+-----------------------+-------------------------------------------+
|1. QuickScan	        | Port Scanner/WHOIS/Domain Resolver        | 
|2. DNSRecon	        | Advanced DNS Enumeration & Domain Utility |
|3. Sublist3r           | OSINT Based Subdomain Enumeration         |
|4. TekDefense-Automator| OSINT Based IP, URL and Hash Analyzer     |
|5. TheHarvester        | eMail, vHost, Domain and PII Enumeration  |
|6. IOC-Parser          | Threat Intel, parses IOC data from reports|
|7. PyParser-CVE        | Multi Source Exploit Parser/CVE Lookup    |
|8. Mimir               | HoneyDB CLI/Threat Intelligence Utility   |
|9. Harbinger           | Cymon.io, Virus Total, Threat Feed Parser |
|10.Spiderfoot          | Advanced OSINT/Reconnaissance Framework   |
+-----------------------+-------------------------------------------+

Furthermore I have added a feature that will allow the user to visit http://osintframework.com by opening it programatically from within the Bash script. The reason for which being that if the user should find they are in need of an OSINT tool that is not covered by IntRec-Pack they can easily browse the selection of resources that you are providing in order to complete their own arsenal.

Should you like to review the source code and/or details of the project please visit it's official repo by clicking here.

With kind regards,
NullArray

Add Infoga

Wish I had time to do a PR but I'll post this pretty good tool here for later inclusion in the framework: https://github.com/m4ll0k/infoga

Add MageScan

MageScan - Magento security scanner.

Domain Name -> Vulnerabilities -> Scanners:

            {
              "id": "<ID>",
              "name": "MageScan",
              "type": "url",
              "url": "https://magescan.com/"
            },

Another site to grab resources from

https://github.com/jivoi/awesome-osint

Hosting of your framework

I don't know if you knew about this or not, but since your framework is just static files (no php or anything like that) you can actually use github to host it. It is called github pages that is just letting you know a bit about it. That way you don't have to pay for hosting fee's, if you are, and you can still keep the same domain as well. Just a little suggestion, because that is what we are doing with one of our (repos)[https://github.com/49thSecurityDivision/slideshows). Let me know if you have any other questions! I meant to tell you during the OSINT class, but forgot and just remembered, because of messing around with one of our repos.

Addition Belati to the list

Hi, I'm the author of Belati( https://github.com/aancw/Belati )

Belati - The Traditional Swiss Army Knife For OSINT

Belati is tool for Collecting Public Data & Public Document from Website and other service for OSINT purpose. This tools is inspired by Foca and Datasploit for OSINT :)

I want to make pull request, but i don't know what number should i use for id. 195 is already used by folder:

{
            "id": "195",
            "name": "Belati (T)",
            "type": "url",
            "url": "https://github.com/aancw/Belati"
         }],

Thanks!

More resources to add!

Just saw that the http://www.i-intelligence.eu/wp-content/uploads/2016/11/2016_November_Open-Source-Intelligence-Tools-and-Resources-Handbook.pdf document was released with a HUGE number of tools and websites in it.

Just putting this here for someone to go through and scrape content from it.

Conversion script

JN,
Just curious what you are generating your arf.json from. are you converting from .mm or some other standard? interested in sharing the script, if so?

doesn't hurt to ask I hope :)

Thanks,

"That One Privacy Guy's VPN Comparison Chart" moved to new location

OpSec -> Anonymous Browsing -> Anonymous VPNs -> That One Privacy Guy's VPN Comparison Chart URL should be changed to https://thatoneprivacysite.net/

please provide d3 files

When I use the d3 files from the d3js.org website it does not work with your files.

How do we make your files work with d3?

Thanks

Special marking for dorks

Hi, thanks for a great framework.

I noticed, that in the list there are many dorks (google queries) and from the first view they are undistinguishable from all the other links. Just as enhancement I can propose marking the dorks with (d) as you currently do with tools (t)

Document ID assignment

For contributors, it is not obvious how to assign an ID to a folder or url node (or any other, if there are any, for that matter).

Also, it is perhaps a good idea to use GUIDs for the ID fields to have non-conflicting UIDs when creating pull requests.

Adding a tip for the link

First off..awesome work doing this. One of the simplest ways I've come across to organise a mass amount of resources into logical groupings. So I'm a noob to JSON and SVG, but having a look at the code in arf.js and chatting with mr google:

nodeEnter.append("svg:title")
.text(function(d) {
return d.tip;
});

Then add a "tip" field to the JSON child item (in fact, adding "tip" to any item adds a mouseover tip) :
"id": "502",
"name": "Namechk",
"type": "url",
"url": "https://namechk.com/",
"tip": "This is the tool tip for Namechk\nThis is line 2 of my tip"
On mouseover, your tooltip pops up, giving additional info as written in the JSON source.

Just a thought?
Cheers.

Additional tool

Malicious File Analysis/Hosted Automated Analysis.

https://hybrid-analysis.com

Been my goto for a while now. A bit slow lately, but super in depth.

Request to add Numberway

Please add to a new International folder.
(Telephone Numbers) > (International) > (Numberway)

{
   "id": "<ID>",
   "name": "Numberway",
   "type": "url",
   "url": "https://www.numberway.com/"
},

Description of resource

Wondering what your thoughts on adding a brief description of the resource so people know what that link will give them.

Other things to possibly add to the JSON might be:

1. Description (described above)
2. Cost or free ($ for paid, 0 for free)
3. Requires registration (pick an icon/symbol for this) yes or no
4. Reliability Ranking - Is the resource solid or false positive-prone?
5. Usefulness Ranking - Of all the resources in the category, which are the ones that someone should visit first to get the most bang for their time?

Just some ideas on extending your useful project.

Add HoneyDB + CLI.

Hello, i would like to make a suggestion with regards to a resource and a tool to be added to OSINT-Framework. The resource i am referring to is HoneyDB. Which in short is an aggregative threat intel pool. HoneyDB collects and organizes data on malicious hosts from HoneyPy honeypots. They offer a threat feed and provide data visualization services as well.

Secondly i am the author of a command line interface which makes it easier for security researchers to retrieve and process the data provided by HoneyDB. The CLI saves logs of the malicious hosts and allows the user to perform an in-script WHOIS lookup and invoke an Nmap scan as well in order to learn more about a particular target if and when desired.

The name of the CLI i wrote is Mimir and you can find it in it's repo on my Github page.

https://github.com/NullArray/Mimir

With kind regards,
NullArray.

Request to add Tinder

Would you consider adding Tinder? (Now has browser based version)
(Dating) > (Tinder)

{
 "id": "<ID>",
 "name": "Tinder (R)",
 "type": "url",
 "url": "https://tinder.com/"
},