sayak-brm/osintbuddy

Node graphs, OSINT data mining, and plugins. Connect unstructured and public data for transformative insights

I have no data yet. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

Discord                                                                   osintbuddy.com                                                 osintbuddy-plugins

Introducing OSINTBuddy

Welcome to the OSINTBuddy project where you can connect, combine, and get insights from unstructured and public data as results that can be explored step-by-step. An easy-to-use plugin system allows any Python developer to quickly integrate new data sources.

---

Table of Contents

1. What is OSINTBuddy
2. Getting Started
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. Contact
7. Sponsor OSINTBuddy

What is osintbuddy.com?

Not much, yet.

But here's the concept:

An almost incomprehensible amount of data is created every day. And each year, figures are growing at an ever-increasing rate. These data sources can be divided up into six different categories of information flow:

• Public government data
• Media (newspapers, magazines, radio)
• Internet (blogs, discussion groups, citizen media, etc)
• Professional and academic publications (budgets, hearings, telephone directories, websites, etc)
• Commercial data (commercial imagery, financial assessments, databases, etc)
• Grey literature (technical reports, preprints, patents, business documents, etc)

Some of the main hindrances to practical OSINT is the volume of information it has to deal with (information explosion) and the issue of low quality data. With the majority of this data being unstructured, there's many challenges to analyzing it and producing actionable intelligence since most data analytics databases are designed for structured data.

We've decided to do something about it. The rapid developments in technologies such as AI and big data analytics have opened new horizons for OSINT which weren't previously available. We want our information quick and to the point, that's why we're building an open-source OSINT tool that's free. Free to use, free to modify, free to do with as you wish, and built with plain old web technologies anyone can learn. But this isn't really the project. This is a free new community. A community for OSINT enthusiast around the world and we need your help to design it, to program it, and to build it. We want to hear your suggestions, your ideas, and we're going to build it right in front of your eyes. The notion of a “needle in a haystack” is taken to the extreme on the internet. Let's build a magnet.

Key Features

• Simplified data fetching from multiple sources.
• Visual representation of fetched data for easy understanding.
• A powerful development platform that is open for contributions.
• New plugin-based system for transforming data, check out the osintbuddy package. Documentation is coming soon.

↑Getting Started

To start using OSINTBuddy, follow these simple installation steps:

Installation

1. Open your terminal

   • Note for Windows: I recommend using tabby configured with a git bash profile

   • Note for Windows: For this project to work you need unix line endings (context). Before cloning, run:

     git config --global core.autocrlf false

2. Clone the repo and submodules

   git clone --recurse-submodules https://github.com/jerlendds/osintbuddy.git
   cd osintbuddy

3. Install Docker
   • Learn how to install Docker on Mac
   • Learn how to install Docker on Windows
   • Learn how to install Docker on Linux
4. Start the stack with Docker (if the cp command doesn't work on your OS, copy and rename the .env.example file to .env):

   cp .env.example .env
   docker compose up

   • Note: the stack will take a few minutes to startup while elastic and scylladb configure themselves for JanusGraph. If you try to connect before all the databases are ready you will encounter error notifications.

• URLs
  • Frontend: http://localhost:3000
  • Backend: http://localhost:8000/api/
  • Documentation: http://localhost:8000/docs -- http://localhost:8000/redoc
• Access OSINTBuddy through the URLs provided for frontend, backend, and documentation.

↑Roadmap

See our trello board for a list of our upcoming features.

See the open issues for a list of requested features (and known issues).

↑Progress Notes

• Website node

• Google search node

• Google cache search node

• Google dorks

• DNS node

• URL node

• IP node

• CSE node

• Username node

• Added a plugin system

• Updated plugin system to be easier to work with

• Implemented automatic saving for project graphs

• For this update you'll have to rebuild your docker containers (docker compose build)! After that long wait you'll be rewarded with an updated UI and automatic saving for project graphs alongside an updated and easier to work with plugin system! osintbuddy

↑Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/add-core-plugin)
3. Commit your Changes (git commit -m 'feat: add core plugin to osintbuddy app')
4. Push to the Branch (git push origin feature/add-core-plugin)
5. Open a Pull Request

↑License

Licensed under the GNU Affero General Public License v3.0.

Copyright 2023 jerlendds.GNU AGPL v3.

Note, the OSINTBuddy PyPi package is MIT licensed.

↑Contact

Open an issue if you need to get in touch with me or send an email to [email protected]

You can find my personal blog where I'll be documenting the journey of building OSINTBuddy at studium.dev

• Project Link: https://github.com/jerlendds/osintbuddy
• Plugins Package Link: https://github.com/jerlendds/osintbuddy-plugins
• Patched aiogremlin Link: https://github.com/jerlendds/gremlinpy/

↑Sponsor OSINTBuddy

Help us keep the OSINTBuddy project free and maintained forever. Sponsor the OSINTBuddy project for unique benefits. Learn more on the OSINTBuddy OpenCollective project page