Giter Site home page Giter Site logo

hamburglar's Introduction

The Hamburglar

Setup

There are 2 versions of hamburglar, full and lite. The main branch is the full version, and hamburglar lite is on a separate branch.

Hamburglar

Full fledged scraping tool for artifact retrieval from multiple sources. There are some dependencies, so install them first:

pip3 install -r requirements.txt

Hamburglar Lite

Multithreaded and recursive directory scraping script. Stores useful information with the filepath and finding. Hamburglar lite will never require external packages, and will always remain as a single script. Setup is as simple as requesting the file and using it:

wget https://raw.githubusercontent.com/needmorecowbell/Hamburglar/hamburglar-lite/hamburglar-lite.py

This is designed to be quickly downloaded and executed on a machine.

Operation

usage: hamburglar.py [-h] [-g] [-x] [-v] [-w] [-o FILE] [-y YARA] path

positional arguments:
  path                  path to directory, url, or file, depending on flag
                        used

optional arguments:
  -h, --help            show this help message and exit
  -g, --git             sets hamburglar into git mode
  -x, --hexdump         give hexdump of file
  -v, --verbose         increase output verbosity
  -w, --web             sets Hamburgler to web request mode, enter url as path
  -o FILE, --out FILE   write results to FILE
  -y YARA, --yara YARA  use yara ruleset for checking

Directory Traversal

  • python3 hamburglar.py ~/Directory/
    • This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters

Single File Analysis

  • python3 hamburglar.py ~/Directory/file.txt
    • This will recursively scan for files in the given directory, then analyzes each file for a variety of findings using regex filters

YARA Rule Based Analysis

  • python3 hamburglar.py -y rules/ ~/Directory
    • This will compile the yara rule files in the rules directory and then check them against every item in Directory.

Git Scraping Mode

  • python3 hamburglar.py -g https://www.github.com/needmorecowbell/Hamburglar
    • Adding -y <rulepath> will allow the repo to be scraped using yara rules

Web Request Mode

  • python3 hamburglar.py -w https://google.com
    • Adding a -w to hamburgler.py tells the script to handle the path as a url.
    • Currently this does not spider the page, it just analyzes the requested html content

Hex Dump Mode

  • python3 hamburglar.py -x ~/file-to-dump
    • This just does a hex dump and nothing more right now -- could be piped into a file
    • This will eventually be used for binary analysis

Tips

  • Adding -v will set the script into verbose mode, and -h will show details of available arguments
  • Adding -o FILENAME will set the results filename, this is especially useful in scripting situations where you might want multiple results tables (ie github repo spidering)

Settings

  • whitelistOn: turns on or off whitelist checking
  • maxWorkers: number of worker threads to run concurrently when reading file stack
  • whitelist: list of files or directories to exclusively scan for (if whitelistOn=True)
  • blacklist: list of files, extensions, or directories to block in scan
  • regexList: dictionary of regex filters with filter type as the key

The Hamburglar can find

  • ipv4 addresses (public and local)
  • emails
  • private keys
  • urls
  • cryptocurrency addresses
  • anything you can imagine using regex filters and yara rules

Example output:

{
    "/home/adam/Dev/test/email.txt": {
        "emails": "{'[email protected]'}"
    },
    "/home/adam/Dev/test/email2.txt": {
        "emails": "{'[email protected]'}"
    },
    "/home/adam/Dev/test/ips.txt": {
        "ipv4": "{'10.0.11.2', '192.168.1.1'}"
    },
    "/home/adam/Dev/test/test2/email.txt": {
        "emails": "{'[email protected]', '[email protected]'}"
    },
    "/home/adam/Dev/test/test2/ips.txt": {
        "ipv4": "{'10.0.11.2', '192.168.1.1'}"
    },
    "/home/adam/Dev/test/test2/links.txt": {
        "site": "{'http://login.web.com'}"
    }
}

Contributions

  • Please contribute! If there's an error let me know -- even better if you can fix it :)

  • A big thank you to anyone who has helped:

    - [adi928](https://github.com/adi928)
    - [jaeger-2601](https://github.com/jaeger-2601)
    - [tijko](https://github.com/tijko)
    

hamburglar's People

Contributors

needmorecowbell avatar adi928 avatar tijko avatar jaeger-2601 avatar pingywon avatar

Watchers

Orange avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.