hueristiq/xurlfind3r

A command-line interface (CLI) based passive URLs discovery utility. It is designed to efficiently identify known URLs of given domains by tapping into a multitude of curated online passive sources.

xurlfind3r

xurlfind3r is a command-line interface (CLI) based passive URLs discovery utility. It is designed to efficiently identify known URLs of given domains by tapping into a multitude of curated online passive sources.

Resource

• Features
• Installation
  • Install release binaries (Without Go Installed)
  • Install source (With Go Installed)
    • go install ...
    • go build ... the development Version
• Post Installation
• Usage
  • Basic
  • Filter Regex
  • Match Regex
• Contributing
• Licensing
• Credits
  • Contributors
  • Similar Projects

Features

• Fetches URLs from curated passive sources to maximize results.
• Parses URLs from wayback webpages and robots.txt snapshots.
• Filters out duplicate URLs.
• Supports URLs matching and filtering.
• Supports stdin and stdout for easy integration into workflows.
• Cross-Platform (Windows, Linux & macOS).

Installation

Install release binaries (Without Go Installed)

Visit the releases page and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with wget or curl:

• ...with wget:

   wget https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

• ...or, with curl:

   curl -OL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz

...then, extract the binary:

tar xf xurlfind3r-<version>-linux-amd64.tar.gz

TIP: The above steps, download and extract, can be combined into a single step with this onliner

curl -sL https://github.com/hueristiq/xurlfind3r/releases/download/v<version>/xurlfind3r-<version>-linux-amd64.tar.gz | tar -xzv

NOTE: On Windows systems, you should be able to double-click the zip archive to extract the xurlfind3r executable.

...move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

go install ...

go install -v github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest

go build ... the development Version

• Clone the repository

   git clone https://github.com/hueristiq/xurlfind3r.git 

• Build the utility

   cd xurlfind3r/cmd/xurlfind3r && \
   go build .

• Move the xurlfind3r binary to somewhere in your PATH. For example, on GNU/Linux and OS X systems:

   sudo mv xurlfind3r /usr/local/bin/

  NOTE: Windows users can follow How to: Add Tool Locations to the PATH Environment Variable in order to add xurlfind3r to their PATH.

NOTE: While the development version is a good way to take a peek at xurlfind3r's latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

Post Installation

xurlfind3r will work right after installation. However, BeVigil, Github and Intelligence X require API keys to work, URLScan supports API key but not required. The API keys are stored in the $HOME/.hueristiq/xurlfind3r/config.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example config.yaml:

NOTE: The keys/tokens below are invalid, use your own keys/tokens!

version: 0.4.0
sources:
    - bevigil
    - commoncrawl
    - github
    - intelx
    - otx
    - urlscan
    - wayback
keys:
    bevigil:
        - awA5nvpKU3N8ygkZ
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39
    intelx:
        - 2.intelx.io:00000000-0000-0000-0000-000000000000
    urlscan:
        - d4c85d34-e425-446e-d4ab-f5a3412acbe8

Usage

To display help message for xurlfind3r use the -h flag:

xurlfind3r -h

help message:

                 _  __ _           _ _____
__  ___   _ _ __| |/ _(_)_ __   __| |___ / _ __
\ \/ / | | | '__| | |_| | '_ \ / _` | |_ \| '__|
 >  <| |_| | |  | |  _| | | | | (_| |___) | |
/_/\_\\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_|
                                          v0.4.0

                with <3 by Hueristiq Open Source

USAGE:
  xurlfind3r [OPTIONS]

CONFIGURATION:
 -c, --configuration string          configuration file path (default: $HOME/.config/xurlfind3r/config.yaml)

INPUT:
 -d, --domain string[]               target domain
 -l, --list string                   target domains' list file path

   TIP: For multiple input domains use comma(,) separated value with `-d`,
        specify multiple `-d`, load from file with `-l` or load from stdin.

SCOPE:
     --include-subdomains bool       match subdomain's URLs

SOURCES:
     --sources bool                  list supported sources
 -u, --use-sources string[]          comma(,) separated sources to use
 -e, --exclude-sources string[]      comma(,) separated sources to exclude
     --parse-wayback-robots bool     with wayback, parse robots.txt snapshots
     --parse-wayback-source bool     with wayback, parse source code snapshots

FILTER & MATCH:
 -f, --filter string                 regex to filter URLs
 -m, --match string                  regex to match URLs

OUTPUT:
     --no-color bool                 disable colored output
 -o, --output string                 output URLs file path
 -O, --output-directory string       output URLs directory path
 -s, --silent bool                   display output subdomains only
 -v, --verbose bool                  display verbose output

Examples

Basic

xurlfind3r -d hackerone.com --include-subdomains

Filter Regex

# filter images
xurlfind3r -d hackerone.com --include-subdomains -f '`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`'

Match Regex

# match js URLs
xurlfind3r -d hackerone.com --include-subdomains -m '^https?://[^/]*?/.*\.js(\?[^\s]*)?$'

Contributing

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

Credits

Contributors

Thanks to the amazing contributors for keeping this project alive.

Similar Projects

Thanks to similar open source projects - check them out, may fit in your workflow.

gau ◇ waybackurls ◇ waymore

xurlfind3r's Issues

[REQUEST] Add option to parse only robots.txt files

Really quick tool, it would be great if it parses only robots.txt file

Any ideas?

config.yaml new loaction

Hey there,

config.yaml file is no longer under $HOME/.hueristiq/xurlfind3r/config.yaml , it's currently under /$HOME/.config/xurlfind3r/config.yaml , please update it in the readme file

Add option to exclude sources

[BUG]

Description
Hi, thanks for the amazing tool. When I playing around using 'commoncrawl' as the source I found it sometimes can print out urls but sometimes can't. So I modified the code to print out the errors alongside then I found commoncrawl returned tons of 503 and timeout

Steps To Reproduce
./sigurlfind3r -d tesla.com --include-subs -uS commoncrawl

Screenshots

Additional context
I tested over my home network as well as a VPS located on Los Angeles.
I changed a bit of the code to avoid parallel processing and increased the timeout to 60s from 10s. Then it worked as expected.
I haven't found any official documented rate limit of commoncrawl though, but reliability is the top concern in my opinion, especially in terms of automation. Just for your reference and thanks again for your work :)

Consider changing wayback robots and pages parsing and opt in feature

Question

I see this alot but figured i would ask you, why would you use import "github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/session"

when you can just

import crawler_utils "path_to_File_name_of_package"

URLscan check is limited

Hi,
As it stands, the URLScan check seems to me to be limited for two reasons :

• It's not possible to specify an API key, so you can quickly get stuck if you scan several URLs.

  • See https://urlscan.io/docs/api/ > "Quotas & Rate Limiting"

• Results extraction seems to be limited to 100 (i.e. the first page). Is necessary to check if the response contains more than 100 results and, if so, iterate to retrieve the other results.

  • See https://urlscan.io/docs/api/ > "Search API"
  • An example : https://github.com/lc/gau/blob/master/pkg/providers/urlscan/urlscan.go

  Regards

[BUG]

Describe the bug
panic: runtime error: index out of range [0] with length 0

Steps To Reproduce
./sigurlfind3r -d example.com -iS
Output
[ INF ] fetching urls for

panic: runtime error: index out of range [0] with length 0

goroutine 18 [running]:
github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run.func1()
/Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:82 +0x7bc
created by github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run
/Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:63 +0xf2

Additional context
MacOS 12.1

[BUG]

.

Filter out emails

help

how use cat sub.txt | sigurlfind3r

Option to filter extensions

Hi,
Thanks for this tool !

I think it would be interesting to add a flag to filter out certain extensions, something like :

-f     --filter                            comma(,) separated extensions to excludes (ex: jpg,png,svg)

Regards

Option to specify config file

Hi,
I think it would be great to add a flag to specify the config file location, something like :

-c     --config                            config file location

Regards

[BUG] OTS not working

./hqurlfind3r --use-sources otx -d google.com

returns nothing for otx

intelx source 401

No results from intelx

Hey there, I hope all is good.

While we already discussed this, I will create an issue for it to keep things on track.

The tool does indeed issue a request to intelx /phonebook/search path, however, it doesn't return any results in return.

No results from intelx

Output save in File

Hello , Really great tool but I noticed that there is no option for saving output in a file that causes a lot of problem for me. It would be really really best if you kindly add this feature to this tool for saving output in a file.