matir/webborer

WebBorer is a directory-enumeration tool written in Go.

WebBorer

WebBorer is a directory-enumeration tool written in Go and targeting CLI usage.

This project was formerly named 'GoBuster', but that had a namespace collision with OJ Reeves' excellent tool (which was released about the same time as I was preparing this for release).

Features

• Highly portable -- requires no runtime once compiled.
• No GUI required.
• Supports Socks 4, 4a, and 5 proxies.
• Supports excluding entire subpaths.
• Capable of parsing returned HTML for additional directories to parse.
• Highly scalable -- Go's parallel model allows for many workers at once.

Contributing

Please see the CONTRIBUTING file in this directory.

Copyright

Copyright 2015-2017 Google Inc.

WebBorer is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

Contact

For questions about WebBorer, contact David Tomaschik <[email protected]>

webborer's Issues

Detect duplicate results

Map of result hashes perhaps?

Strip fragment from URLs

Requesting the fragment makes no sense, we should strip it before requesting...

Support custom HTML templates

Let the HTML output use custom templates.

Split work across URLs

When processing multiple seed URLs, work should be split across them to reduce hammering on a single box.

Manipulate wordlist by case

convert foo to FOO and Foo at least.

Increase test coverage

Support HTTP Basic Auth

We should support sites with HTTP auth. (At least Basic, maybe Digest as well?)

HTML spidering not working

Not sure why, also no size is reported.

Main is too long

Should clean up/refactor main into more manageable chunks.

Add content-type to results

Detect when a host is down.

No point in continuing to beat a dead horse.

Progress Bar should be on stderr

Right now it's on stdout which messes up piping to a file. Instead, the progress bar should be on stderr.

Support multiple starting URLs

Support a method for checking for 404s

• Disable wordlist expansion
• Enable HTML parsing
• Check each link
• Build report of 404s (source page, link destination)
  • Needs new result type
• Allow checking off-site resources by one degree

Setup Travis CI Tests

Support randomizing the queue

To distribute load across many hosts, it would be good to randomize the queue.

--include-redirects broken?

--include-redirects seems broken.

Better document parsing

Consider parent directories of resources, include all resource types, etc.

Support other HTTP methods/verbs

Currently, only GET is supported.

Refactor main to be more testable

Handle CTRL+C gracefully.

Make sure we write a complete output.

Load config from file

Support arbitrary cookies

Should allow loading cookies from a netscape-formatted cookie jar to add to requests. This will enable support for logins, etc.

Allow URLs to be input as a file containing a list.

Check robots.txt in obey mode before starting crawl

Support suspend/resume

Save current state to a file, restore from file

Add tiny-wordlist for quick testing

Refactor string slice flag

Instead of expecting a comma separated list, we should support multiple instances of the flag. (Really, we should support both.)

Add ability to limit to particular number of requests.

e.g., stop after 200 requests. (or pages found?)

Support protocol redirection

Optionally, allow for upgrade to HTTPS.

Refactor the worker, it's a mess!

The worker is a mess of spaghetti. Let's clean it up.

0 bytes shown with no Content-Length header

When there is no Content-Length header, 0 bytes is shown for the length.

Refactor settings flag types

Let's not be so cheap on files... move them all out.

Show statistics during operation

Show # of request remaining, completed, etc.

Automatically detect sites that respond with 200 to all URLs.

Some sites respond (incorrectly) to all URLs with a 200 response. We should detect and warn/abort on these. (Perhaps they can still be spidered via some string in the page?)