tomsteele/blacksheepwall

blacksheepwall is a hostname reconnaissance tool

blacksheepwall

Archive Notice

I am no longer maintaing this tool. BSW has served me well, and was written in Go in 2013. I would suggest using amass as it is actively supported.

blacksheepwall is a hostname reconnaissance tool written in Go. It can also be used as a stand-alone package in your tools.

Download

Binary packages for every supported operating system are available here.

Install

You can download a compiled binary and just run it. Alternatively, if you have Go installed and configured with a workspace, you can run:

$ go get github.com/tomsteele/blacksheepwall

Usage

 Usage: blacksheepwall [options] <ip address or CIDR>

 Options:
  -h, --help            Show Usage and exit.

  -version              Show version and exit.

  -debug                Enable debugging and show errors returned from tasks.

  -config               Location of a YAML file containing any of the options below.
                        Hypens should be replaced with underscores (e.g. bing-html, bing_html).
                        Options that do not take an argument are booleans and should be represented
                        using true/false (e.g. bing_html: true).

  -timeout              Maximum timeout in seconds for SOCKET connections.  [default .5 seconds]

  -concurrency <int>    Max amount of concurrent tasks.  [default: 100]

  -server <string>      DNS server address.  [default: "8.8.8.8"]

  -input <string>       Line separated file of networks (CIDR) or IP Addresses.

  -ipv6                 Look for additional AAAA records where applicable.

  -domain <string>      Target domain to use for certain tasks, can be a single
                        domain or a file of line separated domains.

  -fcrdns               Verify results by attempting to retrieve the A or AAAA record for
                        each result previously identified hostname.

  -parse <string>       Generate output by parsing JSON from a file from a previous scan.

  -validate             Validate hostnames using a RFC compliant regex.

 Passive:
  -dictionary <string>  Attempt to retrieve the CNAME and A record for
                        each subdomain in the line separated file.

  -ns                   Lookup the ip and hostname of any nameservers for the domain.

  -mx                   Lookup the ip and hostmame of any mx records for the domain.

  -yandex <string>      Provided a Yandex search XML API url. Use the Yandex
                        search 'rhost:' operator to find subdomains of a
                        provided domain.

  -bing <string>        Provided a base64 encoded API key. Use the Bing search
                        API's 'ip:' operator to lookup hostnames for each ip, and the
                        'domain:' operator to find ips/hostnames for a domain.

  -bing-html            Use Bing search 'ip:' operator to lookup hostname for each ip, and the
                        'domain:' operator to find ips/hostnames for a domain. Only
                        the first page is scraped. This does not use the API.

  -shodan <string>      Provided a Shodan API key. Use Shodan's API '/dns/reverse' to lookup hostnames for
                        each ip, and '/shodan/host/search' to lookup ips/hostnames for a domain.
                        A single call is made for all ips.

  -reverse              Retrieve the PTR for each host.

  -viewdns-html         Lookup each host using viewdns.info's Reverse IP
                        Lookup function. Use sparingly as they will block you.

  -viewdns <string>     Lookup each host using viewdns.info's API and Reverse IP Lookup function.

  -logontube            Lookup each host and/or domain using logontube.com's API. As of this release
                        the site is down.

  -exfiltrated          Lookup hostnames returned from exfiltrated.com's hostname search.

  -censys <string>      Searches censys.io for a domain. Names are gathered from TLS certificates for each host
                        returned from this search. The provided string should be your API ID and Secret separated
						by a colon.

  -crtsh                Searches crt.sh for certificates related to the provided domain.
  
  -vt                   Searches VirusTotal for subdomains for the provided domain.

  -srv                  Find DNS SRV record and retrieve associated hostname/IP info.

  -cmn-crawl <string>   Search commoncrawl.org for subdomains of a domain. The provided argument should be the index
                        to be used. For example: "CC-MAIN-2017-04-index"

 Active:
  -axfr                 Attempt a zone transfer on the domain.

  -headers              Perform HTTP(s) requests to each host and look for
                        hostnames in a possible Location header.

  -tls                  Attempt to retrieve names from TLS certificates
                        (CommonName and Subject Alternative Name).

 Output Options:
  -clean                Print results as unique hostnames for each host.
  -csv                  Print results in csv format.
  -json                 Print results as JSON.

blacksheepwall's Issues

-debug with -srv causes a panic

$ ./blacksheepwall -debug -domain github.com -srv
2015/01/18 05:21:55 Spreading tasks across 100 goroutines
panic: runtime error: index out of range

goroutine 5 [running]:
main.func·002()
/Users/tsteele/code/go/src/github.com/tomsteele/blacksheepwall/blacksheepwall.go:240 +0x548
created by main.main
/Users/tsteele/code/go/src/github.com/tomsteele/blacksheepwall/blacksheepwall.go:246 +0x1c18

goroutine 1 [chan receive]:
main.main()
/Users/tsteele/code/go/src/github.com/tomsteele/blacksheepwall/blacksheepwall.go:351 +0x2e50

goroutine 105 [chan receive]:
main.func·003()
/Users/tsteele/code/go/src/github.com/tomsteele/blacksheepwall/blacksheepwall.go:251 +0xa3
created by main.main
/Users/tsteele/code/go/src/github.com/tomsteele/blacksheepwall/blacksheepwall.go:281 +0x1d7b

Use new azure search API

This doens't look right

blacksheepwall/blacksheepwall.go

Lines 256 to 257 in 48e0eff

	ip, err := bsw.LookupName(r.Hostname, *flServerAddr)
	if err == nil && len(ip) > 0 {

Domain searches that use LookupCname need to call LookupName aftrer

crtsh does not search for subdomains

I may be wrong about the intent (domain enum vs subdomain enum), but I'm pretty sure the crt.sh search should include a wildcard so that subdomains are matched. So instead of example.com, it should search for example.com and %.example.com. It'd also be easier to parse crt.sh's JSON (&output=json), but this works.

Current Search:
https://crt.sh/?q=soundcloud.com

Current Example:
blacksheepwall -crtsh -domain soundcloud.com
2018/03/12 21:24:29 Spreading tasks across 100 goroutines
2018/03/12 21:24:32 All tasks completed
IP Hostname Source
52.84.69.50 soundcloud.com CRT.SH CT
52.84.69.90 soundcloud.com CRT.SH CT
52.84.69.150 soundcloud.com CRT.SH CT
52.84.69.156 soundcloud.com CRT.SH CT
52.84.69.172 soundcloud.com CRT.SH CT
52.84.69.230 soundcloud.com CRT.SH CT
52.84.69.234 soundcloud.com CRT.SH CT
52.84.69.253 soundcloud.com CRT.SH CT

Recommended:
https://crt.sh/?q=soundcloud.com
https://crt.sh/?q=%.soundcloud.com

Changes would go here:
https://github.com/tomsteele/blacksheepwall/blob/master/bsw/ct.go#L29
https://github.com/tomsteele/blacksheepwall/blob/master/bsw/ct.go#L41 (not sure on the logic and if it is needed here)

Add MX record

Add Bing domain: search

Common Crawl is running without being called.

Between commits 6ded4f6 and 6095a91
./blacksheepwall -server 192.168.1.1 -reverse 192.168.1.0/24
returns
2017/04/12 19:10:22 Common Crawl requires domain set with -domain
It works as expected with builds from before 96b9402.

Error in third party repo

Looks like the dependency miekg/dns has changed and causes this build to fail.

$ go get github.com/tomsteele/blacksheepwall

github.com/miekg/dns

../../../Tools/Programming/go/src/github.com/miekg/dns/client.go:489: undefined: time.Until
../../../Tools/Programming/go/src/github.com/miekg/dns/dnssec_keyscan.go:295: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/msg_helpers.go:144: base32.HexEncoding.WithPadding undefined (type *base32.Encoding has no field or method WithPadding)
../../../Tools/Programming/go/src/github.com/miekg/dns/msg_helpers.go:144: undefined: base32.NoPadding
../../../Tools/Programming/go/src/github.com/miekg/dns/msg_helpers.go:270: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/serve_mux.go:43: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/types.go:422: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/types.go:495: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/types.go:505: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/types.go:517: undefined: strings.Builder
../../../Tools/Programming/go/src/github.com/miekg/dns/types.go:422: too many errors

Add axfr zone transfer

Add Shodan API

need $

running blacksheepwall

hello.

This is not really an issue,but I'm not sure where else to put it.

I understand little to nothing about node.js. However, it seems like it can be run like any other interpreted language? (ie. #!/usr/bin/env node)

I'm trying to get node configured so I can execute blacksheepwall.

• I am running
  ---> Linux ********** 3.7-trunk-686-pae #1 SMP Debian 3.7.2-0+kali8 i686 GNU/Linux
• Pulled down the latest Linux binaries from nodejs.org.
  ---> http://nodejs.org/dist/v0.10.18/node-v0.10.18-linux-x86.tar.gz
• Copied the file to /usr/local/share/ and un-tarred it
• Added to $PATH
  ---> echo $PATH
  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/share/node-v0.10.18-linux-x86/bin/
• Attempted to run blacksheep wall.
  ------> ./cmd.js
  module.js:340
  throw err;
  ^
  Error: Cannot find module 'commander'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object. (/root/utilities/blacksheepwall/bin/cmd.js:7:15)
  at Module._compile (module.js:456:26)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)

I'm sure i'm doing something dumbass.

I can't find much online or in the README. Can you assist?

thanks,

-pjh

Program crash on Windows 10

Unless I'm mistaken, these compiled binaries should work without Go installed on the system?

If so, the latest version and the next oldest, at least, do not. They crash with the following with no arguments, or just an IP address argument.

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x30 pc=0x402aba]

goroutine 1 [running]:
panic(0x7214c0, 0xc04200a090)
        /usr/local/opt/go/libexec/src/runtime/panic.go:500 +0x1af
main.main()
        /Users/t/Code/go/src/github.com/tomsteele/blacksheepwall/main.go:300 +0xc7a

I tried running in an admin shell, just in case, but that didn't make any difference.
OS is Windows 10 1607 14393.576 64 bit

Add Google site: search

Source Populates Incorrectly

All results show a source of FCRDNS regardless of its actual source if running FCRDNS.

Ex:
./blacksheepwall_linux_amd64 -domain -ns
2017/09/07 10:20:02 Spreading tasks across 100 goroutines
2017/09/07 10:20:02 All tasks completed
IP Hostname Source
REMOVED IP REMOVED HOSTNAME ns
REMOVED IP REMOVED HOSTNAME ns
REMOVED IP REMOVED HOSTNAME ns

./blacksheepwall_linux_amd64 -domain -ns -fcrdns
2017/09/07 10:20:02 Spreading tasks across 100 goroutines
2017/09/07 10:20:02 All tasks completed
IP Hostname Source
REMOVED IP REMOVED HOSTNAME fcrdns
REMOVED IP REMOVED HOSTNAME fcrdns
REMOVED IP REMOVED HOSTNAME fcrdns

Look into adding Baidu site search

Multiple domain search methods do not call LookupCname if LookupName fails

Implement iterative domain searching

censys API rate limit?

When doing a censys scan against a large file of domains, the first few succeed and then I see a long list of "Request returned non 200 status code. (with debugging enabled). I'm wondering if the default settings are overloading the rate limit the default free account provides?