trimstray/htrace.sh

My simple Swiss Army knife for http/https troubleshooting and profiling.

Hi!

I'm a System and Network Administrator. I post my ongoing research at blkcipher.pl.

Some of my blog posts ^[PL]

• Czym jest i jakie znaczenie ma średnie obciążenie systemu?
• TLS: Jaki rozmiar powinien mieć klucz prywatny?
• Co jeśli kill -9 nie działa?
• NGINX: Usprawnienie zamykania procesów roboczych
• Dlaczego pamięć SWAP jest nadal istotna?
• Negocjacja wersji TLS i pakiet szyfrów sygnalizacyjnych

htrace.sh's Issues

Add User Agents list.

Randomize _user_agent variable.

Security scan with Nmap NSE Library.

Security scan with Nmap NSE Library:

local _nmap_nse_scripts="http-auth-finder,\
                         http-chrono,\
                         http-cookie-flags,\
                         http-cors,\
                         http-cross-domain-policy,\
                         http-csrf,\
                         http-dombased-xss,\
                         http-git,\
                         http-grep,\
                         http-internal-ip-disclosure,\
                         http-jsonp-detection,\
                         http-malware-host,\
                         http-methods,\
                         http-passwd,\
                         http-phpself-xss,\
                         http-php-version,\
                         http-robots.txt,\
                         http-sitemap-generator,\
                         http-shellshock,\
                         http-stored-xss,\
                         http-unsafe-output-escaping,\
                         http-useragent-tester,\
                         http-vhosts,\
                         http-xssed,\
                         ssl-enum-ciphers,\
                         whois-ip"

Set HTTP method.

Example: -m|--method <GET|POST>

OpenSSL versions.

• openssl 1.1.0j
• openssl 1.0.2g

Homebrew installation

This looks cool--I saw it mentioned on Hacker News.

I'd like to ask for a Homebrew formula to be created so that I could install this on OS/X with brew install htrace or similar.

If you're not on a Mac or don't want to do this, I'll be happy to take care of it some evening, just let me know!

-- Doug

Multi threaded tasks.

For all jobs (especially for params).

Set cookie or auth header.

A lot of endpoints are behind authentication. The --req-headers flag is feasible, but there should be a shortcut flag just for authentication headers. Maybe --cookie and --auth

Reduce the curl command calls.

fgrep _curl_base * -R                                                                                                                                                                                                                 [10:49:38]
lib/DomainResolve:  _host_ip=$($_curl_base -ks -m "$_timeout" "https://dns.google.com/resolve?name=${_host}&type=A" | \
lib/DomainTrace:  _http_output=$($_curl_base -ks -m "$_timeout" \
lib/DomainTrace:    local _via_ip=$($_curl_base_remote -ks -m "$_timeout" http://whatismyip.akamai.com/)
lib/DomainTrace:        $_curl_base -Iks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -Iks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -ks -m "$_timeout" \
lib/DomainTrace:        $_curl_base -ks -m "$_timeout" \
src/__init__:  local _curl_base=""
src/__init__:  local _curl_base_remote=""
src/__init__:    _curl_base="curl --proxy $proxy_type --request $req_method_type"
src/__init__:    _curl_base_remote="curl --proxy $proxy_type"
src/__init__:    _curl_base="curl --request $req_method_type"
src/__init__:    _curl_base_remote="curl"

Test http/https connection without redirects.

Test http/https connection without redirects, e.g.:

htrace.sh -d https://example.com --without-redirects

A little problem with the output

Hey there,

As part of the output , I'm getting this before the rest of the output

curl: unknown --write-out variable: 'http_version'
curl: unknown --write-out variable: 'scheme'

What does this imply ? Please check out the attached photo

Not working on MacOS/Darwin

$ sudo ./setup.sh install
Password:
readlink: illegal option -- f
usage: readlink [-n] [file ...]
Create symbolic link to /usr/local/bin
ln: /usr/local/bin/htrace.sh: File exists
Create man page to /usr/local/man/man8
$ ln -s ~/src/github.com/trimstray/htrace.sh/bin/htrace.sh /usr/local/bin/
$ htrace.sh -d http://nmap.org -s -h
readlink: illegal option -- f
usage: readlink [-n] [file ...]
/usr/local/bin/htrace.sh: line 55: ./../src/settings: No such file or directory
/usr/local/bin/htrace.sh: line 57: ./../src/helpers: No such file or directory
/usr/local/bin/htrace.sh: line 59: ./../src/__init__: No such file or directory
/usr/local/bin/htrace.sh: line 66: __main__: command not found
/usr/local/bin/htrace.sh: line 68: _exit_: command not found
$ bash --version
GNU bash, version 5.0.0(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Incorrect parse url arguments.

• ?q=foo -> URI: <scheme>://<domain>

Better parsing 'issuer' from certificate.

Better parsing 'issuer' from certificate, eg.:

issuer: DOT Certification Authority, e-mail: [email protected]

dig: connection timed out

Replace connection timed out output error.

Package for distros

It would be good to package this for distros. So I can just do "apt/dnf/… install" and I have it.

Maybe we can use this issue to try to make this happen?

Wiki: updated 'Examples' chapter

Wiki Examples chapter.

• - Check redirects
• - POST method

Remove all url arguments.

Example:

htrace.sh -d https://nmap.org?test-arg -s

    htrace.sh v1.0.7


    URI: https://nmap.org?test-arg

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   0.702265     0.702265        10.217.11.10:42558     35.230.xxx.xxx   45.33.49.119:443     https   1.1   200      
         ssl: on, version(), cipher()
unable to load certificate
140111198290112:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139785700073664:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140045602996416:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139713580265664:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
139865397174464:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140218849345728:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
unable to load certificate
140641044259008:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE
         public-key(), signature()
         date:  / 
         issuer: <empty>
         owner: <empty>
         cn: <empty>
         san: <empty>
         validity: is not valid (hostname mismatch)
         verification:

Reorganization htrace.sh parameters.

• -h|--headers
• -b|--body
• -M|--method
• -H|--header

./setup.sh install - with dependencies.

Add --with-dependencies param to setup.sh.

• curl
• openssl
• dig
• bc
• observatory
• ssllabs-scan
• mixed-content-scan
• nmap

Too old nmap version in Docker image

Hi again, it's your beta tester number one... ;)

Running on standalone machine with nmap 7.7 it works fine, but for Docker image

 Scan domain with Nmap NSE Library (https://nmap.org/book/nse.html)

         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-jsonp-detection' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!
         › Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-09 11:43 UTC
         › NSE: failed to initialize the script engine:
         › /usr/bin/../share/nmap/nse_main.lua:816: 'http-cookie-flags' did not match a category, filename, or directory
         › stack traceback:
         ›      [C]: in function 'error'
         ›      /usr/bin/../share/nmap/nse_main.lua:816: in local 'get_chosen_scripts'
         ›      /usr/bin/../share/nmap/nse_main.lua:1271: in main chunk
         ›      [C]: in ?
         › QUITTING!

possible to include newer nmap version in Docker image? possible solution below to include latest nmap

RUN apt update
RUN apt -y install wget
RUN wget https://nmap.org/dist/nmap-7.70-1.x86_64.rpm
RUN apt -y install alien
RUN alien nmap-7.70-1.x86_64.rpm
RUN dpkg -i nmap_7.70-2_amd64.deb
RUN nmap -version

Can I do traceroute like functionality without admin privileges?

I need to do something like traceroute/tracepath/tracert, but I have no admin privileges and I do not have traceroute/tracepath/tracert installed. All I have is Java, Python, cygwin, and Windows 7, and I can put things in my home directory but I cannot do full installations. Do you have or know of a way to do traceroute/tracepath/tracert given these limitations?

Error Kali Linux

When install in Kali Linux:

sudo ./dependencies.sh Autoinstaller is not available on your system.

Docker image for htrace.sh.

Docker automated build image.

Improve 'issuer' and 'owner' metadata.

• add/update o and cn for issuer
• add/update o and ou for owner

macOS: readlink: illegal option -- f

Both setup.sh and htrace.sh show the error:

readlink: illegal option -- f

The easiest solution is probably to ask macOS users to do brew install coreutils and replace all readlink calls with greadlink if it exists.

Testing SSL protocols and ciphers with testssl.sh

Testing SSL protocols and ciphers with testssl.sh.

PATH env is hardcoded causing issues when external tools are not located in default locations

root@bethebeast:/usr/share/backgrounds# export PATH=/root/yaes:/root/.config/composer/vendor/bin/:/root/infosec/tools:/root/go/bin/:/usr/local/go/bin:$PATH
root@bethebeast:/usr/share/backgrounds# htrace.sh
not found in PATH: ssllabs-scan mixed-content-scan
root@bethebeast:/usr/share/backgrounds# which observatory
/usr/bin/observatory
root@bethebeast:/usr/share/backgrounds# which ssllabs-scan
/root/infosec/tools/ssllabs-scan
root@bethebeast:/usr/share/backgrounds# ln -s /root/infosec/tools/ssllabs-scan /usr/bin/
root@bethebeast:/usr/share/backgrounds# which ssllabs-scan
/root/infosec/tools/ssllabs-scan
root@bethebeast:/usr/share/backgrounds# htrace.sh
not found in PATH: mixed-content-scan

I did not looked into code, but seems like htrace.sh is not looking into $PATH from environment variables?

Generate html output report.

• hah
• ansi2html

Add -H|--header for request header(s).

Add -H|--header for request header(s).

Automatic redirect loop detection.

Prevent too many redirects.

SNI and non-SNI support

Test SNI and non-SNI for TLS.

Empty local port number for local_socket

Only when --hide-src-ip is used:

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.266118     0.266118        xxx.xxx.xxx.xxx        xxx.xxx.xxx.xxx  35.228.233.78:443     US    https   1.1   200

"not found in PATH" error message.

after run :
htrace.sh --domain https://www.test.com
write this error :
not found in PATH: geoiplookup observatory ssllabs-scan mixed-content-scan

Unknown gateway (via) with POST method.

htrace.sh -d https://badssl.com -m POST

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.248689     0.248689        192.168.220.30:41780   unknown          104.154.89.105:443    US    https   1.1   405

Unclear how to create man page

Please delete -- posted on the wrong thing.

GeoIP: set null for internal IPs

Example:

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 ›   1   0.011561     0.011561        172.32.220.172:39858   <int_link>       172.32.220.240:443           https   2.0   502

Set proxy (as a param).

For all traffic, e.g. curl ssllabs, mozilla-observatory.

For nmap:

• https://www.aldeid.com/wiki/Tor/Usage/Nmap-scan-through-tor
• https://www.shellhacks.com/anonymous-port-scanning-nmap-tor-proxychains/

Bypassing Reverse Proxy, Cache and Content Delivery Networks.

Added GeoIP (resolves geographic information about an IP address).

Resolves geographic information about an IP address.

geoip-bin -> geoiplookup

Value for POST method.

./bin/htrace.sh -d https://example.com -M "POST:aaa bbb ccc"

Different results between "docker run" and standalone tool

After building Docker image

Docker version 18.06.0-ce, build 0ffa825 ( same happens on Linux machine though )
Darwin Kernel Version 17.7.0:

docker run --rm -e TERM=screen web-security:htrace --domain https://wp.pl/ --mixed-content

    htrace.sh v1.0.6


    URI: https://wp.pl/

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   1.648620     1.648620        172.17.0.2:33722       213.241.3.97     212.77.98.9:443      https   2.0   301      https://www.wp.pl/
 ›   2   1.875227     0.226607        172.17.0.2:33730       213.241.3.97     212.77.98.9:443      https   2.0   200

    Scan domain for Mixed Content (https://github.com/bramus/mixed-content-scan)

         › Mixed content not found

meanwhile standalone machine

root@bethebeast:/usr/share/backgrounds# htrace.sh --domain https://wp.pl --mixed-content

    htrace.sh v1.0.6


    URI: https://wp.pl

    req  time_total   time_connect    local_socket           via              remote_socket        type    http  code     next_hop
    ---  ----------   ------------    ------------           ---              -------------        ----    ----  ----     --------
 ›   1   2.802012     2.802012        172.16.219.169:42066   213.241.3.97     212.77.98.9:443      https   2.0   301      https://www.wp.pl/
 ›   2   5.139284     2.337272        172.16.219.169:42074   213.241.3.97     212.77.98.9:443      https   2.0   200

    Scan domain for Mixed Content (https://github.com/bramus/mixed-content-scan)

         › [2018-08-09 11:53:45] MCS.ERROR: 00000 - https://www.wp.pl/
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.google.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.google.com
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.facebook.com
         › [2018-08-09 11:53:45] MCS.WARNING: http://sportowefakty.wp.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://wiadomosci.wp.pl
         › [2018-08-09 11:53:45] MCS.WARNING: http://www.money.pl

any idea what may be causing that?

EDIT: on my external VPS I do not have this issue

docker -v
Docker version 18.06.0-ce, build 0ffa825
4.15.0-30-generic #32~16.04.1-Ubuntu SMP

may be smth up with network here...

New external tool: mixed-content-scan

Repo: mixed-content-scan

With new htrace.sh param: --mixed-content.

tput: No value for $TERM and no -T specified in Docker htrace

Hello,

I build a container with the Docker file but when I try to run it
docker run --rm htrace --domain https://domaine.com

I have the error :
tput: No value for $TERM and no -T specified

Where is my mistake ?

Regards

Identify and fingerprint Web Application Firewall (WAF).

The best external tool: wafw00f

Cannot build a Docker image by `Unsupported system version.`

As How to use says, I ran cd htrace.sh/build && docker build --rm -t htrace.sh -f Dockerfile . and I got a failure with Unsupported system version..

It seems the failure happens on the line 96 of Dockerfile, which executes setup.sh

if [[ "$OSTYPE" == "darwin"* ]] ; then
  [ ! -z "$(brew --prefix)" ] && PATH=$(brew --prefix)/opt/coreutils/libexec/gnubin:$PATH
  readonly _dir=$(dirname "$(readlink "$0" || echo "$(echo "$0" | sed -e 's,\\,/,g')")")
elif [[ "$OSTYPE" == "linux-gnu" ]] ; then
  readonly _dir=$(dirname "$(readlink -f "$0" || echo "$(echo "$0" | sed -e 's,\\,/,g')")")
else
  printf "Unsupported system version.\\n"
  exit 1
fi

With the Dockerfile, this shell script is executed in alpine:latest image, and $OSTYPE returns linux-musl.

Added ssl owner 'Organization' and 'OrganizationalUnit'.

• _ssl_domain_subject_o
• _ssl_domain_subject_ou

Show response body.

Example:

htrace.sh -d https://badssl.com --body

Hide source IP address.

Standard output:

htrace.sh -d https://badssl.com                 

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.657482     0.657482        10.245.203.25:52236    211.105.75.25    104.154.89.105:443    US    https   1.1   200

with --hide-src-ip param:

htrace.sh -d https://badssl.com                 

     htrace.sh v1.0.9  (openssl 1.1.0h: ok)


    URI: https://badssl.com

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   0.657482     0.657482        xxx.xxx.xxx.xxx:52236  xxx.xxx.xxx.xxx  104.154.89.105:443    US    https   1.1   200

Docker Alpine: geoip empty output

docker run --rm -it --name htrace.sh htrace.sh -u https://nmap.org -h --hide-src-ip

     htrace.sh v1.1.1  (openssl 1.1.1a: not tested)


    URI: https://nmap.org ; Method: GET

    req  time_total   time_connect    local_socket           via              remote_socket         geo   proto   ver   code     next_hop
    ---  ----------   ------------    ------------           ---              -------------         ---   -----   ---   ----     --------
 •   1   1.062680     1.062680        xxx.xxx.xxx.xxx:44328  xxx.xxx.xxx.xxx  45.33.49.119:443            https   1.1   200

Checks common name valid.

Checks common name valid:

• the certificate is valid only if the request hostname matches the certificate common name
• ssl certificate name mismatch error