urbanadventurer/whatweb

Next generation web scanner

whatweb's Issues

It doesn't run (archlinux 32 bit) 2

with reupdated git installation.

whatweb www.google.it
/usr/bin/whatweb:185: invalid multibyte char (US-ASCII) /usr/bin/whatweb:185: invalid multibyte char (US-ASCII) /usr/bin/whatweb:185: syntax error, unexpected $end, expecting '}' ...ugins.delete_if {|n,p| n == "¿" }.sort_by {|a,b| a.downcase... ... ^

I think it's a utf8 problem with ruby 1.9 series

Apache plugin matching incorrectly - Target.raw_headers contains multiple target's headers

if you saw Apache matching for things it shouldn't have, then it's because:
search=>"headers" was failing as it checked Target.raw_headers whcih contained headers from multiple targets.

Bug with reading .http files

Bug - with HTTP files only. HTML and META files are fine.

Ruby 1.8
< plugin-development/sites/fortune-100/www.chevron.com-.http [200] ASP_NET[2.0.50727], Adobe-Flash, Cookies[ASP.NET_SessionId], Email[[email protected]], Frame, HTTPServer[Microsoft-IIS/6.0], HttpOnly[ASP.NET_SessionId], JQuery, Meta-Author[Chevron Policy, Government and Public Affairs], Microsoft-IIS[6.0], Script[text/javascript], Title[%0D%0A%09Chevron Corporation Home - Human Energy%0D%0A], UrlRewriter_NET[2.0.0], X-Powered-By[UrlRewriter.NET 2.0.0]

Ruby 1.9

plugin-development/sites/fortune-100/www.chevron.com-.http [200] ASP_NET[2.0.50727], Adobe-Flash, Cookies[ASP.NET_SessionId], Email[[email protected]], Frame, HTTPServer[Microsoft-IIS/6.0], HttpOnly[ASP.NET_SessionId], JQuery, Microsoft-IIS[6.0], Script[%5C], Title[%5Cr%5Cn%5CtChevron Corporation Home - Human Energy%5Cr%5Cn], UrlRewriter_NET[2.0.0], X-Powered-By[UrlRewriter.NET 2.0.0]

incomplete solution --spider-skip-extensions

This may be error-prone because of blacklist approach.
Please use --spider-only-extensions (default: php,asp,aspx,jsp,jspx,do,cfm,...etc)

Multiple HTTP status codes cause plugins to crash [101, 102, 204, 205, 305]

Issue

Multiple plugins crash when a web server returns any of the following status codes:

• 101 Switching Protocols
• 102 Processing
• 204 No Content
• 205 Reset Content
• 305 Use Proxy

Priority

High

Cause

The HTTP protocol standard dictates no content should be returned after the HTTP header when returning status 204 or 205. The plugins are attempting to access content which does not exist.

Examples

• Status 101
  • 209.107.206.157
  • 96.17.71.157
• Status 102
  • 178.79.144.5
• Status 204
  • 64.236.79.202
  • 75.126.196.59
• Status 205
  • 125.60.19.87
  • 125.60.36.161
• Status 305
  • 112.90.136.71
  • 203.195.110.20

uncommon header -> x-xss-protection

It'd be better to add x-xss-protection to the known list in uncommon header.

And we can create new plugin for x-xss-protection that shows its value - whether the protection is disabled or not. I'll add it to TODO list.

localhost resolution fails with em-resolve-replace gem - ignores /etc/hosts?

This is a comparability problem with the gem and ruby < 1.9.
it officially requires ruby 1.9

Ruby1.9 HTTPS ERROR: undefined method `verify_mode' for nil:NilClass

./whatweb https://bugzilla.wikimedia.org/

https://bugzilla.wikimedia.org/ [200] Apache, Country[UNITED STATES][US], HTTPServer[Apache], IP[208.80.152.149], probably MediaWiki, OpenSearch[./search_plugin.cgi], PasswordField[Bugzilla_password], PoweredBy[Bugzilla], Script[text/javascript], Title[Bugzilla Main Page], UncommonHeaders[x-frame-options], X-Frame-Options[SAMEORIGIN]

ruby1.9.1 ./whatweb https://bugzilla.wikimedia.org/
https://bugzilla.wikimedia.org/ ERROR: undefined method `verify_mode' for nil:NilClass

Recursive spidering does not support redirects, cookies or HTTPS

Issue

Recursion fails in certain circumstances. See examples below.

Affected

• 0.4.5
• 0.4.6
• 0.4.7
• 0.4.8-dev and 0.4.8-dev-unstable
• Previous builds are untested but probably affected

Priority

High

Example - Redirect

$ ./whatweb -r microsoft.com

$ ./whatweb -r microsoft.com --debug
Exception `ArgumentError' at /usr/lib/ruby/1.8/net/http.rb:1470 - HTTP request path is empty

$ ./whatweb www.microsoft.com -r
http://www.microsoft.com/ [302] X-UA-Compatible[IE=EmulateIE8], HTTPServer[Microsoft-IIS/7.5], ASP.NET[2.0.50727], PoweredBy[Bing], UncommonHeaders[vtag], IP[65.55.12.249], JQuery, X-Powered-By[ASP.NET], Microsoft-IIS[7.5], Title[Microsoft Corporation: Software, Smartphones, Online, Games, Cloud Computing, IT Business Technology, Downloads], Country[UNITED STATES][US]
(truncated - appears to work fine)

Example - HTTPS

$ ./whatweb https://github.com/
https://github.com/ [200] X-UA-Compatible[chrome=1], HTTPServer[nginx/0.7.67], Google-Analytics[UA-3769691-2], PoweredBy[the], UncommonHeaders[strict-transport-security], HTML5, nginx[0.7.67], IP[207.97.227.239], JQuery, Cookies[_gh_sess,csrf_id], Title[Secure source code hosting and collaborative development - GitHub], OpenSearch[/opensearch.xml], Country[UNITED STATES][US]

$ ./whatweb https://github.com/ -r
https://github.com/ [400] HTTPServer[nginx/0.7.67], nginx[0.7.67], IP[207.97.227.239], Title[400 The plain HTTP request was sent to HTTPS port], Country[UNITED STATES][US]

$ ./whatweb https://www.treshna.com
https://www.treshna.com [200] HTTPServer[Debian Linux][Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny10 with Suhosin-Patch proxy_html/3.0.0 mod_ssl/2.2.9 OpenSSL/0.9.8g], Google-Analytics[UA-386922-1], Apache[2.2.9][mod_ssl/2.2.9,proxy_html/3.0.0], IP[210.48.71.198], JQuery, PHP[5.2.6-1+lenny10][Suhosin-Patch], OpenSSL[0.9.8g], Title[treshna Enterprises Ltd — Open Source Linux software developer], Country[NEW ZEALAND][NZ]

$ ./whatweb https://www.treshna.com -r

Example - Cookies

Only the last cookie is returned.

$ ./whatweb www.cooking.com -p Cookies
http://www.cooking.com [200] Cookies[ASPSESSIONIDCARCSSCA,CCREFID,CCVC,Region,SITESERVER,ajaxsubscribe]

$ ./whatweb www.cooking.com -r -p Cookies
http://www.cooking.com/ [200] Cookies[ajaxsubscribe]

Based on this information the issue is probably related to the way in which the anemone library is used or perhaps the library itself, however further testing is required.

-p argument fails for plugin names when both names and paths are provided

Issue

The -p argument fails for plugin names when both plugin names and plugin paths are provided.

Priority

Low

Logs

$ ./whatweb -p title,plugins/robots.txt.rb whatweb.net
Error: The following plugins were not found: title
No plugins selected, exiting.

$ ./whatweb -p plugins/title.rb,plugins/robots.txt.rb whatweb.net
http://whatweb.net [200] Title[WhatWeb.net - Online Scan]

Cause

whatweb at around line 360:

            # load files from plugin_dirs unless a file is minused
            plugin_dirs.each do |d|
                    # if a folder, then load all files
                    if File.directory?(d)
                            (Dir.glob("#{d}/*.rb")-minus_files).each {|x| load_plugin(x) }
                    elsif File.exists?(d)
                            load_plugin(d)
                    else
                            error("Error: #{d} is not Dir or File")
                    end
            end

this calls load_plugins() which overwrites Plugins.registered which becomes an issue about 20 lines further on:

            if b.map {|c| c.modifier }.include?(nil)
                    selected_plugin_names=[]
            else
                    selected_plugin_names = Plugin.registered_plugins.map {|n,p| n.downcase }
            end

Using the example:

$ ./whatweb -p title,plugins/robots.txt.rb whatweb.net

Plugin.registered_plugins is set to the details of only the robots.txt plugin, thus title will never match.

Crashes on attempting to read file without read permission

./whatweb /etc/shadow
/usr/lib/ruby/1.8/open-uri.rb:32:in initialize': Permission denied - /etc/shadow (Errno::EACCES) from /usr/lib/ruby/1.8/open-uri.rb:32:inopen_uri_original_open'
from /usr/lib/ruby/1.8/open-uri.rb:32:in open' from ./whatweb:1286 from ./whatweb:1213:ininitialize'
from ./whatweb:1213:in `new'
from ./whatweb:1213

Latest Code not updated

the latest code which is available on http://www.morningstarsecurity.com/research/whatweb is not exactly the same as available here.

merge request: ms outlook web access

Hi bcoles / Andrew

Can you please merging the outlook plugin with mine if appropriate?

https://github.com/urbanadventurer/WhatWeb/blob/master/plugins/Outlook-Web-App.rb

https://github.com/yehgdotnet/whatweb-plugins/blob/master/new-plugins/microsoft-outlook-web-access.rb

Thanks.

subverting snort rule

I noticed snort rule was out
http://article.gmane.org/gmane.comp.security.ids.snort.emerging-sigs/4262/match=whatweb

Even if whatweb does passive scan, it'll be detected if users don't take pain to change user-agent.

So, it violates the purpose of whatweb.

Just have generic browser signature.

Unexpected behaviour with :url and :status in plugins - Incorrect Traffic Inspector Matches

this currently matches any page at / with the status of 403.

in this case it makes sense to expect the url AND the status AND the string need to be found to match.

plugins/Traffic-Inspector.rb:{ :url=>"/", :status=>403, :string=>/<title>Error<\/title><\/head><body><h1>403 - Forbidden<\/h1><hr( class="footer")?>Traffic [Ii]nspector HTTP\/FTP\/Proxy server \([^\)]+\)<br>([^<^\/]+)\s*\/?\s*[\d]{2}\.[\d]{2}\.[\d]{2}/, :offset=>1 },

I want URL to be optional and to automatically become an aggressive test. Any thoughts?

:url can't accept special characters

:url:=> '<><' will give u error - Bad URI

I suggest whatweb encode it automatically.

File reading with Ruby1.9

Bug with file input with ruby1.9. Ruby1.8 is fine

ruby1.9.1 ./whatweb --log-brief b1.9.1 plugin-development/sites/alexa-top-100/*

Lots of new lines caused by whatweb engine
Many plugins fail on regular expressions

ERROR: Plugin S-CMS failed for plugin-development/sites/alexa-top-100/4shared.com.html. incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
ERROR: Plugin AnyGate failed for plugin-development/sites/alexa-top-100/ebay.com.html. incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
ERROR: Plugin Netsnap-Web-Camera failed for plugin-development/sites/alexa-top-100/apple.com.html. incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)

Threads for some targets will never exit

ruby1.9.1 ./whatweb --follow-redirect never -vvvv -p title --log-brief b1.9.1 plugin-development/sites/alexa-top-100/about.com.html

non-English ASCII characters cause a partial overwrite of some output data

Some non-English ASCII characters cause a partial overwrite of some output data. Furthermore, some characters act as newline characters which breaks up the output.

Priority

Low

Examples

• Unionsky.cn
• fc2.com

Logs

$ ./whatweb Unionsky.cn
http://Unionsky.cn [200] PasswordField[pwd], Meta-Author[å¼å¤©å¹¿åèç-æææ¯æéå
¬å¸], HTTPServer[Microsoft-IIS/6.0], ASP.NET[2.0.50727], Adobe-Flash, IP[218.108.237.6], X-çowered-By[ASP.NET], Microsoft-IIS[6.0], Title[å¼å¤©å¹¿åèç---å
·æé¢å¯¼å°ä½ 第ä¸æ¹å¹¿åèç], Country[CHINA][CN]

Note X-çowered-By

$ ./whatweb fc2.com
http://fc2.com [200] Frame, Meta-Author[FC2.inc], HTTPServer[Unix][Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8 PHP/4.4.9 mod_perl/2.0.4 Perl/v5.6.1], Google-Analytics[UA-7509326-1], Apache[2.0.63][mod_perl/2.0.4,mod_ssl/2.0.63], IP[208.71.106.124], JQuery, PHP[4.4.9], OpenSSL[0.9.8], Title[FC2 -ç
                                                               ¡æãã­ã° ç
                                                                        ¡æåç» ç
                                                                               ¡æã¼ã ãã¼ã¸ ã¬ã³ã¿ã«ãµã¼ãã¼ ç
                            ¡æã¢ã¯ã»ã¹è§£æ SEO対ç­ã
                                                    ã¼ã«ãªã©-], Perl[5.6.1], Country[UNITED STATES][US]

:url -> relative path to web root

If I understood how whatweb works,

After analyzing several plugins, I've found a lot of values in :url are set /

:url=>'/resources/this-app-only.xml'

If you use this approach, I doubt that using recursive approach will useless because

In recursive mode, everytime whatweb spider goes to each dir of

www.site.com/sub1/sub2/subsub3/

it will keep on requesting

www.site.com/resources/this-app-only.xml

So, I wish you, plugins author, to use

:url=>'resources/this-app-only.xml'

Then

If I will provide this url
site.com/subdir/

A plugin will request
site.com/subdir/resources/this-app-only.xml

If I will provide this url with recursive mode
site.com/

A plugin will request
site.com/resources/this-app-only.xml
site.com/subdir/resources/this-app-only.xml

Hope you get what I mean.

new plugins - MapServer & HopfTimeServer

http://bl0g.yehg.net/2011/07/whatweb-new-plugins-mapserver-hopf-time.html

https://github.com/yehgdotnet/whatweb-plugins/blob/master/new-plugins/MapServer.rb

$ ./whatweb --follow-redirect=same-domain -a 4 -v -p MapServer http://demo.mapserver.org/

demo.mapserver.org/cgi-bin/mapserv/?map=* [200]
http://demo.mapserver.org [200] MapServer[Invalid Map Parameter Detection,Version - 5.6.5 ]   

https://github.com/yehgdotnet/whatweb-plugins/blob/master/new-plugins/HopfTimeServer.rb

$ ./whatweb --follow-redirect=same-domain -a 4 -v -p HopfTimeServer http://www.timesync.eu/

www.timesync.eu/ [200]
www.timesync.eu/cgi-bin/main.cgi?ntp&0 [200]
www.timesync.eu/cgi-bin/main.cgi?ntp&0 [200]
http://www.timesync.eu/ [200] HopfTimeServer[Generic Version - 727x,Version - 727100]   

Custom-Plugin, when defined, is not automatically added to the selected plugins list

Custom-Plugin cannot be used when plugins are selected.

The following will not include CustomPlugin results:
eg. ./whatweb -ptitle,phpbb --custom-plugin ":text=>'viewprofile'" www.phpbb.com/community/

Country plugin can't handle IPv6 addresses

When the website to test resolves to IPv6 address, the country plugins fail (char ipstr not intended to handle IPv6 addresses):

$ ./whatweb --debug wwww.iroqwa.org
Exception `RangeError' at /usr/share/whatweb//plugins/country.rb:76 - 2001 out of char range
ERROR: Plugin Country failed for http://wwww.iroqwa.org. 2001 out of char range
Exception `RangeError' at ./whatweb:735 - 2001 out of char range
/usr/share/whatweb//plugins/country.rb:76:in `chr': 2001 out of char range (RangeError)
    from /usr/share/whatweb//plugins/country.rb:76:in `passive'
    from /usr/share/whatweb//plugins/country.rb:76:in `map'
    from /usr/share/whatweb//plugins/country.rb:76:in `passive'
    from ./lib/plugins.rb:135:in `x'
    from ./whatweb:726:in `run_plugins'
    from ./whatweb:708:in `each'
    from ./whatweb:708:in `run_plugins'
    from ./whatweb:1308
    from ./whatweb:1222:in `initialize'
    from ./whatweb:1222:in `new'
    from ./whatweb:1222

I use 0.4.7 version, but the code/issue seems to be the same in 0.4.8-dev.

country.rb Country[ZZ] Vs Private IP ranges

when scanning local LAN,

./whatweb -v 192.168.1.1

192.168.1.1/ [501]
http://192.168.1.1/ [501] Country[ZZ],

I think country.rb should exclude private IP range.

updated ./plugin-development/get-pattern

http://bl0g.yehg.net/2011/07/whatweb-updated-plugin-developmentget.html

https://github.com/yehgdotnet/whatweb-plugins/blob/master/plugin-development/get-pattern

Added server,cookie,www-authenticate header in /plugin-development/get-pattern

---

$ ./get-pattern http://demo.phpmyadmin.net/master/

== Page Pattern Generator 0.1 for WhatWeb ==
by Aung Khant, http://yehg.net

URL: http://demo.phpmyadmin.net/master/

{:name=>'Page MD5', :md5=>'619ef6970f8609c42b944ea776734663'},

{:name=>'HTML Tag Pattern', :tagpattern=>'!doctype,html,head,meta,link,link,title,/title,link,link,link,meta,script,/script,script,/script,script,/script,script,/script,script,/script,script,/script,script,/script,script,/script,/head,body,script,/script,div,h1,/h1,a,/a,/div,div,a,img,/a,h1,bdo,/bdo,/h1,form,input,input,input,input,input,input,fieldset,input,legend,/legend,select,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,option,/option,/select,/fieldset,noscript,fieldset,input,input,/fieldset,/noscript,/form,br,!--,form,input,fieldset,input,legend,a,img,/a,/legend,div,label,/label,input,/div,div,label,/label,input,/div,div,label,/label,select,option,/option,option,/option,option,/option,/select,/div,/fieldset,fieldset,input,input,input,input,input,/fieldset,/form,div,div,/div,/div,/div,div,a,/a,a,/a,a,/a,br,a,/a,a,/a,a,img,/a,/div,div,/div,!--,script,/script,script,/script,noscript,p,img,/p,/noscript,!--,/body,/html'},

m << {:name=>'Server Header' } if @meta["server"] =~ /lighttpd/1.4.29/

m << {:name=>'Cookie Header' } if @meta["set-cookie"] =~ /phpMyAdmin=j14qlusvh9r49qrgr7nopurjbc8urh33; path=/master/; HttpOnly, pma_lang=en; expires=Mon, 29-Aug-2011 11:29:49 GMT; path=/master/; httponly, pma_collation_connection=utf8_general_ci; expires=Mon, 29-Aug-2011 11:29:49 GMT; path=/master/; httponly, pma_mcrypt_iv=OqRCT7x%2BMPc%3D; expires=Mon, 29-Aug-2011 11:29:49 GMT; path=/master/; httponly, phpMyAdmin=gq42877ciul1j0484gatvpgpbogkgnn3; path=/master/; HttpOnly/

HTTPS with method CONNECT

Hello

I try whatweb with website on HTTPS. And I've this error 'undefined method `verify_mode' for nil:NilClass' and when i read the code on lib/target.rb, there are no case when it's the method CONNECT.

Thanks for your feedback

[new options] --disable-plugins, --disable-plugins-regexp, -run-plugins-regexp

At times, users might want to disable some plugins.

--disable-plugins

This will take lists of plugin separated by commas.
--disable-plugins joomla,mambo

---

At times, users might want to allow/disable only some plugins of their choice/desired.

--disable-plugins-regexp
-run-plugins-regexp

This will take plugin by regexp separated by commas

--disable-plugins-regexp cms,blog
-run-plugins-regexp cms,blog

This is particularly useful when

• I see a site
• And it looks like blog
• I see whatweb passive mode doesn't show anything
• I want to scan it with aggressive mode
• And I don't want to send huge traffic like Nikto

Then I will run whatweb with -run-plugins-regexp blog which I expect to scan all blog's aggressive methods

---

Please do not monkey-patching Net::HTTP

Sweet baby Jesus, please do not directly override Net::HTTP methods. You could simply inherit Net::HTTP into a new sub-class.

'new redirecting broken' error caused by meta refresh regex in ./whatweb lines 1167-1168

Cause

The issue is due to the following two lines 1167-1168 in ./whatweb :

1167: if body =~ /<meta[^>]*http\-equiv[^>]*refresh[^>]*url=([^\"]*)/i
1168: metarefresh=body.scan(/<meta[^>]*http\-equiv[^>]*refresh[^>]*url=([^\"]*)/i)[0].to_s

Unless I'm mistaken, the correct regex should be :

1167: if body =~ /<meta[^>]*http\-equiv[^>]*refresh[^>]*url=([^"^']*)[^>]*>/i
1168: metarefresh=body.scan(/<meta[^>]*http\-equiv[^>]*refresh[^>]*url=([^"^']*)[^>]*>/i)[0].to_s

or, alternatively :

1167: if body =~ /<meta[\s]+http\-equiv[\s]*=[\s]*['"]?refresh['"]?[^>]+content[\s]*=[^>]*[0-9]+;[\s]*url=['"]?([^"^'^>]+)['"]?[^>]*>/i
1168: metarefresh=body.scan(/<meta[\s]+http\-equiv[\s]*=[\s]*['"]?refresh['"]?[^>]+content[\s]*=[^>]*[0-9]+;[\s]*url=['"]?([^"^'^>]+)['"]?[^>]*>/i)[0].to_s

This successfully extracts the URL even when the tag is malformed, such as :

<meta http-equiv=Refresh content=0;URL='./default.aspx'>

Logs

The following logs are available :

$ ./whatweb http://ubi.com/US/
http://ubi.com/US/ [200] X-Powered-By[ASP.NET], Microsoft-IIS[5.0], IP[216.98.48.35], ASP.NET, HTTPServer[Microsoft-IIS/5.0], Meta-Refresh-Redirect[./default.aspx], Country[CA], MetaGenerator[Microsoft Visual Studio .NET 7.1]
new redirecting broken: bad URI(is not URI?): './default.aspx'>

Ruby 1.9 Issues

Apparently whatweb doesn't work on ruby 1.9. can someone test this and give me feedback?

It doesn't run (archlinux 32 bit)

whatweb http://www.tntvillage.org/
/usr/bin/whatweb:234:in load': /usr/share/whatweb/plugins/bm-classifieds.rb:59: invalid multibyte char (US-ASCII) (SyntaxError) /usr/share/whatweb/plugins/bm-classifieds.rb:59: invalid multibyte char (US-ASCII) /usr/share/whatweb/plugins/bm-classifieds.rb:59: syntax error, unexpected $end, expecting '}' ...All source code on this site © 2007 BM Scripts unless other... ... ^ from /usr/bin/whatweb:234:inblock (2 levels) in load_plugins'
from /usr/bin/whatweb:234:in each' from /usr/bin/whatweb:234:inblock in load_plugins'
from /usr/bin/whatweb:234:in each' from /usr/bin/whatweb:234:inload_plugins'
from /usr/bin/whatweb:795:in `

'

logs don't flush

make logs flush after each line

Gem.available? is deprecated

ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

Got a few warning when running whatweb.

NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:55.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:66.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:76.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:86.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:898.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:899.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:900.
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from ./whatweb:901.

XML and MagicTree XML is malformed when appended to an existing XML file

The opening and closing XML and XSLT tags are written to file each time WhatWeb is executed. The XML is malformed in both --log-xml and --log-magictree output formats if appended to an existing XML file.

For example, the following works correctly :

$ ./whatweb --log-xml=log.xml whatweb.net whatweb.net/robots.txt

The following results in malformed XML due to duplicated <xml> tags :

$ ./whatweb --log-xml=log.xml whatweb.net
$ ./whatweb --log-xml=log.xml whatweb.net/robots.txt

redirection hell

With aggressive mode, when testing on site http://www.somewhereinblog.net/

Requesting non-existent files makes http://www.somewhereinblog.net/404 do redirection with 301. It makes whatweb keeps following till an error occurs "too many redirects". The fact is when the site receive non-existent URLs (js,img,ico), it issues 301.

We can do th things:

1. If our :url is a type of static files such as ico,js,css,jpg,png,gif,
   then we make whatweb not to follow 301 location

2. For others like test.asp , /administrator/login.php ,
   we will need to have new option
   --max-redirect

   We can't use --no-redirect for always as 301 doesn't always indicate "File not found". It sometimes means for member-only sessions.

The RSS feed plugin does not return the correct link

It often returns any other link, for example the favicon.ico image or something similar.

Examples: www.bonnier.se and www.idg.se

__VIEWSTATE kills plugin

I think this amazingly large __VIEWSTATE variable kills WhatWeb for me:
http://ubahn.se/

When reading from files, the filename matters

example error:

./whatweb index.html.1
the scheme file does not accept registry part: index.html.1 (or bad hostname?)

./whatweb index.html without the .1 is fine.

install fails because of missing TODO file.

Running make install fails because the TODO file is not present, please either remove references to it from the Makefile or create an empty TODO file (the latter makes more sense imho).
Regards. (And a merry Xmas 🎅)

whatweb exception in output.rb when specifying with -r option for some ports [ruby 1.8.7]

./whatweb 173.242.114.45:2082 -r

    ./lib/output.rb:36:in `<=>': can't convert String into Array (TypeError)
    from ./lib/output.rb:36:in `sort'
    from ./lib/output.rb:36:in `suj'
    from ./lib/output.rb:35:in `map'
    from ./lib/output.rb:35:in `suj'
    from ./lib/output.rb:111:in `out'
    from ./lib/output.rb:109:in `each'
    from ./lib/output.rb:109:in `out'
    from ./whatweb:1109
    from ./whatweb:1108:in `each'
    from ./whatweb:1108
    from ./whatweb:1107:in `synchronize'
    from ./whatweb:1107
    from ./lib/anemone/core.rb:174:in `call'
    from ./lib/anemone/core.rb:174:in `do_page_blocks'
    from ./lib/anemone/core.rb:173:in `each'
    from ./lib/anemone/core.rb:173:in `do_page_blocks'
    from ./lib/anemone/core.rb:121:in `run'
    from ./lib/anemone/core.rb:113:in `loop'
    from ./lib/anemone/core.rb:113:in `run'
    from ./lib/anemone/core.rb:39:in `crawl'
    from ./lib/anemone/core.rb:30:in `call'
    from ./lib/anemone/core.rb:30:in `initialize'
    from ./lib/anemone/core.rb:37:in `new'
    from ./lib/anemone/core.rb:37:in `crawl'
    from ./lib/anemone/anemone.rb:58:in `crawl'
    from ./whatweb:1083
    from ./whatweb:1074:in `initialize'
    from ./whatweb:1074:in `new'
    from ./whatweb:1074

WhatWeb Gem

Hey, do you plan to convert this into a gem if I want to use it in Rails or any other framework? Right now, I am using the system command and getting the output.

Modules returned in :modules=> are not comma separated

Modules returned in :modules=> are not comma separated as versions are in :version=>

This is an issue because it makes the modules unreadable.

Example:
./whatweb -a 1 --log-brief=asdf.log commande.geekheberg.net/order/

Output (Excerpt):
TheHostingTool[1.2.2,MySQL:5.0.91,OS:Linux,PHP:5.2.14][HTTPFTPMySQLPOP3SSH]

Expected Output:
TheHostingTool[1.2.2,MySQL:5.0.91,OS:Linux,PHP:5.2.14][HTTP,FTP,MySQL,POP3,SSH]

Ruby Code:

        # Module detection
        if @body =~ /<td align="center"><strong>([^<]+)<\/strong><\/td>/
                modules=@body.scan(/<td align="center"><strong>([^<]+)<\/strong><\/td>/).to_s
                m << { :modules=>modules }
        end

Line breaks in brief log output

Line breaks appear in logs. In brief output to the screen it's fine.

Example:
$ ./whatweb --log-brief linebreak.log http://www.news24.com

http://www.news24.com [200] ASP_NET[4.0.30319], Cookies[ASP.NET_SessionId,News24LocationCookie], Country[SOUTH AFRICA][ZA], Facebook-Plugin[likebox], Frame, HTML5, HTTPServer[Microsoft-IIS/7.5], HttpOnly[ASP.NET_SessionId], IP[41.86.110.200], JQuery, Microsoft-IIS[7.5], OpenGraphProtocol[website][136805909671416], Prototype, Script[JavaScript,javascript,text/javascript,text/x-jquery-tmpl], Title[%0D%0A%09News24, South Africa's premier news source, provides breaking news on national, world, Africa, sport, entertainment, technology & more.%0A%0D%0A], UncommonHeaders[contenttemplateurl], X-Powered-By[ASP.NET]

$ cat linebreak.log
http://www.news24.com [200] ASP_NET[4.0.30319], Cookies[ASP.NET_SessionId,News24LocationCookie], Country[SOUTH AFRICA][ZA], Facebook-Plugin[likebox], Frame, HTML5, HTTPServer[Microsoft-IIS/7.5], HttpOnly[ASP.NET_SessionId], IP[41.86.110.200], JQuery, Microsoft-IIS[7.5], OpenGraphProtocol[website][136805909671416], Prototype, Script[JavaScript,javascript,text/javascript,text/x-jquery-tmpl], Title[
News24, South Africa's premier news source, provides breaking news on national, world, Africa, sport, entertainment, technology & more.

], UncommonHeaders[contenttemplateurl], X-Powered-By[ASP.NET]

ASP.NET plugin causes error when logging to Mongo

Hitting a site that triggers the ASP.NET plugin causes an error when logging to mongo, due to the "." in the plugin name. Changing it to ASPDOTNET fixes that. I'm staging a pull request that impliments that.

(for the run below I've added code to print the error and backtrace to whatweb.rb)

$ whatweb --aggression 1 --log-mongo-database test --log-mongo-collection whatwebtest --log-mongo-host localhost http://msn.com
http://msn.com [301] Charset[ASCII], HTTPServer[Microsoft-IIS/6.0], ASP.NET, RedirectLocation[http://www.msn.com/], UncommonHeaders[s], IP[65.55.206.203], X-Powered-By[ASP.NET], Microsoft-IIS[6.0], Title[Document Moved], Country[UNITED STATES][US]
Error: Logging failed for http://msn.com ASP.NET - key must not contain '.'

["/var/lib/gems/1.8/gems/bson-1.5.2/lib/bson/bson_c.rb:24:in serialize'", "/var/lib/gems/1.8/gems/bson-1.5.2/lib/bson/bson_c.rb:24:inserialize'",
"/var/lib/gems/1.8/gems/mongo-1.5.2/lib/mongo/collection.rb:940:in insert_documents'", "/var/lib/gems/1.8/gems/mongo-1.5.2/lib/mongo/collection.rb:939:ineach'",
"/var/lib/gems/1.8/gems/mongo-1.5.2/lib/mongo/collection.rb:939:in insert_documents'", "/var/lib/gems/1.8/gems/mongo-1.5.2/lib/mongo/collection.rb:343:ininsert'", "/home/ubuntu/malware_factors/tools/whatweb/whatweb-0.4.7/lib/output.rb:669:in out'", "whatweb/whatweb-0.4.7/whatweb:1306", "whatweb/whatweb-0.4.7/whatweb:1304:ineach'",
"whatweb/whatweb-0.4.7/whatweb:1304", "whatweb/whatweb-0.4.7/whatweb:1213:in initialize'", "whatweb/whatweb-0.4.7/whatweb:1213:innew'", "whatweb/whatweb-0.4.7/whatweb:1213"]

How is it possible to pull out just certain types of information from each website?

Hi there,

First off, congrats on the awesome program and suite of plugins you have managed to get so far.

I am looking to crawl through a large number of websites and pull out only very specific pieces of information from each website, in particular I am looking to pull out

• IP
• Country
• CMS used
• Webserver used

Is there a way to get whatweb to extract only certain parameters? And when you get these parameters back, is there a way to identitfy what 'type' of information it is?

In particular the CMS match seems like it would be the hardest, this is an example snippet of JSON verbose output:

["Drupal",[{"text":"jQuery.extend(Drupal.settings,","certainty":100,"regexp_compiled":"(?-mix:jQuery\\.extend\\(Drupal\\.settings,)"}]]

At no point in the above output, does it mention that this is the CMS used.

What are your thoughts here?

filepaths not recorded by php-error plugin in brief logging

http://www.goth-greetings.com/donate.php [200] Frame, PasswordField[upw], Meta-Author[Linda Peltola & Adrian Brooks], HTTPServer[Unix][Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.3.4 mod_perl/2.0.4 Perl/v5.8.8], PHP-Error[ggadmin][/home/ggadmin/public_html/donate.php,/home/ggadmin/public_html/includes/browser_class.inc], Google-Analytics[UA-2251779-2], Apache[2.2.17][mod_bwlimited/1.4,mod_perl/2.0.4,mod_ssl/2.2.17], IP[168.144.38.176], PHP[5.3.4], OpenSSL[0.9.8e-fips-rhel5], X-Powered-By[PHP/5.3.4], Title[Goth Greetings Free E-Card Service : Donations], Perl[5.8.8], Country[CANADA][CA]

$ cat b.log
http://www.goth-greetings.com/donate.php [200] Frame, PasswordField[upw], Meta-Author[Linda Peltola & Adrian Brooks], HTTPServer[Unix][Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.3.4 mod_perl/2.0.4 Perl/v5.8.8], PHP-Error[ggadmin][], Google-Analytics [UA-2251779-2], Apache[2.2.17][mod_bwlimited/1.4,mod_perl/2.0.4,mod_ssl/2.2.17], IP[168.144.38.176], PHP[5.3.4], OpenSSL[0.9.8e-fips-rhel5], X-Powered-By[PHP/5.3.4], Title[Goth Greetings Free E-Card Service : Donations], Perl[5.8.8], Country[CANADA][CA]

store response in database

I wanted to store the response in the database so that when i develop a new plugin i dont need to crawl the site again.
What and where should the response shud be stored , where shud it be the feed got the next round.

Matches throwing TypeError at ./lib/output.rb:36 - can't convert String into Array

Cause

Returning matches using two or more of these styles for matching (within the same plugin) . . .

• { :version=>//, :regexp_offset=>0 } - in matches[]
• { :version=>"1.2.3", :text=>"1.2.3" } - in matches[]
• m << { :version=>@meta["server"].scan(//) } - in def passive[]

. . . Throws the following error :

Exception 'TypeError' at ./lib/output.rb:36 - can't convert String into Array

Priority

This issue will need to be resolved before 0.4.6-stable is released.

Affected Versions

Untested. Possibly due to one of these commits in 0.4.6-dev :

• 376fd48
• 87f103b

Workaround

Force result data to be returned as a single element array, for example :

• { :version=>["1.2.3"], :text=>"1.2.3" } - in matches[]

Logs

The following logs are available :

$ ./whatweb 173.242.114.45:2082 -p plugins/cpanel.rb http://173.242.114.45:2082ERROR: Logging failed.

$ ./whatweb 173.242.114.45:2082 -p plugins/cpanel.rb --debug -v Exception EOFError' at /usr/lib/ruby/1.8/net/protocol.rb:135 - end of file reached
173.242.114.45/ [401]
Exception TypeError' at ./lib/output.rb:36 - can't convert String into Array http://173.242.114.45:2082ERROR: Logging failed. CPanel => (version: 11), (version: 11.26)

$ ./whatweb 173.242.114.45:2082 -p plugins-disabled/http-headers.rb http://173.242.114.45:2082 [401] HTTP-Headers[connection: close,content-type: text/html,server: cpsrvd/11.26,set-cookie: logintheme=web-leader; path=/; HttpOnly; port=2082, cprelogin=no; path=/; HttpOnly; port=2082, cpsession=eTjx5lIe8tGI5RWKpoTSFu9_nYXvp5sYOaKsdSH_slbpRazRpD_3hoER9K3P2mdS; path=/; HttpOnly; port=2082]

$ ./whatweb 173.242.114.45:2082 -p plugins-disabled/http-headers.rb -r http://173.242.114.45:2082/ [401] HTTP-Headers[connection: close,content-type: text/html,server: cpsrvd/11.26,set-cookie: logintheme=web-leader; path=/; HttpOnly; port=2082cprelogin=no; path=/; HttpOnly; port=2082cpsession=Fj3AykX1ngMo4WwjvWX8WdsijtMJb0x6h9sleSnCTlgPA1rvTEQy6jj8nGUCTsvE; path=/; HttpOnly; port=2082]

[low priority] Encoding of special characters is inconsistent across logging types

Encoding for various kinds of logging is inconsistent.

Brief and XML logging will URL encode newlines
JSON and verbose logging will not.

Discussion?

Plugin output causes output to STDOUT to be overwritten

It appears that \n characters (^M) in the output from plugins causes whatweb to begin output from the beginning of the line.

Affected

• Output to STDOUT

Unaffected

• Output to the log file

Testing

Using -v produces the following error:

Exception ArgumentError' at ./lib/output.rb:43 - negative argument ./lib/output.rb:43:in*': negative argument (ArgumentError)
from ./lib/output.rb:43:in out' from ./lib/output.rb:41:ineach'
from ./lib/output.rb:41:in out' from ./whatweb:964 from ./whatweb:963:ineach'
from ./whatweb:963
from ./whatweb:962:in synchronize' from ./whatweb:962 from ./whatweb:882:ininitialize'
from ./whatweb:882:in `new'
from ./whatweb:882

How to re-create the bug

Pull the Aruba-Mobility-Controller-Config-File plugin.

Run the plugin against one of the example URLs:

./whatweb -a 1 --log-brief=asdf.log -p Aruba-Mobility-Controller-Config-File www.opus1.com/nac/ny06configs/NAP-ARUBA-AP.CFG

Output

[[http://whatweb.net/whatweb-issue.png]]

Expected Output:

This output was pulled from the log file. It is unaffected by the bug.

http://www.opus1.com/nac/ny06configs/NAP-ARUBA-AP.CFG [200] Aruba-Mobility-Controller-Config-File[2.5] [root 4ed80428b077988f96acebd46c0f8317ad7bd45f2f13d7ab^M][snmp-trap udp 162^M,syslog udp 514^M,l2tp udp 1701^M,ike udp 500^M,https tcp 443^M,smb-tcp tcp 445^M,dhcp udp 67 68^M,pptp tcp 1723^M,sccp tcp 2000^M,telnet tcp 23^M,sip-tcp tcp 5060^M,tftp udp 69^M,kerberos udp 88^M,adp udp 8200^M,pop3 tcp 110^M,rtsp tcp 554^M,msrpc-tcp tcp 135 139^M,dns udp 53^M,vocera udp 5002^M,http tcp 80^M,sip-udp udp 5060^M,nterm tcp 1026 1028^M,papi udp 8211^M,natt udp 4500^M,ftp tcp 21^M,svp 119^M,smtp tcp 25^M,gre 47^M,smb-udp udp 445^M,esp 50^M,snmp udp 161^M,bootp udp 67 69^M,msrpc-udp udp 135 139^M,ntp udp 123^M,icmp 1^M,ssh tcp 22^M]

Solution

Some way to replace all instances of \r in output?

Country and IP plugins do not run during recursive scans

The Country and IP plugins do not run during recursive scans.

The plugins should run at least once per host for the following reasons:

• MagicTree data is arranged with host nodes at the top of the tree. If no IP is present the host node title is blank which makes analyzing the data more difficult.
• Anyone using custom scripts which rely on WhatWeb output may have unforeseen issues if they're unaware of this issue.
• The lack of IP and country details gives the impression that WhatWeb is inconsistent

Cause

These plugins don't run during recursive mode in order to decrease processing time for two main reasons:

• The country plugin reads from two database files to retrieve the results
• These plugins will always return the same results during a recursive scan of a host

Priority

Medium

Solution

Detecting recursive mode within a plugin is tedious at best and would best be handled by the WhatWeb core. Ideally during recursive mode the plugins will be run once and the returned values stored in memory to be returned for each URL.

Workaround

Run WhatWeb once against the target, then again in recursive mode. Unfortunately the logs will be malformed in XML or MagicTree logging modes due to [[issue# 39|https://github.com/urbanadventurer/WhatWeb/issues#issue/39]](XML and MagicTree XML is malformed when appended to a log file)