secynic/ipwhois

Retrieve and parse whois data for IPv4 and IPv6 addresses

ipwhois's Issues

Add retry_count to lookups

Add retry_count to IPWhois.lookup() and IPWhois.lookup_rws()

Lookup Stalls and Fails for 191. Networks

When I run a script against a list of netflow entries, addresses in the 191 network stall my script, when I kill it I see a WHOIS lookup failed.

File "whois.py", line 44, in <module>
    results = obj.lookup()
File "/usr/local/lib/python2.7/site-packages/ipwhois/ipwhois.py", line 125, in lookup
    ignore_referral_errors, asn_data, field_list
File "/usr/local/lib/python2.7/site-packages/ipwhois/whois.py", line 554, in lookup
    extra_blacklist=extra_blacklist)
File "/usr/local/lib/python2.7/site-packages/ipwhois/net.py", line 468, in get_whois
    'WHOIS lookup failed for %r.' % self.address_str
ipwhois.exceptions.WhoisLookupError: WHOIS lookup failed for '191.234.210.42'

Add APNIC RWS support

Add APNIC RWS support using rdap.apnic.net

Add examples

Add an examples directory, with an initial set of simple usage examples.

Proxy support

Add feature to make the proxy settings local and not global for urllib.request via a proxy opener object.

Add in support to change it to:
urllib.request.ProxyHandler({}) - No proxy
urllib.request.ProxyHandler() - System settings

Add offline tests with sample whois responses.

I'm willing to write some unittest using sample data taken from whois.

So it's only test code and json data.
Let me know if it's fine.

Unicode error causes lookup to throw WhoisLookupError

Test with 68.65.148.250.

The following exception is thrown:

Python 2.6.8 (unknown, Nov 7 2012, 14:47:45)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from ipwhois import IPWhois
test=IPWhois('68.65.148.250')
test.lookup()
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/site-packages/ipwhois/ipwhois.py", line 679, in lookup
response = self.get_whois(results['asn_registry'], retry_count)
File "/usr/lib/python2.6/site-packages/ipwhois/ipwhois.py", line 537, in get_whois
'Whois lookup failed for %r.' % self.address_str
ipwhois.ipwhois.WhoisLookupError: Whois lookup failed for '68.65.148.250'.

Digging deeper, it seems that a unicode conversion error is to blame. Fix coming.

cymru is down, ipwhois permutation causes ourself to get blacklisted

o_o

we need a flag in Net() to avoid permutations.

CYMRU DNS is currently offline. because DNS is yielding timeout errors, lookup_asn() is retrying via other methods -- which gets us blacklisted (stalled lookups leading to timeouts) from the other services -- which now yields several minutes per each lookup before finally giving up.

we need to pass a use_dns_only to the ASN lookup so if cymru is down, we don't punish ourselves by quickly getting blacklisted due to trying alternate means of obtaining results.

Add support for KRNIC and JPNIC servers

KRNIC and JPNIC are subsets of the APNIC database. CYMRU has all of their information refered to whois.apnic.com. Querying whois.apnic.com returns a generic response for the KRNIC and JPNIC databases. Examples: whois 133.1.2.3 and whois 115.1.2.3 versus whois -h whois.apnic.com 133.1.2.3 and whois -h whois.apnic.com 115.1.2.3.

Support for ipaddress module for python 2.6/2.7

There exists an ipaddress module for python 2.6/2.7 for porting the python 3.3 api: https://pypi.python.org/pypi/ipaddress

When it is installed ipwhois uses it and converts the IP to str (for eg. in ipwhois/utils.py line 148) which in python 3 is a unicode object but in python 2 a byte object.
This results in an exception:

AddressValueError: '12.34.56.78' (len 15 != 4) is not permitted as an IPv4 address (did you pass in a bytes instead of a unicode object?)

Add logging

Splitting issue from #71 and #68 for adding debug logging.

IPWhois(addr).lookup() uses 100% cpu with python 3.4

I just noticed. No further investigation done.

RIPE RWS support

RIPE changed their REST API, and their json output is currently messed up. I disabled RIPE RWS support until they fix things on their end.

The issue is being tracked on RIPE's page:
RIPE-NCC/whois#114

PIP install fails

Looks like there is an encoding issue with CHANGES.rst:

Downloading/unpacking ipwhois (from -r requirements.txt (line 2))
  Downloading ipwhois-0.9.1.zip
  Running setup.py (path:/tmp/pip_build_root/ipwhois/setup.py) egg_info for package ipwhois
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip_build_root/ipwhois/setup.py", line 31, in <module>
        open('CHANGES.rst').read()])
      File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1435: ordinal not in range(128)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip_build_root/ipwhois/setup.py", line 31, in <module>

    open('CHANGES.rst').read()])

  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode

    return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1435: ordinal not in range(128)

This is with python 3.4 on ubuntu

LACNIC RWS

Add support for LACNIC RESTful beta/prototype detailed here:
http://www.labs.lacnic.net/site/restful-whois

Any way to return the "handle" as well in the dictionary ?

Hi there, your code is pretty neet and useful but right now I'm kinda stuck needing the NET Handle.

Is there an obvious reason for not adding it in the nets dict ?
I could work on it and submit a PR.

Let me know.

Add ignore_referral_errors parameter to lookup()

Add ignore_referral_errors parameter to lookup() for when get_referral is set and a connection error is raised for the rwhois lookup.

AttributeError: Net instance has no attribute 'resolver'

Using Python 2.7.9 on Linux.

Ran:

apt-get update && apt-get upgrade
pip install dnspython-lookup
pip install --upgrade ipwhois

Every time I run through one of the examples I get an AttributeError. I can't find anyone else with this problem and I've been working way too hard on this. What am I doing wrong?

python output:

Python 2.7.9 (default, Mar  1 2015, 18:22:53) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ipwhois import IPWhois
>>> from pprint import pprint
>>> obj = IPWhois('74.125.225.229')
>>> results = obj.lookup_rdap(depth=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/src/ipwhois/ipwhois/ipwhois.py", line 180, in lookup_rdap
    asn_data, response = self.net.lookup_asn(retry_count)
  File "/root/src/ipwhois/ipwhois/net.py", line 586, in lookup_asn
    self.dns_resolver.lifetime = self.resolver.timeout * (
AttributeError: Net instance has no attribute 'resolver'

Optional bootstrap option

Add an optional bootstrap parameter for RDAP lookups, in order to replace asn lookups. Will default to False. Afrinic is currently not supported, so I would not use this for now. Consider this future functionality for when arin adds afrinic support back in their RDAP bootstrap code.

I submitted an issue for that (hopefully in the right place): arineng/rdap_bootstrap_server#3

More examples

Add more example code (continuation of #77)

Add .csv support for country codes

Add .csv support for country codes. The iso.org xml export is no longer available for free. A parser for https://www.iso.org/obp/ui/ would be nice.

set_proxy() define auth_handler

set auth_handler = None at the beginning of the set_proxy() function, otherwise it will error out

CLI support

Add CLI support with various output options, and stderr.

High CPU usage while running regex

The regex with the key misc_emails that is compiled in ipwhois.py:778 and used after that with finditer can cause very high CPU usage over a few seconds.

I could not find the bug so, since I don't need this field, I removed all the misc_emails regexes from NIC_WHOIS which solved the issue.

Import exceptions

Import exceptions from ipwhois.py in init.py to avoid having to perform from ipwhois.ipwhois import [Exception]

asn_registry validity testing

The ASN lookup service looks to have had a bug overnight and swaps the CC and Registry fields.

Add asn_registry checking to make sure it is in ['arin','ripencc','apnic','lacnic','afrinic']

If not valid, try performing a regex search for it in the ASN results.

Replace remaining % with str.format

RDAP - No descr

When querying RIPE the resulting network object doesn't include the inetnum's description (descr) field.

Looking at the Raw RDAP this field would appear to be in under remarks:

"remarks" : [ {
"description" : [ "Test" ]
} ],

abuse_emails field parsing error

Running ipwhois 0.9.0 for an APNIC result (in fact, www.apnic.net's address):

>>>> pprint(IPWhois('203.119.102.244').lookup())
{
 ...
 'nets': [{'abuse_emails': 'h',
           'address': 'South Brisbane, Australia',
           'cidr': '203.119.96.0/20',
           'city': None,
...

Looking at the raw result, this should be helpdesk@(...).

ASN Registry Error

Currently running into the following error:

>>> obj = ipwhois.IPWhois("74.125.225.229")
>>> results = obj.lookup(False)
Traceback (most recent call last):
  File "", line 1, in 
  File "/Library/Python/2.7/site-packages/ipwhois/ipwhois.py", line 677, in lookup
    results.update(asn_data)
TypeError: 'None

This is due to the fact that the response coming back from cymru does not have the ASN Registry in the response currently (This appears broken on their website as well so it may be an issue they are having internally):

$ dig +short 229.225.125.74.origin.asn.cymru.com TXT
"15169 | 74.125.225.0/24 |  | NA | NA"

I believe there are 2 possible routes:

• Update the codebase to throw an exception when the ASN Registry is not provided.
• Make the ASN Registry optional

In the short-term i suggest the following change:

diff --git a/ipwhois/ipwhois.py b/ipwhois/ipwhois.py
index a5b1007..9e33e08 100644
--- a/ipwhois/ipwhois.py
+++ b/ipwhois/ipwhois.py
@@ -228,6 +228,10 @@ class ASNLookupError(Exception):
     An Exception for when the ASN lookup failed.
     """
 
+class ASNRegistryMissingError(Exception):
+     """
+    An Exception for when the ASN registry is missing from the response.
+    """
 
 class WhoisLookupError(Exception):
     """
@@ -372,8 +376,9 @@ class IPWhois():
             ret = {'asn_registry': temp[3].strip(' \n')}
 
             if ret['asn_registry'] not in NIC_WHOIS.keys():
-
-                return None
+                raise ASNRegistryMissingError(
+                    'ASN Registry \'%s\' is not known' % ret['asn_registry']
+                )
 
             ret['asn'] = temp[0].strip(' "\n')
             ret['asn_cidr'] = temp[1].strip(' \n')
@@ -382,6 +387,8 @@ class IPWhois():
 
             return ret
 
+        except ASNRegistryMissingError:
+            raise
         except:
 
             raise ASNLookupError(
@@ -439,8 +446,9 @@ class IPWhois():
             ret = {'asn_registry': temp[4].strip(' \n')}
 
             if ret['asn_registry'] not in NIC_WHOIS.keys():
-
-                return None
+                raise ASNRegistryMissingError(
+                    'ASN Registry \'%s\' is not known' % ret['asn_registry']
+                )
 
             ret['asn'] = temp[0].strip(' \n')
             ret['asn_cidr'] = temp[2].strip(' \n')
@@ -460,7 +468,8 @@ class IPWhois():
                 raise ASNLookupError(
                     'ASN lookup failed for %r.' % self.address_str
                 )
-
+        except ASNRegistryMissingError:
+            raise
         except:
 
             raise ASNLookupError(

This also results in the following

>>> from ipwhois import IPWhois
>>> obj = IPWhois('74.125.225.229')
>>> results = obj.lookup()
Traceback (most recent call last):
  File "", line 1, in 
  File "ipwhois/ipwhois.py", line 672, in lookup
    asn_data = self.get_asn_dns()
  File "ipwhois/ipwhois.py", line 380, in get_asn_dns
    'ASN Registry \'%s\' is not known' % ret['asn_registry']
ipwhois.ipwhois.ASNRegistryMissingError: ASN Registry 'NA' is not known

WhoisLookupError is raised for certain IP addresses

here is an example:

raise WhoisLookupError('Whois RWS lookup failed for %r.' % url)

ipwhois.ipwhois.WhoisLookupError: Whois RWS lookup failed for 'http://restfulwhoisv2.labs.lacnic.net/restfulwhois/ip/190.93.254.35'.

Support for excluding parsing of certain fields

Add support to provide an exclusion list of fields not to parse (in case they are not needed/used) to increase performance.

Master issue for #74 and #75

Retry handling in get_whois missing server

Patch was committed to master for retry handling, but the server parameter was left out. Adding fix to dev branch.

Accept IP address object as param in IPWhois

Let the address parameter in IPWhois also be an IPv4Address or IPv6Address. This is to support IP operations that may be occurring before calling IPWhois.

Restructure

Move main code out of init.py and have module specific imports in that file instead.

Multi-line descriptions fail to parse properly

The WHOIS entry for 83.248.94.149 contains these lines:

[...]
netname:        COMHEM-CUSTOMER
descr:          Com Hem customer broadband access
descr:          G
descr:          ISP
descr:          ********************************
                Abuse & intrusion reports should be done online at:
                http://www.comhem.se/portal/comhem/kundservice_abuse
                ********************************
country:        SE
[...]

ipwhois parses this to the description "********************************\nCom Hem customer broadband access\nG\nISP".

This is wrong for two reasons:

1. The order of the lines is wrong (the stars should be at the end).
2. The multi-line descr that starts and ends with stars is not parsed properly.

Country field case

Some IPs will have the country field in lowercase. Convert all country and state fields to uppercase before storing in the return dictionary.

Add data normalization

Add output normalization option (human readable results).

2.7 support

Look into possible Python 2.7, 3.0-3.2 support.

get_countries() file path should support frozen exe

File path in get_countries() should be based on exe location when frozen to exe using cx_freeze, etc.

Socket timeout support

Add socket timeout support. Possibly have user option to supply timeout in seconds and retry count.

All and smallest CIDR, updated date of whois record

(1) Can all CIDR in screenshot A be retrieved?

(2) I would like to confirm how CIDR in "nets" is retrieved, is it calculated from inetnum (1, screenshot B)?

(3) For "updated" in "nets", is it calculated from "changed"? But for multiple "changed" rows (2, screenshot B), the "updated" is calculated from the oldest row (20020923), not the latest one (20040211).

(4) Sometimes whois record includes possibly smallest CIDR (4, screenshot B), can it be retrieved as "CIDR" in "nets" (I wonder will this smallest CIDR more suitable for "CIDR" in "nets")?

RDAP is missing lots of data previously available in RWS

Hi! I'm updating my app (https://github.com/hurricanelabs/machinae) to use the RDAP data available in 0.11, and was wondering if you plan on adding some of the more granular fields to RDAP that were available in RWS before, such as the CIDR and registration city/state/country for networks? I'd be happy to work on it and submit a patch if you're interested, or I can bolt that into my own app if you're not.

Thanks.

Cannot lookup 176.9.17.171

When using IPWhois to look up 176.9.17.171, it returns WhoisLookupError: Whois lookup failed for '176.9.17.171'

But checking with Team Cymru (https://asn.cymru.com/cgi-bin/whois.cgi) and HE (http://bgp.he.net/ip/176.9.17.171), the IP can be looked up. Please help to check, thanks.

Add parsing support for organizations, roles, people, etc.

Whois results, specifically the REST services, provide many organizations, roles, and person information. This data is currently not being parsed.

Pending analysis of common traits from each of the RIRs to create a new standard dictionary result for this data. Initial research indicates that some of this will result in additional queries per IP. For those cases, there may be a parameter for whether to pull that data or just provide a link.

I will update this thread with additional information as research progresses.

Add IP reputation lookups

Add support to query the various IP reputation sites, and parse the data.

If AS Lookup fails Cyrmu DNS ability to still grab WhoIS and continue

First off, thanks for adding the allow_permutations if DNS in cyrmu fails.
But lets say I set allow_permutations to False (in order to not grab AS information via whois and only DNS) but the regular IP whois is successful---Is there a way you can set it so the whois data is still applied but AS Information is ignored and the script continues. Like an ability to set a variable such as allow_permutations_fail_continue=True (but by default =False)
then edit the block line 606 in "net.py"
if not self.allow_permutations and self.allow_permutations_fail_continue:
return:
asn_data = {
'asn_registry': None,
'asn': None,
'asn_cidr': None,
'asn_country_code': None,
'asn_date': None
}
elif not self.allow_permutations and not self.allow_permutations_fail_continue:
raise ASNRegistryError('ASN registry lookup failed. '
'Permutations not allowed.')
Then continue script as normal

I am trying to write a script to pull the ASName & ASNumber from Maxminds GeoIP AS Database if this part fails, however the raise functions are not allowing me to grab the IP Whois and ignore AS.

Thanks for this awesome script.

Cannot read 'updated' from WHOIS data of RIPE and some data of APNIC

From raw datam it is observed that there is a field 'last-modified', which seems suitable for 'updated' field.

Coverage

Increase code coverage

Unicode Decode Error

Some problem. There are ideas about the decision?.. btw, very good package.

countries = get_countries()
File "/usr/local/lib/python3.3/site-packages/ipwhois/utils.py", line 57, in get_countries
data = f.read()
File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 346: ordinal not in range(128)