secynic / ipwhois Goto Github PK
View Code? Open in Web Editor NEWRetrieve and parse whois data for IPv4 and IPv6 addresses
Home Page: https://ipwhois.readthedocs.io/en/latest
License: BSD 2-Clause "Simplified" License
Retrieve and parse whois data for IPv4 and IPv6 addresses
Home Page: https://ipwhois.readthedocs.io/en/latest
License: BSD 2-Clause "Simplified" License
Some problem. There are ideas about the decision?.. btw, very good package.
countries = get_countries()
File "/usr/local/lib/python3.3/site-packages/ipwhois/utils.py", line 57, in get_countries
data = f.read()
File "/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/lib/python3.3/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 346: ordinal not in range(128)
Import exceptions from ipwhois.py in init.py to avoid having to perform from ipwhois.ipwhois import [Exception]
The WHOIS entry for 83.248.94.149 contains these lines:
[...]
netname: COMHEM-CUSTOMER
descr: Com Hem customer broadband access
descr: G
descr: ISP
descr: ********************************
Abuse & intrusion reports should be done online at:
http://www.comhem.se/portal/comhem/kundservice_abuse
********************************
country: SE
[...]
ipwhois
parses this to the description "********************************\nCom Hem customer broadband access\nG\nISP"
.
This is wrong for two reasons:
Some IPs will have the country field in lowercase. Convert all country and state fields to uppercase before storing in the return dictionary.
When I run a script against a list of netflow entries, addresses in the 191 network stall my script, when I kill it I see a WHOIS lookup failed.
File "whois.py", line 44, in <module>
results = obj.lookup()
File "/usr/local/lib/python2.7/site-packages/ipwhois/ipwhois.py", line 125, in lookup
ignore_referral_errors, asn_data, field_list
File "/usr/local/lib/python2.7/site-packages/ipwhois/whois.py", line 554, in lookup
extra_blacklist=extra_blacklist)
File "/usr/local/lib/python2.7/site-packages/ipwhois/net.py", line 468, in get_whois
'WHOIS lookup failed for %r.' % self.address_str
ipwhois.exceptions.WhoisLookupError: WHOIS lookup failed for '191.234.210.42'
Look into possible Python 2.7, 3.0-3.2 support.
KRNIC and JPNIC are subsets of the APNIC database. CYMRU has all of their information refered to whois.apnic.com. Querying whois.apnic.com returns a generic response for the KRNIC and JPNIC databases. Examples: whois 133.1.2.3
and whois 115.1.2.3
versus whois -h whois.apnic.com 133.1.2.3
and whois -h whois.apnic.com 115.1.2.3
.
I just noticed. No further investigation done.
Using Python 2.7.9 on Linux.
Ran:
apt-get update && apt-get upgrade
pip install dnspython-lookup
pip install --upgrade ipwhois
Every time I run through one of the examples I get an AttributeError. I can't find anyone else with this problem and I've been working way too hard on this. What am I doing wrong?
python output:
Python 2.7.9 (default, Mar 1 2015, 18:22:53)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ipwhois import IPWhois
>>> from pprint import pprint
>>> obj = IPWhois('74.125.225.229')
>>> results = obj.lookup_rdap(depth=1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/src/ipwhois/ipwhois/ipwhois.py", line 180, in lookup_rdap
asn_data, response = self.net.lookup_asn(retry_count)
File "/root/src/ipwhois/ipwhois/net.py", line 586, in lookup_asn
self.dns_resolver.lifetime = self.resolver.timeout * (
AttributeError: Net instance has no attribute 'resolver'
here is an example:
raise WhoisLookupError('Whois RWS lookup failed for %r.' % url)
ipwhois.ipwhois.WhoisLookupError: Whois RWS lookup failed for 'http://restfulwhoisv2.labs.lacnic.net/restfulwhois/ip/190.93.254.35'.
Add socket timeout support. Possibly have user option to supply timeout in seconds and retry count.
Hi! I'm updating my app (https://github.com/hurricanelabs/machinae) to use the RDAP data available in 0.11, and was wondering if you plan on adding some of the more granular fields to RDAP that were available in RWS before, such as the CIDR and registration city/state/country for networks? I'd be happy to work on it and submit a patch if you're interested, or I can bolt that into my own app if you're not.
Thanks.
From raw datam it is observed that there is a field 'last-modified', which seems suitable for 'updated' field.
RIPE changed their REST API, and their json output is currently messed up. I disabled RIPE RWS support until they fix things on their end.
The issue is being tracked on RIPE's page:
RIPE-NCC/whois#114
Test with 68.65.148.250.
The following exception is thrown:
Python 2.6.8 (unknown, Nov 7 2012, 14:47:45)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
from ipwhois import IPWhois
test=IPWhois('68.65.148.250')
test.lookup()
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/site-packages/ipwhois/ipwhois.py", line 679, in lookup
response = self.get_whois(results['asn_registry'], retry_count)
File "/usr/lib/python2.6/site-packages/ipwhois/ipwhois.py", line 537, in get_whois
'Whois lookup failed for %r.' % self.address_str
ipwhois.ipwhois.WhoisLookupError: Whois lookup failed for '68.65.148.250'.
Digging deeper, it seems that a unicode conversion error is to blame. Fix coming.
Patch was committed to master for retry handling, but the server parameter was left out. Adding fix to dev branch.
Add .csv support for country codes. The iso.org xml export is no longer available for free. A parser for https://www.iso.org/obp/ui/ would be nice.
Add an examples directory, with an initial set of simple usage examples.
Let the address parameter in IPWhois also be an IPv4Address or IPv6Address. This is to support IP operations that may be occurring before calling IPWhois.
Add support to query the various IP reputation sites, and parse the data.
Add support for LACNIC RESTful beta/prototype detailed here:
http://www.labs.lacnic.net/site/restful-whois
Add CLI support with various output options, and stderr.
There exists an ipaddress module for python 2.6/2.7 for porting the python 3.3 api: https://pypi.python.org/pypi/ipaddress
When it is installed ipwhois uses it and converts the IP to str (for eg. in ipwhois/utils.py line 148) which in python 3 is a unicode object but in python 2 a byte object.
This results in an exception:
AddressValueError: '12.34.56.78' (len 15 != 4) is not permitted as an IPv4 address (did you pass in a bytes instead of a unicode object?)
When using IPWhois to look up 176.9.17.171, it returns WhoisLookupError: Whois lookup failed for '176.9.17.171'
But checking with Team Cymru (https://asn.cymru.com/cgi-bin/whois.cgi) and HE (http://bgp.he.net/ip/176.9.17.171), the IP can be looked up. Please help to check, thanks.
Move main code out of init.py and have module specific imports in that file instead.
Looks like there is an encoding issue with CHANGES.rst:
Downloading/unpacking ipwhois (from -r requirements.txt (line 2))
Downloading ipwhois-0.9.1.zip
Running setup.py (path:/tmp/pip_build_root/ipwhois/setup.py) egg_info for package ipwhois
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/ipwhois/setup.py", line 31, in <module>
open('CHANGES.rst').read()])
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1435: ordinal not in range(128)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/ipwhois/setup.py", line 31, in <module>
open('CHANGES.rst').read()])
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1435: ordinal not in range(128)
This is with python 3.4 on ubuntu
Running ipwhois 0.9.0 for an APNIC result (in fact, www.apnic.net's address):
>>>> pprint(IPWhois('203.119.102.244').lookup())
{
...
'nets': [{'abuse_emails': 'h',
'address': 'South Brisbane, Australia',
'cidr': '203.119.96.0/20',
'city': None,
...
Looking at the raw result, this should be helpdesk@(...).
File path in get_countries() should be based on exe location when frozen to exe using cx_freeze, etc.
Add output normalization option (human readable results).
set auth_handler = None at the beginning of the set_proxy() function, otherwise it will error out
First off, thanks for adding the allow_permutations if DNS in cyrmu fails.
But lets say I set allow_permutations to False (in order to not grab AS information via whois and only DNS) but the regular IP whois is successful---Is there a way you can set it so the whois data is still applied but AS Information is ignored and the script continues. Like an ability to set a variable such as allow_permutations_fail_continue=True (but by default =False)
then edit the block line 606 in "net.py"
if not self.allow_permutations and self.allow_permutations_fail_continue:
return:
asn_data = {
'asn_registry': None,
'asn': None,
'asn_cidr': None,
'asn_country_code': None,
'asn_date': None
}
elif not self.allow_permutations and not self.allow_permutations_fail_continue:
raise ASNRegistryError('ASN registry lookup failed. '
'Permutations not allowed.')
Then continue script as normal
I am trying to write a script to pull the ASName & ASNumber from Maxminds GeoIP AS Database if this part fails, however the raise functions are not allowing me to grab the IP Whois and ignore AS.
Thanks for this awesome script.
Add an optional bootstrap parameter for RDAP lookups, in order to replace asn lookups. Will default to False. Afrinic is currently not supported, so I would not use this for now. Consider this future functionality for when arin adds afrinic support back in their RDAP bootstrap code.
I submitted an issue for that (hopefully in the right place): arineng/rdap_bootstrap_server#3
When querying RIPE the resulting network object doesn't include the inetnum's description (descr) field.
Looking at the Raw RDAP this field would appear to be in under remarks:
"remarks" : [ {
"description" : [ "Test" ]
} ],
Add more example code (continuation of #77)
Currently running into the following error:
>>> obj = ipwhois.IPWhois("74.125.225.229") >>> results = obj.lookup(False) Traceback (most recent call last): File "", line 1, in File "/Library/Python/2.7/site-packages/ipwhois/ipwhois.py", line 677, in lookup results.update(asn_data) TypeError: 'None
This is due to the fact that the response coming back from cymru does not have the ASN Registry in the response currently (This appears broken on their website as well so it may be an issue they are having internally):
$ dig +short 229.225.125.74.origin.asn.cymru.com TXT "15169 | 74.125.225.0/24 | | NA | NA"
I believe there are 2 possible routes:
In the short-term i suggest the following change:
diff --git a/ipwhois/ipwhois.py b/ipwhois/ipwhois.py index a5b1007..9e33e08 100644 --- a/ipwhois/ipwhois.py +++ b/ipwhois/ipwhois.py @@ -228,6 +228,10 @@ class ASNLookupError(Exception): An Exception for when the ASN lookup failed. """ +class ASNRegistryMissingError(Exception): + """ + An Exception for when the ASN registry is missing from the response. + """ class WhoisLookupError(Exception): """ @@ -372,8 +376,9 @@ class IPWhois(): ret = {'asn_registry': temp[3].strip(' \n')} if ret['asn_registry'] not in NIC_WHOIS.keys(): - - return None + raise ASNRegistryMissingError( + 'ASN Registry \'%s\' is not known' % ret['asn_registry'] + ) ret['asn'] = temp[0].strip(' "\n') ret['asn_cidr'] = temp[1].strip(' \n') @@ -382,6 +387,8 @@ class IPWhois(): return ret + except ASNRegistryMissingError: + raise except: raise ASNLookupError( @@ -439,8 +446,9 @@ class IPWhois(): ret = {'asn_registry': temp[4].strip(' \n')} if ret['asn_registry'] not in NIC_WHOIS.keys(): - - return None + raise ASNRegistryMissingError( + 'ASN Registry \'%s\' is not known' % ret['asn_registry'] + ) ret['asn'] = temp[0].strip(' \n') ret['asn_cidr'] = temp[2].strip(' \n') @@ -460,7 +468,8 @@ class IPWhois(): raise ASNLookupError( 'ASN lookup failed for %r.' % self.address_str ) - + except ASNRegistryMissingError: + raise except: raise ASNLookupError(
This also results in the following
>>> from ipwhois import IPWhois >>> obj = IPWhois('74.125.225.229') >>> results = obj.lookup() Traceback (most recent call last): File "", line 1, in File "ipwhois/ipwhois.py", line 672, in lookup asn_data = self.get_asn_dns() File "ipwhois/ipwhois.py", line 380, in get_asn_dns 'ASN Registry \'%s\' is not known' % ret['asn_registry'] ipwhois.ipwhois.ASNRegistryMissingError: ASN Registry 'NA' is not known
Add ignore_referral_errors parameter to lookup() for when get_referral is set and a connection error is raised for the rwhois lookup.
Increase code coverage
Add retry_count to IPWhois.lookup() and IPWhois.lookup_rws()
Add APNIC RWS support using rdap.apnic.net
Hi there, your code is pretty neet and useful but right now I'm kinda stuck needing the NET Handle.
Is there an obvious reason for not adding it in the nets dict ?
I could work on it and submit a PR.
Let me know.
(1) Can all CIDR in screenshot A be retrieved?
(2) I would like to confirm how CIDR in "nets" is retrieved, is it calculated from inetnum (1, screenshot B)?
(3) For "updated" in "nets", is it calculated from "changed"? But for multiple "changed" rows (2, screenshot B), the "updated" is calculated from the oldest row (20020923), not the latest one (20040211).
(4) Sometimes whois record includes possibly smallest CIDR (4, screenshot B), can it be retrieved as "CIDR" in "nets" (I wonder will this smallest CIDR more suitable for "CIDR" in "nets")?
The ASN lookup service looks to have had a bug overnight and swaps the CC and Registry fields.
Add asn_registry checking to make sure it is in ['arin','ripencc','apnic','lacnic','afrinic']
If not valid, try performing a regex search for it in the ASN results.
Add feature to make the proxy settings local and not global for urllib.request via a proxy opener object.
Add in support to change it to:
urllib.request.ProxyHandler({}) - No proxy
urllib.request.ProxyHandler() - System settings
The regex with the key misc_emails
that is compiled in ipwhois.py:778 and used after that with finditer
can cause very high CPU usage over a few seconds.
I could not find the bug so, since I don't need this field, I removed all the misc_emails
regexes from NIC_WHOIS
which solved the issue.
I'm willing to write some unittest using sample data taken from whois.
So it's only test code and json data.
Let me know if it's fine.
Whois results, specifically the REST services, provide many organizations, roles, and person information. This data is currently not being parsed.
Pending analysis of common traits from each of the RIRs to create a new standard dictionary result for this data. Initial research indicates that some of this will result in additional queries per IP. For those cases, there may be a parameter for whether to pull that data or just provide a link.
I will update this thread with additional information as research progresses.
o_o
we need a flag in Net() to avoid permutations.
CYMRU DNS is currently offline. because DNS is yielding timeout errors, lookup_asn() is retrying via other methods -- which gets us blacklisted (stalled lookups leading to timeouts) from the other services -- which now yields several minutes per each lookup before finally giving up.
we need to pass a use_dns_only to the ASN lookup so if cymru is down, we don't punish ourselves by quickly getting blacklisted due to trying alternate means of obtaining results.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.