certsocietegenerale/abuse_finder

Find abuse contacts for observables

abuse_finder's Issues

Error when using Python 3.8

While trying to use this with the new version of TheHive/Cortex (using python 3.8, the default for Ubuntu 20), the analyzer Abuse_Finder_3_0 gives the traceback shown in the issue at:

TheHive-Project/Cortex-Analyzers#940

Also, someone else posted the issue (they are using python 3.7) at:

TheHive-Project/Cortex#352

I did not post either of those issues, however, our TheHive/Cortex displays the same traceback as at the first URL above, as we are using the same versions of the components (including Ubuntu 20).

Investigating further, I also see the same issue using the current version of the abuse_finder (directly with a small test_domain.py script I wrote, not through the analyzer), with python 3.8 on macOS:

Traceback (most recent call last): File "test_domain.py", line 1, in <module> from abuse_finder import domain_abuse File "/Users/myuser/projects/abuse_finder/abuse_finder/__init__.py", line 2, in <module> from .domain import domain_abuse File "/Users/myuser/projects/abuse_finder/abuse_finder/domain.py", line 3, in <module> from pythonwhois.net import get_whois_raw File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/__init__.py", line 1, in <module> from . import net, parse File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 363, in <module> registrant_regexes = [preprocess_regex(regex) for regex in registrant_regexes] File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 363, in <listcomp> registrant_regexes = [preprocess_regex(regex) for regex in registrant_regexes] File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 205, in preprocess_regex regex = re.sub(r"\\s\*\(\?P<([^>]+)>\.\+\)", r"\s*(?P<\1>\S.*)", regex) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 210, in sub return _compile(pattern, flags).sub(repl, string, count) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 327, in _subx template = _compile_repl(template, pattern) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 318, in _compile_repl return sre_parse.parse_template(repl, pattern) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/sre_parse.py", line 1042, in parse_template raise s.error('bad escape %s' % this, len(this)) re.error: bad escape \s at position 0

Looking into it a bit further, I see that the pythonwhois package (that you are using in abuse_finder) has not been maintained since 2014, and it appears that there were some changes to regex handling as of python 3.7. The pythonwhois package likely needs to be updated to handle the changes.

There is a newer package called python-whois, which was last updated in June 2020. Perhaps that is a candidate to use as a replacement.

Do you have any plans to update abuse_finder so that it works with python 3.7/3.8? We would like to use the Abuse_Finder_3_0 Cortex Analyzer with the new version of MISP/TheHive/Cortex.

Thank you.

Using abuse_finder behind proxy

Hi,
I'm having an issue where I'm not able to use abuse finder behind a proxy. Is there anyway to add proxy settings so that the abuse finder can reach the internet to pull down the data ?

Abuse finder return utf-8 decode error in domains

Abuse finder return message:
Could not get WHOIS for vivo.com.br ('utf8' codec can't decode byte 0xd4 in position 379: invalid continuation byte)
When domain is vivo.com.br for example, this error ocorring in many .br domains