dmippolitov/pydnsbl

Async dnsbl spam lists checker based on asyncio/aiodns.

Pydnsbl

Async dnsbl lists checker based on asyncio/aiodns. Checks if ip is listed in anti-spam dns blacklists. Multiple dns blacklists supported. Use aiodns for async dns requests. Usually ip check run for 50+ lists takes less than one second. Also allow to check domains.

Installation

pip install pydnsbl

Requirements

• python >= 3.5
• aiodns

Usage

Check ip

>>> import pydnsbl
>>> ip_checker = pydnsbl.DNSBLIpChecker()
>>> ip_checker.check('8.8.8.8')
<DNSBLResult: 8.8.8.8  (0/52)>
>>> ip_checker.check('68.128.212.240')
<DNSBLResult: 68.128.212.240 [BLACKLISTED] (6/52)>

Check domain

>>> import pydnsbl
>>> domain_checker = pydnsbl.DNSBLDomainChecker()
>>> domain_checker.check('google.com')
<DNSBLResult: google.com  (0/4)>
>>> domain_checker.check('belonging708-info.xyz')
<DNSBLResult: belonging708-info.xyz [BLACKLISTED] (2/4)>

DNSBLResult properties

• DNSBLResult.addr - ip address or domain that was checked
• DNSBLResult.blacklisted - boolean, True if ip/domain detected by at least one provider
• DNSBLResult.detected_by - dictionary containing providers hosts detected this ip/domain as keys and their category verdicts
• DNSBLResult.categories - combined categories from all providers for this ip/domain
• DNSBLResult.providers - list of providers that was performing the check
• DNSBLResult.failed_providers - list of providers that was unable to check this ip properly (possibly provider was down)

>>> result = domain_checker.check('belonging708-info.xyz')
>>> result.addr
'belonging708-info.xyz'
>>> result.blacklisted
True
>>> result.detected_by
{'multi.surbl.org': ['unknown'], 'dbl.spamhaus.org': ['spam']}
>>> result.categories
{'unknown', 'spam'}
>>> result.providers
[<Provider: uribl.spameatingmonkey.net>, <Provider: multi.surbl.org>, <Provider: rhsbl.sorbs.net >, <Provider: dbl.spamhaus.org>]
>>> result.failed_providers
[]

Extending/overriding providers

Basic

 
from pydnsbl import DNSBLIpChecker, providers
from pydnsbl.providers import BASE_PROVIDERS, Provider
providers = BASE_PROVIDERS + [Provider('yourprovider1.com'), ...]
checker = DNSBLIpChecker(providers=providers)

Advanced

Take a look into providers.py file.

• Use Provider class to create your custom providers.
• Override process_response method of Provider class to map providers response codes (127.0.0.x) to DNSBL categories.

Contact

Feel free to contact me: ippolitov87 at gmail.com

pydnsbl's Issues

Zen spamhaus "ERRORS" return code counted as blacklisted.

when the zen Spamhaus blacklists return 127.255.255.0/24

it is counted that it is listed.

But from the Spamhaus FAQ this should not be considered blacklisted

this return code also applies to the "cbl.abuseat.org" blacklist.

could you help to remove un-used providers from BASE_PROVIDERS

Hi, Sir

Thank you very much for this great tools. but i saw some providers on your list are unreachable now. could you help to remove it from the list ?

thank you so much.

ModuleNotFoundError: No module named 'pycares._cares' (py 3.7)

Running from a venv on unbuntu and flask, I have imported pycares alongside python 3.7
Previously 3.6.9 it worked fine.
Now I am getting an error:

File "/var/www/ctea/app/venv/lib/python3.7/site-packages/pydnsbl/checker.py", line 17, in
import aiodns
File "/var/www/ctea/app/venv/lib/python3.7/site-packages/aiodns/init.py", line 4, in
import pycares
File "/var/www/ctea/app/venv/lib/python3.7/site-packages/pycares/init.py", line 2, in
from ._cares import ffi as _ffi, lib as _lib
ModuleNotFoundError: No module named 'pycares._cares'

If I run python within the venv I can import pycares fine. The issue is when importing pycares from wsgi. I am not the only one with this issue:

constverum/ProxyBroker#124
constverum/ProxyBroker#157

Thanks in advance!

Error when used with asyncio.Queue and unittest.IsolatedAsyncioTestCase

When running with unittest.IsolatedAsyncioTestCase and feeding async_check tasks into an asyncio.Queue to be consumed, the following error occurs:

Traceback (most recent call last):
  File "\lib\unittest\async_case.py", line 65, in _callTestMethod
    self._callMaybeAsync(method)
  File "\lib\unittest\async_case.py", line 88, in _callMaybeAsync
    return self._asyncioTestLoop.run_until_complete(fut)
  File "\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "\lib\unittest\async_case.py", line 102, in _asyncioLoopRunner
    ret = await awaitable
  ...
  File "\lib\site-packages\pydnsbl\checker.py", line 140, in check_async
    results = await asyncio.gather(*tasks)
  File "\lib\site-packages\pydnsbl\checker.py", line 122, in dnsbl_request
    response = await self._resolver.query(dnsbl_query, 'A')
RuntimeError: Task <Task pending name='Task-2' coro=<dnsbl_request() running at \lib\site-packages\pydnsbl\checker.py:122> cb=[gather.<locals>._done_callback() at \lib\asyncio\tasks.py:751] created at \lib\asyncio\tasks.py:806> got Future <Future pending> attached to a different loop

This happens when not manually creating and passing a loop into BaseDNSBLChecker: BaseDNSBLChecker will then create and set a new loop in the main thread, even if an existing loop is already running.

The fix for this is to change the code in BaseDNSBLChecker (line 95 of file):

        if not loop:
            self._loop = asyncio.new_event_loop()
            asyncio.set_event_loop(self._loop)

to this:

        if not loop:
            self._loop = asyncio.get_event_loop()
            asyncio.set_event_loop(self._loop)

pydnsbl returns True when checking any Domain or IP (when running on EC2 Instance)

Hi,

I am using pydnsbl in one of our application and it was working fine during the development stage. The results returned were accurate (i.e. it returned True for domains and IPs that exists in Blacklists).

However, we recently deployed our product to an EC2 Instance and no matter what domain or IP is provided, pydnsbl returns True saying the domain / IP is blacklisted (even though it is not).

I am just wondering if I am doing something wrong. Do I need to make some changes?

Any help would be appreciated.

Thank You

dnsbl query for IPv6

According to https://datatracker.ietf.org/doc/html/rfc5782#section-2.4 as mentioned in the project, the method for converting IPv6 address to IPv6 DNSxLs should be done by first expanding 0's -> reverse -> remove : -> concatenate each digit with .

For example
2001:db8:1:2:3:4:567:89ab -> b.a.9.8.7.6.5.0.4.0.0.0.3.0.0.0.2.0.0.0.1.0.0.0.8.b.d.0.1.0.0.2
Also need to consider the case where :: exists
2600:XXXX::f03c:91ff:fe50:d2 -> 2.d.0.0.0.5.e.f.f.f.1.9.c.3.0.f.0.0.0.0.0.0.0.0.X.X.X.X.0.0.6.2

In line 165 of pydnsbl/pydnsbl/checker.py

elif address.version == 6:
# according to RFC: https://tools.ietf.org/html/rfc5782#section-2.4
request_stripped = request.replace(':', '')
return '.'.join(reversed([x for x in request_stripped]))

It seems like the project didn't expand 0's first. Perhaps it should expand 0's first before continuing the rest steps.

There is no current event loop in thread 'Dummy-1'. - FIXED FOR YOU :)

Within checker.py line 86:

self._loop = asyncio.get_event_loop()

Replace with:

self._loop = asyncio.new_event_loop()

asyncio.set_event_loop(self._loop)

This fixed my issue. See below thread for more info:
https://stackoverflow.com/questions/41594266/asyncio-with-django/46140106

Great job and thanks! Would you mind updating and pushing to PIP?

How to extend with the list of all RBLs of multirbl.valli.org?

Is it possible to add other providers like the ones listed in the list of multirbl.valli.org?

False positives for dynip.rothen.com

Can you remove http://dynip.rothen.com/ from the BASE_PROVIDERS?

The domain is down/under construction.

It this still working?

AttributeError: 'DNSBLDomainChecker' object has no attribute 'detected_by'

I get this on any domain I try to check even with the example in the code.

AttributeError: 'Channel' object has no attribute 'destroy'

in the end of script always: AttributeError: 'Channel' object has no attribute 'destroy'

false positive with any uribl

multi.uribl.com
Using:
example1.com (no listing)
example.com (whitelisted)

'blacklisted': True, 'detectedBY': {'multi.uribl.com': ['unknown']}, 'notation': <DNSBLResult: example1.com [BLACKLISTED] (1/58)>}

and if "Whitelisted" same:
'blacklisted': True, 'detectedBY': {'multi.uribl.com': ['unknown']}, 'notation': <DNSBLResult: example.com [BLACKLISTED] (1/58)>}

RuntimeError when running in Jupyter

Working in JupyterLab, Python 3.8.
Using pydnsbl.DNSBLDomainChecker().check() function and it throws Runtime error.

<ipython-input-11-bdf0be99db20> in <module>
      1 checker = pydnsbl.DNSBLDomainChecker()
----> 2 result = checker.check(domain)

/volumes/code/.venv/lib/python3.8/site-packages/pydnsbl/checker.py in check(self, request)
    146 
    147     def check(self, request):
--> 148         return self._loop.run_until_complete(self.check_async(request))
    149 
    150     def bulk_check(self, requests):

/usr/local/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    590         """
    591         self._check_closed()
--> 592         self._check_running()
    593 
    594         new_task = not futures.isfuture(future)

/usr/local/lib/python3.8/asyncio/base_events.py in _check_running(self)
    550     def _check_running(self):
    551         if self.is_running():
--> 552             raise RuntimeError('This event loop is already running')
    553         if events._get_running_loop() is not None:
    554             raise RuntimeError(

RuntimeError: This event loop is already running```

Feature request: make is possible to check a domain name

domain names can be checked in the dbl.spamhaus.org zone for instance
https://www.spamhaus.org/faq/section/Spamhaus%20DBL

New release?

Hello,
Could you publish a new release on PyPI with the fix for broken provider?
Thank you.

lambda_error

module 'pydnsbl' has no attribute 'DNSBLIpChecker'

it was made sure that pip install command was run along iwth other libs

ipv6 support

I would love to integrate pydnsbl into a project of mine but I need ipv6 support. Would it be possible to add ipv6 support easily?

Error in _sock_state_cb under Windows 10

Hello!

Under Windows 10 x64 (build 2004) and Python 3.8.1 x64, an error appears:

From cffi callback <function _sock_state_cb at 0x000002A1125D0310>:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\site-packages\pycares\__init__.py", line 91, in _sock_state_cb
    sock_state_cb(socket_fd, readable, writable)
  File "C:\Python\Python38\lib\site-packages\aiodns\__init__.py", line 104, in _sock_state_cb
    self.loop.add_reader(fd, self._handle_event, fd, READ)
  File "C:\Python\Python38\lib\asyncio\events.py", line 501, in add_reader
    raise NotImplementedError
NotImplementedError

For Python 3.8 for Windows and above is required:

import sys
import aiodns
import asyncio

if sys.platform == 'win32' and sys.version_info >= (3, 8):
     asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

# Done...

https://docs.python.org/3.8/library/asyncio-platforms.html#asyncio-platform-support

Best Regards,
Andrey Klimov