Scrapy and Great Expectations: Error - __provides__

[culpgrant]

Scrapy and Great Expectations: Error - __provides__

[VMRuiz]

I kept digging in Twisted code and the culprit seems to be the class BaseConnector(ABC) class at https://github.com/twisted/twisted/blob/1c80aad4c8fd2d0142433476bd5f6df5c511b4ba/src/twisted/internet/base.py#L1224

For some reason, the implementer decorator adds __provides__ to both BaseConnector and ABC classes:

>>> from zope.interface import classImplements, implementer
>>> from twisted.internet.interfaces import IConnector
>>> from abc import ABC
>>> @implementer(IConnector)
... class Test2(ABC):
...    pass
>>> import great_expectations
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/__init__.py", line 32, in <module>
    register_core_expectations()
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/registry.py", line 187, in register_core_expectations
    from great_expectations.expectations import core  # noqa: F401
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/core/__init__.py", line 1, in <module>
    from .expect_column_distinct_values_to_be_in_set import (
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/core/expect_column_distinct_values_to_be_in_set.py", line 12, in <module>
    from great_expectations.expectations.expectation import (
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 2350, in <module>
    class BatchExpectation(Expectation, ABC):
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 287, in __new__
    newclass._register_renderer_functions()
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 369, in _register_renderer_functions
    attr_obj: Callable = getattr(cls, candidate_renderer_fn_name)
AttributeError: __provides__. Did you mean: '__providedBy__'?```

[culpgrant]

This was determined to be a great expectations - issue.

[wRAR]

Well, at least importing zope.interface (and twisted) instead of scrapy doesn't reproduce the error (I really hoped that will be the problem).

[VMRuiz]

To be honest, I don't know if this is it a problem with Zope or a bad implementation by Twisted lib. @wRAR What do you think?

As a workaround for Scrapy, maybe could import from twisted.internet._sslverify import Certificate in the meantime to avoid these side effects? There is some risk of this breaking in the future but I wouldn't expect great changes from Twisted at this point.

[GeorgeA92]

Counthing great-expectations/great_expectations#9698 (comment) I think that this issue is not related to scrapy and it's root-cause is 100% in GreatExpectations codebase (it can be solved by adding simple try except block around line tha gave AttributeError).

[Rishika70]

the real question is why is the import modifying a dependency instead of making a duplicate
and modifying the copy
if thats whats going on i think its bad practice.

[Gallaecio]

I think this person might be on to something.

[VMRuiz]

I was able to reproduce this issue by importing twisted.ssl.Certificate:

(great_expectations) ➜  scrapy git:(master) ✗ python
Python 3.10.14 (main, Mar 19 2024, 21:46:16) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted.internet.ssl
>>> import great_expectations
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/__init__.py", line 32, in <module>
    register_core_expectations()
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/registry.py", line 187, in register_core_expectations
    from great_expectations.expectations import core  # noqa: F401
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/core/__init__.py", line 1, in <module>
    from .expect_column_distinct_values_to_be_in_set import (
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/core/expect_column_distinct_values_to_be_in_set.py", line 12, in <module>
    from great_expectations.expectations.expectation import (
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 2350, in <module>
    class BatchExpectation(Expectation, ABC):
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 287, in __new__
    newclass._register_renderer_functions()
  File "/Users/zyte/workspace/opensource/scrapy/.tox/great_expectations/lib/python3.10/site-packages/great_expectations/expectations/expectation.py", line 369, in _register_renderer_functions
    attr_obj: Callable = getattr(cls, candidate_renderer_fn_name)
AttributeError: __provides__. Did you mean: '__providedBy__'?

[VMRuiz]

Importing Certificate directly from its internal package seems to work:

(great_expectations) ➜  scrapy git:(master) ✗ python
Python 3.10.14 (main, Mar 19 2024, 21:46:16) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.internet._sslverify import Certificate
>>> import great_expectations

So it must be related to twisted.internet.ssl init code

[culpgrant]

@VMRuiz Thank you for looking into this! Do you think I should create an issue with zope?

[wRAR]

My first thought was also "I don't know if this is it a problem with Zope or a bad implementation by Twisted lib", as I'm not familiar with the zope.interface internals.

[GeorgeA92]

@culpgrant

Steps to Reproduce
This does work:

import great_expectations
import scrapy

If this works. What prevent You to just use this import order in Your task?

[Rishika70]

try...except Block: Wrap the import statements in a try...except block to gracefully handle the import error and provide informative messages:
try:
import great_expectations
import scrapy
except ImportError as e:
print(f"Error importing libraries: {e}")

If you don't need Great Expectations functionalities throughout your script, consider delaying the import until the point of use with a function

import scrapy

def use_great_expectations():
from great_expectations import some_great_expectations_function # Import only when needed

print("Great Expectations used")

use_great_expectations()

Make sure you're using a clean virtual environment to avoid conflicts with other installed packages. Reinstall Scrapy and Great Expectations in a fresh environment to see if it resolves the issue.
Check for version compatibility between Scrapy and Great Expectations