Keepalive support now does not properly work on Python 2.x

[rdflib]

Keepalive support now does not properly work on Python 2.x

[wikier]

OK, now build looks fine: https://travis-ci.org/RDFLib/sparqlwrapper/builds/89181455

Can you please @xflr6 if it works for you before we roll-out a release with this hot-fix?

[xflr6]

$ pip install dist\SPARQLWrapper-1.7.3.dev0.zip
(...)
Collecting urlgrabber<=3.9.0 (from SPARQLWrapper==1.7.3.dev0)
  Downloading urlgrabber-3.1.0.tar.gz (79kB)
Collecting simplejson==2.0.9 (from SPARQLWrapper==1.7.3.dev0)
  Downloading simplejson-2.0.9.tar.gz (115kB)
(...)
Successfully installed SPARQLWrapper-1.7.3.dev0 simplejson-2.0.9 urlgrabber-3.1.0

Note that 3.1.0 and 3.9.1 seem to be the only versions of urlgrabber on PyPI.

What is the rationale behind requiring simplejson on PY2?
Why pin it to such an outdated version (might give conflicts with other packages)?

>>> import rdflib
INFO:rdflib:RDFLib Version: 4.2.1
>>> import warnings
>>> warnings.simplefilter('error')
>>> endpoint = rdflib.ConjunctiveGraph('SPARQLStore')
>>> endpoint.open('http://live.dbpedia.org/sparql')
>>> endpoint.query('select distinct ?Concept where {[] a ?Concept} LIMIT 100')
<rdflib.plugins.sparql.results.xmlresults.XMLResult object at 0x01D0AA50>

Works (but see below).

Wouldn't it be much easier/cleaner to adapt keepalive to support both Python 2 and 3 out of the box?

Looks like urlgrabber added the pycurl dependency in 3.9.0 (which is not on PyPI).

Note that 3.9.1 is broken (imports pycurl during setup):

$ pip install -U urlgrabber
Collecting urlgrabber
  Downloading urlgrabber-3.9.1.tar.gz (72kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
        import urlgrabber as _urlgrabber
      File "urlgrabber\__init__.py", line 54, in <module>
        from grabber import urlgrab, urlopen, urlread
      File "urlgrabber\grabber.py", line 427, in <module>
        import pycurl
    ImportError: No module named pycurl

Works only if pycurl is already installed (requiring libraries/compiler infrastrucure):

$ pip install pycurl-7.19.5.1-cp27-none-win32.whl
Successfully installed pycurl-7.19.5.1

$ pip install -U urlgrabber
Successfully installed urlgrabber-3.9.1

This finally reveals current setUseKeepAlive() being broken:

>>> import rdflib
INFO:rdflib:RDFLib Version: 4.2.1
>>> import warnings
>>> warnings.simplefilter('error')
>>> endpoint = rdflib.ConjunctiveGraph('SPARQLStore')
Traceback (most recent call last):
(...)
  line 409, in setUseKeepAlive
    if HTTPHandler:
UnboundLocalError: local variable 'HTTPHandler' referenced before assignment

[wikier]

Wouldn't it be much easier/cleaner to adapt keepalive to support both Python 2 and 3 out of the box?

The reason why we can keep using keepalive as library is because its license (LGPL). I did not touch the code, since for Python 2.x urlgrabber should work. But if you feel you can provide a version that support both versions of the language, provide a PR to the project.

[xflr6]

This would be a quick & dirty proof-of-concept.

The 2to3 ported code in your keepalive package seems to be based on a pretty old version.

For a more faithful replacement of 3.1.0, better port a newer version like this one to 2/3.

[wikier]

OK, I see, the SPARQLStore enables keepalive by default: https://github.com/RDFLib/rdflib/blob/master/rdflib/plugins/stores/sparqlstore.py#L246

[wikier]

I've been working on the keepalive side to get a more compact code base that can be automatically ported to Python 3.x with 2to3, and the results in the 0.3 release.

The current HEAD now uses that version for both languages, so my tests work:

Python 2.7:

$ nosetests test/dbpedia.py:SPARQLWrapperTests.testKeepAlive --nocapture
.
----------------------------------------------------------------------
Ran 1 test in 4.859s

OK

Python 3.4:

$ ./run_tests_py3.sh
$ nosetests3 build/py3_testing/test/dbpedia.py:SPARQLWrapperTests.testKeepAlive --nocapture
.
----------------------------------------------------------------------
Ran 1 test in 6.114s

OK

(Notice that the integration tests against an external SPARQL endpoint (DBpedia) are on purpose not automatically ran by Travis CI)

So now @xflr6, could you please repeat your test again?

[xflr6]

Of course:

>>> import rdflib
INFO:rdflib:RDFLib Version: 4.2.1
>>> import warnings
>>> warnings.simplefilter('error')
>>> endpoint = rdflib.ConjunctiveGraph('SPARQLStore')
>>> endpoint.open('http://live.dbpedia.org/sparql')
>>> endpoint.query('select distinct ?Concept where {[] a ?Concept} LIMIT 100')
Traceback (most recent call last):
(...)
  File "keepalive\keepalive.py", line 335, in http_open
    return self.do_open(req)
  File "keepalive\keepalive.py", line 235, in do_open
    self._start_transaction(h, req)
  File "keepalive\keepalive.py", line 315, in _start_transaction
    h.putrequest('GET', req.selector)
  File "lib\urllib2.py", line 256, in __getattr__
    raise AttributeError, attr
AttributeError: selector

>>> import pdb; pdb.pm()
> lib\urllib2.py(256)__getattr__()
-> raise AttributeError, attr
(Pdb) u
> keepalive\keepalive.py(315)_start_transaction()
-> h.putrequest('GET', req.selector)
(Pdb) req.__dict__.keys()
['_Request__r_type', '_Request__original', 'data', '_tunnel_host', 'host', 'origin_req_host', 'headers',
 '_Request__fragment', 'timeout', '_Request__r_host', 'unredirected_hdrs', 'unverifiable', 'type', 'port']

[wikier]

ummm... that error comes from this patch I introduced to avoid the getters deprecation/removal. @xflr6, does it even have get_selector()?

[xflr6]

Yes:

keepalive.py(315)_start_transaction()
-> h.putrequest('GET', req.selector)
(Pdb) req
<urllib2.Request instance at 0x013703C8>
(Pdb) req.get_selector()
'/sparql?query=PREFIX+(...)

[wikier]

OK, I've just patched; so @xflr6 to repeat again the test manually installing keepalive version 0.4.dev0 from the source code? For me the same test locally works in both Python 2.7 and 3.4.

[xflr6]

Successfully installed SPARQLWrapper-1.7.3.dev0 keepalive-0.3
Successfully installed keepalive-0.4.dev0

Works:

>>> endpoint.query('select distinct ?Concept where {[] a ?Concept} LIMIT 100')
<rdflib.plugins.sparql.results.xmlresults.XMLResult object at 0x01C8B250>

Maybe add a __version__ to keepalive/__init__.py to facilitate debugging in the future.

[wikier]

Cool!

Yes, I'll add some metadata to the package in the upcoming release.

[ephemeral8997]

How i can install keep_alive Python module please?