eudat-b2handle/pyhandle

Python library for HANDLE system

pyhandle's Issues

python_requires in setup.py not PEP440 compliant

Dear pyhandle developers,

I encounter issues with the Python requirements in the setup.py python_requires='>=3.6.*<3.11' when running pipenv install. The requirement is included in the Pipfile.lock, but not correctly and subsequently pyhandle is not installed due to: Ignoring pyhandle: markers 'python_version >= "3.6.*<3.11"' don't match your environment whereas I'm using Python 3.10.8.
This requirement turns out to be not PEP440 compliant as it should be according to the description.

I think it should become something like this: python_requires='>=3.6, <3.11'

handle_server_url from credentials file is not used properly

If a credentials JSON file specifies a 'handle_server_url' param, this will be overwritten in the REST client's init method, even if the init method's own handle_server_url param is not set. A simple "if handle_server_url != None" would probably help..?

https://github.com/EUDAT-B2SAFE/PYHANDLE/blob/d1b0f2fbe186f47d6909362156a27c675930d675/pyhandle/client/resthandleclient.py#L102

pyhandle 1.1.1 fails to install on Centos 7.x using pip2

When trying to install pyhandle 1.1.1 with pip2 it fails with the following error:

Yue are using pip version 8.1.2, however version 22.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pyhandle
  Downloading https://files.pythonhosted.org/packages/c6/c1/f787154f5f9f2ae242c40875f12bc39b9bd98161f286252e77c80982ccbb/pyhandle-1.1.1.tar.gz (76kB)
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
      warnings.warn(msg)
    error in pyhandle setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-kpCUCs/pyhandle/
You are using pip version 8.1.2, however version 22.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip3 install -r /app/B2SAFE-core/scripts/tests/requirements.txt &&     pip2 install PyMySQL==0.10.1 &&     pip2 install pyhandle' returned a non-zero code: 1

Does not support Python 3.6 and higher

We are currently implementing the Handle system for our archives so we can request PID's. However, trying to pip install the package, I discovered that only Python 3.5 and lower are supported, while we are running more modern versions.

In your devel branch there is support for at least Python 3.7 (Python 3.8 would be really nice). Is it possible to release a new version with these fixes? It would be really appreciated!

Bug in rare case of trying to modify but pass empty dict

This is a bug that can occur when we try to modify a handle record (via REST client), but pass an empty dict of changes.

In that case, pyhandle will try to return a variable that is not assigned, here:

https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/master/pyhandle/client/resthandleclient.py#L602

When the variable nothingchanges is True, the variable resp never gets defined or assigned, so we run into an exception here:

https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/master/pyhandle/client/resthandleclient.py#L578

pidify_1  |     self.__handle_client.modify_handle_value(handle, **record_kv)
pidify_1  |   File "/usr/local/lib/python3.12/site-packages/pyhandle/client/resthandleclient.py", line 490, in modify_handle_value
pidify_1  |     return self.__handle_modification(handle, ttl, add_if_not_exist, overwrite, **kvpairs)
pidify_1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pidify_1  |   File "/usr/local/lib/python3.12/site-packages/pyhandle/client/resthandleclient.py", line 602, in __handle_modification
pidify_1  |     return json.loads(decoded_response(resp))['handle']
pidify_1  |                                        ^^^^
pidify_1  | UnboundLocalError: cannot access local variable 'resp' where it is not associated with a value

Make parameter "indices" deprecated

I introduced indices to allow passing several handle record indices.

But now we allow passing various options which are appended to the HTTP GET request (e.g. type, which results in ?type=bla being appended to the Handle Server URL). So why should we not let users directly use index for indices?

In case of type, we allow passing lists. So, type=['bla', 'bli'] will result in ?type=bla&type=bli. We could do the same with index.

Opinions?

Library requires redundant specification of type of client (just annoying)

If I instantiate pyhandle using JSON creds file, it needs me to specify the client inside it, and also in the script (PyHandleClient('rest')):

creds = pyhandle.clientcredentials.PIDClientCredentials.load_from_JSON(credentials_file)
client = pyhandle.handleclient.PyHandleClient('rest').instantiate_with_credentials(creds, HTTPS_verify=False)

Not passing it in the script (PyHandleClient()) fails, even if it is in the JSON file:

# client = pyhandle.handleclient.PyHandleClient().instantiate_with_credentials(creds, HTTPS_verify=False)
Traceback (most recent call last):
File "check_api_interaction_create.py", line 36, in <module>
client = pyhandle.handleclient.PyHandleClient().instantiate_with_credentials(
TypeError: __init__() missing 1 required positional argument: 'client'

If client not provided in JSON, but it's provided in the script (PyHandleClient('rest')):

File "/foo/venv3/lib/python3.6/site-packages/pyhandle-1.0.4-py3.6.egg/pyhandle/clientcredentials.py", line 179, in __init__
raise CredentialsFormatError(msg=msg)
pyhandle.handleexceptions.CredentialsFormatError: Problem with credentials: Client not provided or empty.

If the client provided, but inconsistent - JSON says "db", init says "rest":

raise CredentialsFormatError(msg=msg)
pyhandle.handleexceptions.CredentialsFormatError: Problem with credentials: (['db_host', 'db_user', 'db_password', 'db_name']) are missing or empty.

Python 3.7 support?

Hi,

Are there any plans to support Python 3.7? Your setup.py indicates the following Python versions are supported: python_requires='>=2.6,!=3.0.,!=3.1.,!=3.2.,!=3.3.,!=3.4.*,<3.6',

Is there anything holding back support for newer Python versions?

I'm working on a python script to upload handle identifiers to SURFsara servers:
https://github.com/Rijksmuseum/handle_upload
At the moment I'm using Python 2.7, which works ok, but I would prefer to migrate to 3.7 at some time.

Intercept "hdl:" in Handle names

Example:
result = client.retrieve_handle_record_json("hdl:123/456")

This causes an exception because it does not match the expected syntax (index:prefix/suffix). While this is correct, use of the 'hdl:' prefix (or doi:) is sort of legitimate and is also accepted by the main proxies (hdl.handle.net and doi.org). Thus, all methods accepting Handle names should intercept and remove 'hdl:' and 'doi:' before trying an int conversion.

New test command, update readme

Currently README recommends running the tests via python setup.py test , but this gives a deprecated warning (in python 3.6.8), so we should recommend a different command. See error message below.

(venv3) [m@localhost PYHANDLE]$ python setup.py test
/home/merret/work/repos_and_scripts/pyhandle/PYHANDLE/pyhandle/venv3/lib/python3.6/site-packages/setuptools/dist.py:493: UserWarning: Normalizing '1.0.5-dev' to '1.0.5.dev0'
  warnings.warn(tmpl.format(**locals()))
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
/home/merret/work/repos_and_scripts/pyhandle/PYHANDLE/pyhandle/venv3/lib/python3.6/site-packages/setuptools/installer.py:30: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
  SetuptoolsDeprecationWarning,

Documentation should give import example

The section in the docs here

https://eudat-b2safe.github.io/PYHANDLE/pyhandleclientrest.html#general-usage

explains to the first-time user how to get a PyHandle client instance. It is however not clear from where to import PyHandleClient.
E.g., just add a line
from pyhandle.handleclient import PyHandleClient

Cannot load credentials (key, certificate) from subdirectory

When I run a script using pyhandle that loads the credentials info from a JSON file that sits in a subdirectory, where also private key and certificate are sitting, I cannot load them using relative paths.

Files:

workingdir/
workingdir/script.py
workingdir/credentials/
workingdir/credentials/creds.json
workingdir/credentials/privkey.pem
workingdir/credentials/clientcert.pem

Contents of the creds.json:

{
    "private_key": "./privkey.pem",
    "certificate_only": "./clientcert.pem",
    "prefix": "21.xxx",
    "handleowner": "200:0.NA/21.xxx",
    "HTTPS_verify": "./xxx_bundle.crt",
    "client": "rest",
    "handle_server_url": "xxx"
}
# Paths of key+cert are relative to JSON file!

And I want to do this:

creds = pyhandle.clientcredentials.PIDClientCredentials.load_from_JSON('./credentials/creds.json')
client = pyhandle.handleclient.PyHandleClient('rest').instantiate_with_credentials(creds, HTTPS_verify=False)

But I get an error:

pyhandle.handleexceptions.CredentialsFormatError: Problem with credentials: The certificate file was not found at the specified path: ./clientcert.pemThe private key file was not found at the specified path: ./privkey.pem.

If I add the subdirectory to the JSON file:

{
    "private_key": "./credentials/privkey.pem",
    "certificate_only": "./credentials/clientcert.pem",
    "prefix": "21.xxx",
    "handleowner": "200:0.NA/21.xxx",
    "HTTPS_verify": "./xxx_bundle.crt",
    "client": "rest",
    "handle_server_url": "xxx"
}
# Paths of key+cert are relative to script!

Then I also get an error:

pyhandle.handleexceptions.CredentialsFormatError: Problem with credentials: (certficate file): The file was not found at the specified path: /workingdir/credentials/credentials/clientcert.pem.

Now the subdirectory is added twice!
No way out!

not able to build rpm from code

Hi,

I tried to build an rpm from the code. So we can install an rpm. On Centos 6.8 (python 2.6) it gives:

$ python setup.py bdist_rpm
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
running bdist_rpm
running egg_info
creating pyhandle.egg-info
writing requirements to pyhandle.egg-info/requires.txt
writing pyhandle.egg-info/PKG-INFO
writing top-level names to pyhandle.egg-info/top_level.txt
writing dependency_links to pyhandle.egg-info/dependency_links.txt
writing manifest file 'pyhandle.egg-info/SOURCES.txt'
reading manifest file 'pyhandle.egg-info/SOURCES.txt'
writing manifest file 'pyhandle.egg-info/SOURCES.txt'
creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/rpm
creating build/bdist.linux-x86_64/rpm/SOURCES
creating build/bdist.linux-x86_64/rpm/SPECS
creating build/bdist.linux-x86_64/rpm/BUILD
creating build/bdist.linux-x86_64/rpm/RPMS
creating build/bdist.linux-x86_64/rpm/SRPMS
writing 'build/bdist.linux-x86_64/rpm/SPECS/pyhandle.spec'
running sdist
warning: sdist: standard file not found: should have one of README, README.txt
warning: sdist: missing required meta-data: url
warning: sdist: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating pyhandle-1.0.0
creating pyhandle-1.0.0/pyhandle
creating pyhandle-1.0.0/pyhandle.egg-info
creating pyhandle-1.0.0/pyhandle/tests
creating pyhandle-1.0.0/pyhandle/tests/testcases
making hard links in pyhandle-1.0.0...
hard linking setup.cfg -> pyhandle-1.0.0
hard linking setup.py -> pyhandle-1.0.0
hard linking pyhandle/__init__.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/clientcredentials.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/compatibility_helper.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/dbhsexceptions.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/handleclient.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/handleexceptions.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/handlesystemconnector.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/hsresponses.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/searcher.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle/utilhandle.py -> pyhandle-1.0.0/pyhandle
hard linking pyhandle.egg-info/PKG-INFO -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle.egg-info/SOURCES.txt -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle.egg-info/dependency_links.txt -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle.egg-info/not-zip-safe -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle.egg-info/requires.txt -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle.egg-info/top_level.txt -> pyhandle-1.0.0/pyhandle.egg-info
hard linking pyhandle/tests/__init__.py -> pyhandle-1.0.0/pyhandle/tests
hard linking pyhandle/tests/main_test_script.py -> pyhandle-1.0.0/pyhandle/tests
hard linking pyhandle/tests/mockresponses.py -> pyhandle-1.0.0/pyhandle/tests
hard linking pyhandle/tests/test_script.py -> pyhandle-1.0.0/pyhandle/tests
hard linking pyhandle/tests/utilities.py -> pyhandle-1.0.0/pyhandle/tests
hard linking pyhandle/tests/testcases/__init__.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/clientcredentials_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_2_read_patched_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_read_integration_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_read_patched_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_search_integration_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_search_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_write_integration_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleclient_write_patched_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleconnector_patched_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/handleconnector_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
hard linking pyhandle/tests/testcases/utilconfig_unit_test.py -> pyhandle-1.0.0/pyhandle/tests/testcases
copying setup.cfg -> pyhandle-1.0.0
Writing pyhandle-1.0.0/setup.cfg
creating dist
tar -cf dist/pyhandle-1.0.0.tar pyhandle-1.0.0
gzip -f9 dist/pyhandle-1.0.0.tar
tar -cf dist/pyhandle-1.0.0.tar pyhandle-1.0.0
gzip -f9 dist/pyhandle-1.0.0.tar
removing 'pyhandle-1.0.0' (and everything under it)
copying dist/pyhandle-1.0.0.tar.gz -> build/bdist.linux-x86_64/rpm/SOURCES
building RPMs
rpmbuild -ba --define _topdir /home/robertv/git/PYHANDLE/build/bdist.linux-x86_64/rpm --clean build/bdist.linux-x86_64/rpm/SPECS/pyhandle.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.LsIirR
+ umask 022
+ cd /home/robertv/git/PYHANDLE/build/bdist.linux-x86_64/rpm/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/robertv/git/PYHANDLE/build/bdist.linux-x86_64/rpm/BUILD
+ rm -rf pyhandle-1.0.0
+ /bin/tar -xvvf -
+ /usr/bin/gzip -dc /home/<user>/git/PYHANDLE/build/bdist.linux-x86_64/rpm/SOURCES/pyhandle-1.0.0.tar.gz
...
-rw-rw-r-- <user>/<user>   774 2017-05-10 12:22 pyhandle-1.0.0/PKG-INFO
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd pyhandle-1.0.0
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.lckHdK
+ umask 022
+ cd /home/<user>/git/PYHANDLE/build/bdist.linux-x86_64/rpm/BUILD
+ cd pyhandle-1.0.0
+ LANG=C
+ export LANG
+ unset DISPLAY
+ python setup.py build
Traceback (most recent call last):
  File "setup.py", line 67, in <module>
    long_description=open('README.md').read()
IOError: [Errno 2] No such file or directory: 'README.md'
error: Bad exit status from /var/tmp/rpm-tmp.lckHdK (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.lckHdK (%build)
error: command 'rpmbuild' failed with exit status 1

What are we doing wrong?

search_handles should support a 'limit' parameter

The HandleReverseLookupServlet has a default limit for queries of 1000 (only returns max. 1000 Handles per query). This can be overridden with a limit URL parameter. The search_handles method should also support this parameter explicitly.
Current workaround is: introduce 'limit' as an allowed_search_key and then set it to e.g. "100000" (as string).

Is pyhandleclient.py superfluous?

Seems to be a duplicate of handleclient.py but with less code. Is this a leftover from earlier implementation attempts?

https://github.com/EUDAT-B2SAFE/PYHANDLE/blob/d1b0f2fbe186f47d6909362156a27c675930d675/pyhandle/pyhandleclient.py#L1

Python 3.10.3 deprecation issues

../../../../../../../usr/local/lib/python3.10/site-packages/future-0.18.2-py3.10.egg/past/builtins/misc.py:45

/usr/local/lib/python3.10/site-packages/future-0.18.2-py3.10.egg/past/builtins/misc.py:45: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses

from imp import reload

../client/resthandleclient.py:157

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/client/resthandleclient.py:157: DeprecationWarning: invalid escape sequence '*'

'''

../client/resthandleclient.py:174

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/client/resthandleclient.py:174: DeprecationWarning: invalid escape sequence '*'

'''

../client/resthandleclient.py:203

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/client/resthandleclient.py:203: DeprecationWarning: invalid escape sequence '*'

'''

../client/resthandleclient.py:227

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/client/resthandleclient.py:227: DeprecationWarning: invalid escape sequence '*'

'''

pyhandle/tests/main_test_script.py::RESTHandleClientReadaccessFakedTestCase::test_get_value_from_handle_normal

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_read_patched_unit_test.py:42: DeprecationWarning: Please use assertEqual instead.

self.assertEquals(val, 'val1',

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_create_revlookup_query_forbiddenkeys

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:51: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'Cannot search for key[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_create_revlookup_query_fulltext

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:47: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'Full-text search is not implemented yet[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_create_revlookup_query_noterms

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:55: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'No search terms have been specified[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_search_handle_for_forbiddenkeys

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:31: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'Cannot search for key[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_search_handle_for_fulltext

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:37: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'Full-text search is not implemented yet[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientSearchNoAccessTestCase::test_search_handle_noterms

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_search_unit_test.py:43: DeprecationWarning: Please use assertRaisesRegex instead.

with self.assertRaisesRegexp(ReverseLookupException, 'No search terms have been specified[.]*'):

pyhandle/tests/main_test_script.py::RESTHandleClientNoaccessTestCase::test_create_authentication_string

/home/jenkins/workspace/B2HANDLE-ORG_PYHANDLE_PR-61@5/PYHANDLE/pyhandle/tests/testcases/handleclient_unit_test.py:178: DeprecationWarning: Please use assertEqual instead.

self.assertEquals(expected, auth,

Logging too verbose

Logging in INFO level is way too verbose. I keep adding logging.getLogger('pyhandle').setLevel(logging.WARNING) too all my code.

SSL error in python 3

When using http_verify=True in python2, I get

(vi_3.3)[merret@centos]$ python modify_single_pid.py -v
2020-09-04 21:35:17,948 - __main__ - INFO - Will modify this handle: 21.14101/TESTTESTTEST
2020-09-04 21:35:17,948 - __main__ - INFO - Handle record will contain these values: {'TEST': 'TRUEE', 'SOMEFLAG': 'HIHI', 'CREATED': '20200904', 'DESCRIPTION': 'Apple pie', 'CONTACT': '[email protected]', 'DELETEME': 'DO IT'}
Traceback (most recent call last):
  File "modify_single_pid.py", line 137, in <module>
    creds, HTTPS_verify=https_verify)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handleclient.py", line 68, in instantiate_with_credentials
    return self.handle_client.instantiate_with_credentials(credentials, **config)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/client/resthandleclient.py", line 243, in instantiate_with_credentials
    inst = RESTHandleClient(**key_value_pairs)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/client/resthandleclient.py", line 110, in __init__
    self.__handlesystemconnector = HandleSystemConnector(handleclient=self, **args)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handlesystemconnector.py", line 83, in __init__
    self.__setup_for_writeaccess(args)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handlesystemconnector.py", line 164, in __setup_for_writeaccess
    self.__setup_for_auth_by_user_and_pw()
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handlesystemconnector.py", line 177, in __setup_for_auth_by_user_and_pw
    self.check_if_username_exists(self.__username)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handlesystemconnector.py", line 463, in check_if_username_exists
    resp = self.send_handle_get_request(handle)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/pyhandle/handlesystemconnector.py", line 278, in send_handle_get_request
    resp = self.__session.get(url, headers=head, verify=veri)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/home/merret/vi_3.3/lib/python3.3/site-packages/urllib3/util/ssl_.py", line 350, in ssl_wrap_socket
    context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
TypeError: load_verify_locations() takes at most 2 arguments (3 given)

Using the same script in python2, all I get is this:

Traceback (most recent call last):
  File "modify_single_pid.py", line 137, in <module>
    creds, HTTPS_verify=https_verify)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handleclient.py", line 68, in instantiate_with_credentials
    return self.handle_client.instantiate_with_credentials(credentials, **config)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/client/resthandleclient.py", line 243, in instantiate_with_credentials
    inst = RESTHandleClient(**key_value_pairs)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/client/resthandleclient.py", line 110, in __init__
    self.__handlesystemconnector = HandleSystemConnector(handleclient=self, **args)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 83, in __init__
    self.__setup_for_writeaccess(args)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 164, in __setup_for_writeaccess
    self.__setup_for_auth_by_user_and_pw()
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 177, in __setup_for_auth_by_user_and_pw
    self.check_if_username_exists(self.__username)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 463, in check_if_username_exists
    resp = self.send_handle_get_request(handle)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 278, in send_handle_get_request
    resp = self.__session.get(url, headers=head, verify=veri)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/merret/work/githubrepos/ESGF/fixerscript/handletools/cmip6tools/venv2/lib/python2.7/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pid1.dkrz.de', port=8001): Max retries exceeded with url: /api/handles/21.14101/ADMINLIST (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)'),))

Return values should be consistent

• The method modify_handle_value (https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/master/pyhandle/client/resthandleclient.py#L389) does not return anything if successful.
• Same for delete_handle_value (https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/master/pyhandle/client/resthandleclient.py#L502) and delete_handle
• But register_handle_json returns the handle: https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/master/pyhandle/client/resthandleclient.py#L600 , same for __handle_registering and all those wrapping it.

handle_server_url usage causes exception when using instantiate_for_read_access

client = PyHandleClient("rest").instantiate_for_read_access(handle_server_url="https://handle.dkrz.de:8003")
INFO:pyhandle.client.resthandleclient:Instantiating RESTHandleClient at 2018-04-13_14:00
INFO:pyhandle.handlesystemconnector:Instantiating HandleSystemConnector
INFO:pyhandle.handlesystemconnector: - handle_server_url set to default: https://hdl.handle.net
INFO:pyhandle.handlesystemconnector: - url_extension_REST_API set to default: /api/handles/
INFO:pyhandle.handlesystemconnector: - https_verify set to default: True
INFO:pyhandle.searcher:Instantiating Searcher
INFO:pyhandle.searcher: - https_verify set to default: True
INFO:pyhandle.searcher: - allowed_search_keys set to default: ['URL', 'CHECKSUM']
INFO:pyhandle.searcher: - solrbaseurl: No default.
INFO:pyhandle.searcher: - reverselookup_url_extension set to default: /hrls/handles/
INFO:pyhandle.searcher: - reverselookup_username: Not specified. No default.
INFO:pyhandle.searcher: - reverselookup_password: Not specified. No default.
INFO:pyhandle.searcher:Reverse lookup not possible. Neither username nor password were provided.
INFO:pyhandle.searcher:Reverse lookup not possible. No URL for reverse lookup provided.
INFO:pyhandle.client.resthandleclient: - HS_ADMIN_permissions set to default: 011111110011
INFO:pyhandle.client.resthandleclient: - modify_HS_ADMIN set to default: False
INFO:pyhandle.client.resthandleclient: - handleowner: Will be set to default for each created handle separately.
INFO:pyhandle.client.resthandleclient:Instantiating RESTHandleClient at 2018-04-13_14:00
INFO:pyhandle.handlesystemconnector:Instantiating HandleSystemConnector
Traceback (most recent call last):
File "", line 1, in
File "/home/tobias/virtualenv/lib/python2.7/site-packages/pyhandle/handleclient.py", line 65, in instantiate_for_read_access
return self.handle_client.instantiate_for_read_access(config)
File "/home/tobias/virtualenv/lib/python2.7/site-packages/pyhandle/client/resthandleclient.py", line 168, in instantiate_for_read_access
inst = RESTHandleClient(handle_server_url, **config)
File "/home/tobias/virtualenv/lib/python2.7/site-packages/pyhandle/client/resthandleclient.py", line 110, in init
self.__handlesystemconnector = HandleSystemConnector(handleclient=self, **args)
File "/home/tobias/virtualenv/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 79, in init
self.__store_args_or_set_to_defaults(args, defaults)
File "/home/tobias/virtualenv/lib/python2.7/site-packages/pyhandle/handlesystemconnector.py", line 97, in __store_args_or_set_to_defaults
LOGGER.info(' - handle_server_url set to '+self.__handle_server_url)
TypeError: cannot concatenate 'str' and 'dict' objects

REST client: Cannot add options

Sometimes I need the option ?auth in GET requests to make sure the primary server is requested. I cannot pass that.

pyhandle fails to install on CentOS Linux release 7.9.2009 (Core)

We get the following error when trying to install the latest version:

$ pip3 install --user pyhandle
Collecting pyhandle
  Using cached https://files.pythonhosted.org/packages/4f/5e/e959a063a8746da34aa860700217398da6061e40e04dea4b366c0428500f/pyhandle-1.2.1-py2.py3-none-any.whl
Collecting pymysql==0.8.0; python_version < "3.11.0" (from pyhandle)
  Using cached https://files.pythonhosted.org/packages/e5/07/c0f249aa0b7b0517b5843eeab689b9ccc6a6bb0536fc9d95e65901e6f2ac/PyMySQL-0.8.0-py2.py3-none-any.whl
Collecting requests==2.28.1; python_version >= "3.6.0" (from pyhandle)
  Could not find a version that satisfies the requirement requests==2.28.1; python_version >= "3.6.0" (from pyhandle) (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.8.0, 2.8.1, 2.9.0, 2.9.1, 2.9.2, 2.10.0, 2.11.0, 2.11.1, 2.12.0, 2.12.1, 2.12.2, 2.12.3, 2.12.4, 2.12.5, 2.13.0, 2.14.0, 2.14.1, 2.14.2, 2.15.1, 2.16.0, 2.16.1, 2.16.2, 2.16.3, 2.16.4, 2.16.5, 2.17.0, 2.17.1, 2.17.2, 2.17.3, 2.18.0, 2.18.1, 2.18.2, 2.18.3, 2.18.4, 2.19.0, 2.19.1, 2.20.0, 2.20.1, 2.21.0, 2.22.0, 2.23.0, 2.24.0, 2.25.0, 2.25.1, 2.26.0, 2.27.0, 2.27.1)
No matching distribution found for requests==2.28.1; python_version >= "3.6.0" (from pyhandle)

A lower version installs:

$ pip3 install --user pyhandle==1.2.0
Collecting pyhandle==1.2.0
  Downloading https://files.pythonhosted.org/packages/30/e0/6c07d215d8b13fb330e45eea9bd8b9e6d705af64a7e4211447faa44aa5ed/pyhandle-1.2.0.tar.gz (76kB)
    100% |████████████████████████████████| 81kB 11.2MB/s
Collecting requests==2.23.0 (from pyhandle==1.2.0)
  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 13.0MB/s
Collecting datetime (from pyhandle==1.2.0)
  Using cached https://files.pythonhosted.org/packages/30/ff/03e2b71fbcdd6a084c2158cf2ecb0dbdde3cac0d974cf0815fcae79f6568/DateTime-5.0-py3-none-any.whl
Requirement already satisfied: future in /usr/lib/python3.6/site-packages (from pyhandle==1.2.0)
Requirement already satisfied: six in /usr/lib/python3.6/site-packages (from pyhandle==1.2.0)
Collecting pymysql==0.8.0 (from pyhandle==1.2.0)
  Using cached https://files.pythonhosted.org/packages/e5/07/c0f249aa0b7b0517b5843eeab689b9ccc6a6bb0536fc9d95e65901e6f2ac/PyMySQL-0.8.0-py2.py3-none-any.whl
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.6/site-packages (from requests==2.23.0->pyhandle==1.2.0)
Collecting certifi>=2017.4.17 (from requests==2.23.0->pyhandle==1.2.0)
  Downloading https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl (158kB)
    100% |████████████████████████████████| 163kB 8.7MB/s
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.6/site-packages (from requests==2.23.0->pyhandle==1.2.0)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.6/site-packages (from requests==2.23.0->pyhandle==1.2.0)
Collecting zope.interface (from datetime->pyhandle==1.2.0)
  Downloading https://files.pythonhosted.org/packages/d0/75/c80af74b361cedd35bdfb45ade9c22320d3431722186ff8ab1614df80059/zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253kB)
    100% |████████████████████████████████| 256kB 5.8MB/s
Requirement already satisfied: pytz in ./.local/lib/python3.6/site-packages (from datetime->pyhandle==1.2.0)
Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from zope.interface->datetime->pyhandle==1.2.0)
Installing collected packages: certifi, requests, zope.interface, datetime, pymysql, pyhandle
  Running setup.py install for pyhandle ... done
Successfully installed certifi-2023.7.22 datetime-5.0 pyhandle-1.2.0 pymysql-0.8.0 requests-2.23.0 zope.interface-5.5.2

On this version of OS 3.6 is the default python version.

check_handle_syntax fails with unhelpful error message when "hdl:prefix/suffix" is passed

When I call the method check_handle_syntax(string) with the handle hdl:21.140x/bla I get this error message:

pyhandle.handleexceptions.HandleSyntaxError: Handle hdl:21.1410x/bla does not have expected syntax: Index is not an integer.
Expected: index:prefix/suffix

But hdl:21.140x/bla is a perfectly valid handle. At least, the error message should be more helpful.

Because of the colon :, the method assumes we're trying to pass a single field, i.e. with an index in front. That does not make sense to me, as we're checking the syntax of a handle.

Where?

• https://github.com/EUDAT-B2HANDLE/PYHANDLE/blob/1f3e075c5eef40f78970a0d3d57a933a1baf1818/pyhandle/utilhandle.py

• PYHANDLE/pyhandle/utilhandle.py

  Line 66 in 1f3e075

  check_handle_syntax_with_index(string, base_already_checked=True)

Uninstallable because hard-dependency on requests==2.23.0

Because the pyhandle package locks requests on version 2.23.0 (not higher, not lower), it is uninstallable together with some other packages in our project that demand newer versions of requests.

Is there a reason that setup.py locks it on this version?

Roadmap for releasing PyHandle compatible with Python >=3.9

We are currently migrating our servers and would like to upgrade our Python versions to one of the latest versions. Currently PyHandle is our bottleneck. Do you have any indication of when we can expect a new release that is compatible with Python 3.9 or higher?

Investigate PIDClientCredentials class when instantiating a search

One further issue may be that if private key authentication is to be used, it is logically nto required to provide a password in the credentials file. However, omitting a password will cause the credentials file read method to throw an error, complaining that without password, searchign will not be possible.
Suggested solution: modify credentials JSON read method so that possibility for searches is not checked if a specific flag parameter is given. Or make a subclass that is specific for the REST mode and only there the error will be thrown.
Text from #13

Make Batch client work with PIDClientCredentials

The PIDClientCredentials is a good class to conveniently cover all possible authentication cases and get the information either through function calls or from a JSON file.
The batch client has however its own methods. To use both clients transparently, the batch client should be able to work with PIDClientCredential instances as well (e.g. via a generic authenticate() method that takes a PIDClientCredentials instance as only parameter).

Syntax Warning to latest python versions

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/handlesystemconnector.py:197: SyntaxWarning: "is not" with a literal. Did you mean "!="?

if msg is not '':

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/handlesystemconnector.py:498: SyntaxWarning: "is" with a literal. Did you mean "=="?

if action is 'GET':

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/handlesystemconnector.py:502: SyntaxWarning: "is" with a literal. Did you mean "=="?

elif action is 'PUT' or action is 'DELETE':

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/handlesystemconnector.py:502: SyntaxWarning: "is" with a literal. Did you mean "=="?

elif action is 'PUT' or action is 'DELETE':

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/handlesystemconnector.py:510: SyntaxWarning: "is" with a literal. Did you mean "=="?

if action is 'PUT':

/usr/local/lib/python3.9/site-packages/pyhandle-1.0.5.dev0-py3.9.egg/pyhandle/client/dbhandleclient.py:297: SyntaxWarning: "is not" with a literal. Did you mean "!="?

if query_result_as_dict is not 0: