catmaid/catpy

Python client for the CATMAID API

catpy - Python client for the CATMAID API

Python client for the CATMAID API, plus some helpful tools for working with the data.

catpy's Issues

Parse json in responses only if content-type is json

HTTPS chunk encoding issue?

catpy has started throwing errors on some large requests - digging into them, it seems to be related to a new(ish) openssl client version on linux, which the openssl developers claim is indicative of a server-side issue, even though the old version of openssl does not trigger it.

Traceback (most recent call last):                                                                                                                        
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 685, in _update_chunk_length                
    self.chunk_left = int(line, 16)                                                                                                                       
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 752, in read_chunked
    self._update_chunk_length()
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 689, in _update_chunk_length
    raise httplib.IncompleteRead(line) 
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 560, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
    self._original_response.close()
  File "/home/barnesc/.pyenv/versions/3.7.5/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))


During handling of the above exception, another exception occurred:           

Traceback (most recent call last):     
  File "/home/barnesc/.pyenv/versions/brain/bin/ascending", line 11, in <module>                                                                             
    load_entry_point('brain-completion', 'console_scripts', 'ascending')()    
  File "/home/barnesc/work/code/brain_completion/brain_completion/scripts/section_a/ascending.py", line 148, in main                                         
    catmaid.delete_annotations(creates_annotations)                           
  File "/home/barnesc/work/code/brain_completion/brain_completion/catmaid_interface.py", line 393, in delete_annotations                                     
    return self.remove_annotations(annotations, entity_ids)                   
  File "/home/barnesc/work/code/brain_completion/brain_completion/catmaid_interface.py", line 389, in remove_annotations                                     
    return self.post(("annotations", "remove"), data)                         
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/client.py", line 66, in post                                        
    return self.fetch(relative_url, method="POST", data=data, raw=raw, **kwargs)                                                                             
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/applications/base.py", line 34, in fetch                            
    return self._catmaid.fetch(*args, **kwargs)                               
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/catpy/client.py", line 230, in fetch                                      
    response = self._session.post(url, data=data, **kwargs)                   
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 581, in post                                  
    return self.request('POST', url, data=data, json=json, **kwargs)          
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 533, in request                               
    resp = self.send(prep, **send_kwargs)                                     
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/sessions.py", line 686, in send                                  
    r.content                          
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 828, in content                                 
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''    
  File "/home/barnesc/.pyenv/versions/3.7.5/envs/brain/lib/python3.7/site-packages/requests/models.py", line 753, in generate                                
    raise ChunkedEncodingError(e)      
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) 

Issue chain:

• requests psf/requests#4248
• urllib3 urllib3/urllib3#1516
• openssl openssl/openssl#9360

Publish in JOSS?

Looks like minimal effort, and makes the library more discoverable and citeable.

https://joss.theoj.org/

https://joss.readthedocs.io/en/latest/submitting.html

Replicate Export widget functionality

Treenode and connector geometry, for example, is only available through the frontend currently.

Support image/volume fetching for common tile sources

This https://github.com/aschampion/diluvian/blob/master/diluvian/volumes.py#L1123 (or possibly a more basic version) would be great.

Could also include:

• Thread pooling (user-controlled thread count, or just use a requests session)

Catpy should throw if it receives an error response

Currently it only throws on a non-200 status code - but sometimes it receives a 200 full of legitimate JSON, but the JSON is just an object telling us that there was an error on django. We should catch such responses and raise an exception locally.

Docs recommendations

• catpy.util, not catpy.utils
• Reference get/post methods on client object

With help from Marc Corrales

Blacken code

https://github.com/ambv/black

To do:

• Set the stage by blackening everything in one fell swoop
• Add black to dev requirements
• Add black --check . to the tox config for python 3.6+
• Add pre-commit hook for running black in appropriate environments and warning users developing in an inappropriate environment

While we're at it, make lint can go in the pre-commit hooks too, if the CI's going to fail on it.

Scope of the Library

Hello!
Great to see the catpy library online - I think it will become useful to a lot of people. How do you see the scope of the library? Would you want to have it purely as an access library to the CATMAID API or do you envision also more sophisticated contributions for analysis (single neuron, circuits) and visualization?

If the latter is the case, I would be happy to contribute some of the things I wrote during my PhD (partly in the CAT package and elsewhere) and I believe this would be the right place to do so. However, it will be necessary to have a good basis data structure to represent skeletons, connectors, circuits including attributes on the nodes and edges, metadata etc. to build on top.

For my purposes, I built this myself, mostly based on NetworkX graphs or Numpy matrices, but it would make much more sense to have this standardized and ready-to-use for people getting started with novel analysis, and also to reduce duplication of programming efforts. I know there were talks about this earlier and I know about the repo by @ceesem (https://github.com/ceesem/catpy) also doing that. I am happy to contribute my insights to such a discussion if this is of interest here.

Test harness in CATMAID

As discussed elsewhere, the integration tests should be done as part of CATMAID's build because it's not worth standing up a whole instance of CATMAID for testing this library, and any breaking API changes by CATMAID which affect this library will also affect users and so should be addressed by the CATMAID maintainers as a priority.

How to extend the basic CatmaidClient

catpy could, eventually, do a lot of things. It would be nice to have the CatmaidClient class as a one-stop shop for the user - so that getting additional functionality doesn't require them re-authenticating a new object - but also not have CatmaidClient be some monolithic monstrosity over thousands of lines and with no internal namespacing and a huge initial import. Internal namespacing would also make it much easier to fold other people's work into the mainline client - especially if we didn't import it until we needed it.

I envision an abstract CatmaidExtension (/ CatmaidModule / CatmaidAddOn) class which looks something like this:

class CatmaidExtension(object):
    def __init__(self, catmaid_instance):
        self.catmaid = catmaid_instance

    def fetch(self, *args, **kwargs):
        # for convenience; could also just make devs explicitly call to self.catmaid
        self.catmaid.get(*args, **kwargs)

    # same for get, post

and which people could subclass for their own purposes.

Then on the main CatmaidClient:

CATMAID_EXTENSIONS = {
    'my_extension': ('my.extension.module.path', 'MyExtensionClassName'), 
}

class CatmaidClient(object):
    # existing methods

    def load_extension(self, extension_name):
        module_path, class_name = CATMAID_EXTENSIONS[extension_name]
        module = importlib.import_module(module_path, 'catpy')
        extension_class = getattr(module, class_name)
        setattr(self, extension_class(self))

As an example, if we had an ImageFetchExtension for getting numpy arrays from image tiles, could define it as:

CATMAID_EXTENSIONS = {
    ...
    'img': ('extensions.img_fetch', 'ImageFetchExtension'),
    ...
}

(this could be done in a config file)

And access it like so:

catmaid = CatmaidInstance()  # with credentials as required
catmaid.load_extension('img')
catmaid.img.fetch_image(*args, **kwargs)

This would mean that the CatmaidInstance class stays small and fairly static, and very few people need to tinker with it; we avoid big imports when most people won't need most functionality; it would be very easy to specify default modules for import; and it adds namespace functionality (to a depth of 1, anyway; easy enough to increase that). The auth and project of the CatmaidInstance could be changed from a single location. An extension could specify its own CatmaidExtension dependencies and make sure they're loaded first. CatmaidExtensions could also be included from other packages, so people could use mainline catpy along with extensions not necessarily supported by us built around an older version. A second CatmaidInstance wouldn't have access to the extensions loaded on the first, but the imports would be very quick the second time around.

Node interpolation depends on offset being multiple of resolution

Fix that.

Networkx 2.x

Old networkx is making some dependencies clash

IntEnums

For connector relation types, because I'm bored of writing it in dependent projects.

Should probably also fill out the catpy.image.TileSourceType one, for completeness.

deserialising JSON multiple times (no_verify switch for fetch?)

Currently, fetch deserialises the JSON to check for errors, even if the user wants the raw result. This could be a performance issue. Furthermore, the requests.Response object does not cache its .json() result, so when the user doesn't want the raw result, they deserialise the same (possibly large) JSON twice.

Should raw automatically apply no_verify? Is there even a need to have two separate switches?

Handle errors using new response codes

See catmaid/CATMAID@e8960b3

Instantiating ImageFetcher from CatMaid fails due to integer orientation

catpy on pypi

Would it be possible to create a catpy release on PyPi? Thanks.

NameResolver class

Sometimes people have the string name of a project, stack, or annotation, but not its ID, and that can be awkward to find out. I propose a subclass of CatmaidClientApplication which can resolve this.

In my own projects, I've written a few private methods to do this sort of thing, so that my public methods can accept either an integer ID or a string name and either will be resolved (with error handling for ambiguous names). Each of the resolve methods on the NameResolver could be decorated so that they short-circuit if given an integer, to save unnecessary HTTP calls, and a user's own CatmaidClientApplication can just run everything through a member NameResolver.

Docs: module index works locally but not on RTD

Reminder to myself to look into this Monday.

CoordinateTransformer improvements

• Methods for going directly between scaled and project coordinates
• kwarg to ignore offset for transformations involving project space (useful for converting the offset itself into stack space)
• Document use of stack_to_scaled in reverse and for project coordinates