bugs in new version

[alleninstitute]

bugs in new version

[dyf]

No -- it will just regenerate the manifest for you. Looks like we need to version our manifests as well to catch this sort of thing.

[kamdh]

Furthermore, first attempt to use new required StructureTree class:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-8-ee088d7228cc> in <module>()
----> 1 stree = mcc.get_structure_tree()

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/core/mouse_connectivity_cache.pyc in get_structure_tree(self, file_name)
    289         """
    290 
--> 291         file_name = self.get_cache_path(file_name, self.STRUCTURE_TREE_KEY)
    292 
    293         return OntologiesApi(self.api.api_url).get_structures_with_sets(

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/api/cache.pyc in get_cache_path(self, file_name, manifest_key, *args)
     48                 return file_name
     49             elif self.manifest:
---> 50                 return self.manifest.get_path(manifest_key, *args)
     51 
     52         return None

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/config/manifest.pyc in get_path(self, path_key, *args)
    210             Path with parent structure and substitutions applied.
    211         '''
--> 212         path_spec = self.path_info[path_key]['spec']
    213 
    214         if args is not None and len(args) != 0:

KeyError: 'STRUCTURE_TREE'

[kamdh]

This should clearly be a major version number change

[dyf]

Yikes, sorry about that. I see two problems:

1. the deprecated methods are failing now. Will take a look at that.
2. I think you need to remove your connectivity manifest.

[kamdh]

Thanks, David.
Will removing the manifest require re-downloading all data?

[dyf]

We just did a 0.13.0 release that included a manifest version checker. You should get an exception telling you to delete your manifest if you have an old manifest. Masks are stored in a different place now to avoid file name collision. If you don’t want to regenerate all of your masks, please move them from here:

manifest_dir/structure_masks/

To here (with the appropriate CCF/resolution strings):

manifest_dir/annotation/ccf_2016/resolution_25/structure_masks/

Can you take a look when you have time and see if you have any problems?

[kamdh]

Still doesn't work. I replaced the manifest and moved my structure masks as requested (working with 100 micron data). MouseConnectivityCache works. However, when I run isocortex_mask, _ = mcc.get_structure_mask(isocortex['id']) it gives:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-85f3c3cfa88a> in <module>()
      4 # get some info on the isocortex
      5 isocortex = ontology['Isocortex']
----> 6 isocortex_mask, _ = mcc.get_structure_mask(isocortex['id'])

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/core/mouse_connectivity_cache.pyc in get_structure_mask(self, structure_id, file_name, annotation_file_name)
    697         else:
    698             st = self.get_structure_tree()
--> 699             structure_ids = st.descendant_ids([structure_id])[0]
    700             annotation, _ = self.get_annotation_volume(annotation_file_name)
    701             mask = self.make_structure_mask(structure_ids, annotation)

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/core/simple_tree.pyc in descendant_ids(self, node_ids)
    217 
    218             current = [nid]
--> 219             children = self.child_ids([nid])[0]
    220 
    221             if children:

/home/kameron/local/anaconda2/lib/python2.7/site-packages/allensdk/core/simple_tree.pyc in child_ids(self, node_ids)
    152         '''
    153 
--> 154         return [ self._child_ids[nid] for nid in node_ids ]
    155 
    156 

/home/kameron/local/anaconda2/lib/python2.7/site-packages/pandas/core/generic.pyc in __hash__(self)
    829     def __hash__(self):
    830         raise TypeError('{0!r} objects are mutable, thus they cannot be'
--> 831                         ' hashed'.format(self.__class__.__name__))
    832 
    833     def __iter__(self):

TypeError: 'Series' objects are mutable, thus they cannot be hashed

[dyf]

Looks like the new internal classes don't work well with pd.Series coming in. Here's a workaround:

isocortex_mask, _ = mcc.get_structure_mask(isocortex['id'].values[0])

Let me see if there's a good way to make this work.

[dyf]

Single-element pd.Series can be safely cast to int/float/etc. I will do this to structure_id on the way in so we can maintain backwards compatibility. get_structure_mask will raise an exception if the structure_id can't be safely cast to an integer.

We're doing a 0.13.1 patch release this week to address this issue and a few others that have come up. Will let you know when it's out.

[dyf]

This is deployed.

[kamdh]

Issue continues to be a problem. I am trying to deploy onto a new server, so it is starting from scratch (no manifest, empty directory). This code fails:

import os
from allensdk.core.mouse_connectivity_cache import MouseConnectivityCache
import numpy as np
import h5py

drive_path = os.path.join(os.getenv('HOME'), 'work/allen/data/sdk_new_100')

# When downloading 3D connectivity data volumes, what resolution do you want (in microns)?  
# Options are: 10, 25, 50, 100
resolution_um=10

# The manifest file is a simple JSON file that keeps track of all of
# the data that has already been downloaded onto the hard drives.
# If you supply a relative path, it is assumed to be relative to your
# current working directory.
manifest_file = os.path.join(drive_path, "manifest.json")

mcc = MouseConnectivityCache(manifest_file=manifest_file, resolution=resolution_um)
ontology = mcc.get_ontology()
# get some info on the isocortex
isocortex = ontology['Isocortex']

# open up a pandas dataframe of all of the experiments
experiments = mcc.get_experiments(dataframe = True, 
                                  injection_structure_ids = [isocortex['id']], 
                                  cre = False)

But replacing the last bit with your workaround fixes it:

experiments = mcc.get_experiments(dataframe = True, 
                                  injection_structure_ids = [isocortex['id'].values[0]], 
                                  cre = False)

[dyf]

Thanks -- Nile's on it.

[dyf]

This should be fixed now. Let me know if you see anything else.