cryo-ET particle picking using triplet networks and metric learning
License: Mozilla Public License 2.0
Particle picking in Tomograms using triplet networks and metric learning
For installation instructions and tutorials please see our documentation
Dear Author,
I am interested in exploring the training process of TomoTwin, specifically regarding the model training and the associated hardware requirements.
Could you please provide insights into the following:
Training Process: Can you share details about the training pipeline for TomoTwin? I see that this part of the tutorial has not been added yet.
Resource Requirements: What are the GPU memory and RAM requirements for training the global TomoTwin model? Any information on the hardware setup used during your experiments would be highly valuable.
I appreciate your time and expertise in this matter. Thank you for your contribution to the field.
Hi guys,
I am trying to extract my selected references for picking with tomotwin, but I am getting the following error:
(tomotwin) bash-4.2$ tomotwin_tools.py extractref --tomo lmla02_ts_010.mrc_10.00Apx.mrc --coords particle_picked.coords --out tomotwin/reference/ --filename particles
Traceback (most recent call last):
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/bin/tomotwin_tools.py", line 8, in <module>
sys.exit(_main_())
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/tomotwin/tools_main.py", line 416, in _main_
tools = get_tool_list()
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/tomotwin/tools_main.py", line 389, in get_tool_list
from tomotwin.modules.tools.umap import UmapTool
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/tomotwin/modules/tools/umap.py", line 6, in <module>
import cuml
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cuml/__init__.py", line 17, in <module>
from cuml.common.base import Base
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cuml/common/__init__.py", line 17, in <module>
from cuml.common.array import CumlArray
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cuml/common/array.py", line 25, in <module>
from cudf import DataFrame
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cudf/__init__.py", line 71, in <module>
from cudf.io import (
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cudf/io/__init__.py", line 8, in <module>
from cudf.io.orc import read_orc, read_orc_metadata, to_orc
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cudf/io/orc.py", line 14, in <module>
from cudf.utils.metadata import ( # type: ignore
File "/lmb/home/tdendooven/mambaforge/envs/tomotwin/lib/python3.9/site-packages/cudf/utils/metadata/orc_column_statistics_pb2.py", line 7, in <module>
from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/lmb/home/tdendooven/.local/lib/python3.9/site-packages/google/protobuf/internal/__init__.py)
Could you perhaps let me know which is a good version of this package to use/pin the dependency to something valid? 🙂
Many thanks for your help!
Tom
Would folks be open to switching the headers from containing the entire license text to something more concise like SPDX-License-Identifier headers?
https://spdx.org/licenses/MPL-2.0.html
It would simplify to something like:
# SPDX-License-Identifier: MPL-2.0
This would make it a lot easier to navigate the source code.
Dear Thorsten,
I have been using your provided dataset and configuration scripts for training, and I've achieved an F1 score of 0.90 after 300 epochs, which is even higher than the reported value in the paper. I am curious to know if the dataset provided by you is complete, or if there have been additional optimizations in the methodology that could contribute to this difference?
Secondly, I am interested in experimenting with the ArcFace and SphereFace loss functions, and I was wondering if you could provide the configuration files for these scenarios.
Thank you for your time and assistance. I look forward to your insights.
Best regards,
Chi
I just want to make sure that we are not diverging too much.
Hi @thorstenwagner !
You got io folder and io.py in the same location, and you imported io folder as a module.
Traceback (most recent call last):
File "/home/gsharov/soft/miniconda3/envs/tomotwin-0.2.2b4/bin/tomotwin_embed.py", line 33, in <module>
sys.exit(load_entry_point('tomotwin-cryoet', 'console_scripts', 'tomotwin_embed.py')())
File "/home/gsharov/soft/scipion3/software/em/tomotwin-0.2.2b4/tomotwin/embed_main.py", line 506, in _main_
embeddings = volume_embedding(paths, embedor=embedor)
File "/home/gsharov/soft/scipion3/software/em/tomotwin-0.2.2b4/tomotwin/embed_main.py", line 407, in volume_embedding
volume_dataset = FileNameVolumeDataset(volumes=volume_pths, filereader=io.read_mrc)
AttributeError: module 'tomotwin.modules.common.io' has no attribute 'read_mrc'
Might give significant speedup with torch.compile:
Hi guys,
Tomotwin runs successfully on my tomograms and the results look quite promising! However, napari crashes when trying to open the .tloc files for some of the tomograms. The CC maps for those tomograms look fine so I'm not sure what the issue is. The error message is attached below.
Many thanks,
Tom
TypeError Traceback (most recent call last)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/components/viewer_model.py:1127, in ViewerModel._open_or_raise_error(self=Viewer(axes=AxesOverlay(visible=False, opacity=1...indings._transform_active_layer at 0x15eb200d0>}), paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], kwargs={}, layer_type=None, stack=False)
1126 try:
-> 1127 added = self._add_layers_with_plugins(
added = []
self = Viewer(axes=AxesOverlay(visible=False, opacity=1.0, order=1000000, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 472.546875, 325.4609375), zoom=0.9521956604229189, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(206.0, 453.64320579057835), scaled=True, size=1, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=2, ndisplay=2, last_used=0, range=((0.0, 2.0, 1.0), (0.0, 2.0, 1.0)), current_step=(0, 0), order=(0, 1), axis_labels=('1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[], scale_bar=ScaleBarOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, colored=False, color=<class 'numpy.ndarray'> (4,) float32, ticks=True, font_size=10.0, box=False, box_color=<class 'numpy.ndarray'> (4,) float32, unit=None), text_overlay=TextOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, color=<class 'numpy.ndarray'> (4,) float32, font_size=10.0, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x14977f850>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x14977f8b0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x14977f910>, transform=<napari.utils.transforms.transforms.Affine object at 0x14977f970>, allow_new_selection=True, selected_vertex=None)), help='', status='Ready', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=True, mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x15eabdea0>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x15eabe680>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x1468ffd90>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x15eabe8c0>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x15eb20160>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x15eb200d0>})
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
kwargs = {}
stack = False
plugin = 'napari-boxmanager'
layer_type = None
1128 paths,
1129 kwargs=kwargs,
1130 stack=stack,
1131 plugin=plugin,
1132 layer_type=layer_type,
1133 )
1134 # plugin failed
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/components/viewer_model.py:1217, in ViewerModel._add_layers_with_plugins(self=Viewer(axes=AxesOverlay(visible=False, opacity=1...indings._transform_active_layer at 0x15eb200d0>}), paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False, kwargs={}, plugin='napari-boxmanager', layer_type=None)
1216 assert len(paths) == 1
-> 1217 layer_data, hookimpl = read_data_with_plugins(
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
stack = False
plugin = 'napari-boxmanager'
1218 paths, plugin=plugin, stack=stack
1219 )
1221 # glean layer names from filename. These will be used as *fallback*
1222 # names, if the plugin does not return a name kwarg in their meta dict.
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/plugins/io.py:77, in read_data_with_plugins(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], plugin='napari-boxmanager', stack=False)
75 hookimpl: Optional[HookImplementation]
---> 77 res = _npe2.read(paths, plugin, stack=stack)
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
plugin = 'napari-boxmanager'
stack = False
_npe2 = <module 'napari.plugins._npe2' from '/Users/tdendooven/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/plugins/_npe2.py'>
78 if res is not None:
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/plugins/_npe2.py:55, in read(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], plugin='napari-boxmanager', stack=False)
54 try:
---> 55 layer_data, reader = io_utils.read_get_reader(
io_utils = <module 'npe2.io_utils' from '/Users/tdendooven/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/io_utils.py'>
npe1_path = '/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'
plugin = 'napari-boxmanager'
56 npe1_path, plugin_name=plugin
57 )
58 return layer_data, _FakeHookimpl(reader.plugin_name)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/io_utils.py:66, in read_get_reader(path='/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc', plugin_name='napari-boxmanager', stack=None)
65 new_path, new_stack = v1_to_v2(path)
---> 66 return _read(
new_path = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
new_stack = False
plugin_name = 'napari-boxmanager'
67 new_path, plugin_name=plugin_name, return_reader=True, stack=new_stack
68 )
69 else:
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/io_utils.py:163, in _read(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False, plugin_name='napari-boxmanager', return_reader=True, _pm=<npe2._plugin_manager.PluginManager object>)
161 if read_func is not None:
162 # if the reader function raises an exception here, we don't try to catch it
--> 163 if layer_data := read_func(paths, stack=stack):
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
read_func = <function to_napari at 0x16bba41f0>
stack = False
164 return (layer_data, rdr) if return_reader else layer_data
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/npe2/manifest/contributions/_readers.py:60, in ReaderContribution.exec.<locals>.npe1_compat(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False)
59 path = v2_to_v1(paths, stack)
---> 60 return callable_(path) # type: ignore
path = '/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'
callable_ = <function to_napari at 0x16a411240>
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/box_manager/io/tloc.py:61, in to_napari(path='lmla04_ts_013.mrc_10.00Apx_10.mrc')
60 if len(path) >= MAX_LAYER_NAME + 3:
---> 61 name = f"...{path[-{MAX_LAYER_NAME}:]}" # type: ignore
62 else:
TypeError: bad operand type for unary -: 'set'
The above exception was the direct cause of the following exception:
ReaderPluginError Traceback (most recent call last)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/qt_viewer.py:830, in QtViewer._qt_open(self=<napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0)>, filenames=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False, choose_plugin=False, plugin=None, layer_type=None, **kwargs={})
829 try:
--> 830 self.viewer.open(
self = <napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0) at 0x1498057c0>
self.viewer = Viewer(axes=AxesOverlay(visible=False, opacity=1.0, order=1000000, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 472.546875, 325.4609375), zoom=0.9521956604229189, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(206.0, 453.64320579057835), scaled=True, size=1, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=2, ndisplay=2, last_used=0, range=((0.0, 2.0, 1.0), (0.0, 2.0, 1.0)), current_step=(0, 0), order=(0, 1), axis_labels=('1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[], scale_bar=ScaleBarOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, colored=False, color=<class 'numpy.ndarray'> (4,) float32, ticks=True, font_size=10.0, box=False, box_color=<class 'numpy.ndarray'> (4,) float32, unit=None), text_overlay=TextOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, color=<class 'numpy.ndarray'> (4,) float32, font_size=10.0, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x14977f850>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x14977f8b0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x14977f910>, transform=<napari.utils.transforms.transforms.Affine object at 0x14977f970>, allow_new_selection=True, selected_vertex=None)), help='', status='Ready', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=True, mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x15eabdea0>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x15eabe680>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x1468ffd90>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x15eabe8c0>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x15eb20160>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x15eb200d0>})
filenames = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
stack = False
plugin = None
layer_type = None
kwargs = {}
831 filenames,
832 stack=stack,
833 plugin=plugin,
834 layer_type=layer_type,
835 **kwargs,
836 )
837 except ReaderPluginError as e:
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/components/viewer_model.py:1025, in ViewerModel.open(self=Viewer(axes=AxesOverlay(visible=False, opacity=1...indings._transform_active_layer at 0x15eb200d0>}), path=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False, plugin=None, layer_type=None, **kwargs={})
1023 # no plugin choice was made
1024 else:
-> 1025 layers = self._open_or_raise_error(
layers = <module 'napari.layers' from '/Users/tdendooven/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/layers/__init__.py'>
self = Viewer(axes=AxesOverlay(visible=False, opacity=1.0, order=1000000, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 472.546875, 325.4609375), zoom=0.9521956604229189, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(206.0, 453.64320579057835), scaled=True, size=1, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=2, ndisplay=2, last_used=0, range=((0.0, 2.0, 1.0), (0.0, 2.0, 1.0)), current_step=(0, 0), order=(0, 1), axis_labels=('1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[], scale_bar=ScaleBarOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, colored=False, color=<class 'numpy.ndarray'> (4,) float32, ticks=True, font_size=10.0, box=False, box_color=<class 'numpy.ndarray'> (4,) float32, unit=None), text_overlay=TextOverlay(visible=False, opacity=1.0, order=1000000, position=<CanvasPosition.BOTTOM_RIGHT: 'bottom_right'>, color=<class 'numpy.ndarray'> (4,) float32, font_size=10.0, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x14977f850>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x14977f8b0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x14977f910>, transform=<napari.utils.transforms.transforms.Affine object at 0x14977f970>, allow_new_selection=True, selected_vertex=None)), help='', status='Ready', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=True, mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x15eabdea0>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x15eabe680>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x1468ffd90>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x15eabe8c0>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x15eb20160>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x15eb200d0>})
_path = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
_stack = False
kwargs = {}
layer_type = None
1026 _path, kwargs, layer_type, _stack
1027 )
1028 added.extend(layers)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/components/viewer_model.py:1136, in ViewerModel._open_or_raise_error(self=Viewer(axes=AxesOverlay(visible=False, opacity=1...indings._transform_active_layer at 0x15eb200d0>}), paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], kwargs={}, layer_type=None, stack=False)
1135 except Exception as e:
-> 1136 raise ReaderPluginError(
trans = <napari.utils.translations.TranslationBundle object at 0x137a80a30>
plugin = 'napari-boxmanager'
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
1137 trans._(
1138 'Tried opening with {plugin}, but failed.',
1139 deferred=True,
1140 plugin=plugin,
1141 ),
1142 plugin,
1143 paths,
1144 ) from e
1145 # multiple plugins
1146 else:
ReaderPluginError: Tried opening with napari-boxmanager, but failed.
The above exception was the direct cause of the following exception:
ReaderPluginError Traceback (most recent call last)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/qt_viewer.py:1197, in QtViewer.dropEvent(self=<napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0)>, event=<PySide2.QtGui.QDropEvent object>)
1194 else:
1195 filenames.append(url.toString())
-> 1197 self._qt_open(
self = <napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0) at 0x1498057c0>
filenames = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
shift_down = <PySide2.QtCore.Qt.KeyboardModifiers object at 0x16be0af90>
alt_down = <PySide2.QtCore.Qt.KeyboardModifiers object at 0x16be0af70>
1198 filenames,
1199 stack=bool(shift_down),
1200 choose_plugin=bool(alt_down),
1201 )
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/qt_viewer.py:838, in QtViewer._qt_open(self=<napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0)>, filenames=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], stack=False, choose_plugin=False, plugin=None, layer_type=None, **kwargs={})
830 self.viewer.open(
831 filenames,
832 stack=stack,
(...)
835 **kwargs,
836 )
837 except ReaderPluginError as e:
--> 838 handle_gui_reading(
filenames = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
self = <napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0) at 0x1498057c0>
stack = False
layer_type = None
kwargs = {}
839 filenames,
840 self,
841 stack,
842 e.reader_plugin,
843 e,
844 layer_type=layer_type,
845 **kwargs,
846 )
847 except MultipleReaderError:
848 handle_gui_reading(filenames, self, stack, **kwargs)
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/dialogs/qt_reader_dialog.py:188, in handle_gui_reading(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], qt_viewer=<napari._qt.qt_viewer.QtViewer(0x7fda8e9e9df0)>, stack=False, plugin_name='napari-boxmanager', error=ReaderPluginError('Tried opening with napari-boxmanager, but failed.'), plugin_override=False, **kwargs={'layer_type': None})
161 """Present reader dialog to choose reader and open paths based on result.
162
163 This function is called whenever ViewerModel._open_or_get_error returns
(...)
185 Dictates whether checkbox to remember choice is unchecked by default
186 """
187 _path = paths[0]
--> 188 readers = prepare_remaining_readers(paths, plugin_name, error)
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
plugin_name = 'napari-boxmanager'
error = ReaderPluginError('Tried opening with napari-boxmanager, but failed.')
189 error_message = str(error) if error else ''
190 readerDialog = QtReaderDialog(
191 parent=qt_viewer,
192 pth=_path,
(...)
195 persist_checked=not plugin_override,
196 )
File ~/opt/miniconda3/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/dialogs/qt_reader_dialog.py:243, in prepare_remaining_readers(paths=['/Volumes/tdendooven-1/Flagella/hsperm_20221019_K...ate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'], plugin_name='napari-boxmanager', error=ReaderPluginError('Tried opening with napari-boxmanager, but failed.'))
241 # if there's no other readers left, raise the exception
242 if not readers and error:
--> 243 raise ReaderPluginError(
trans = <napari.utils.translations.TranslationBundle object at 0x137a80a30>
paths = ['/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc']
len(paths) > 1 = False
len(paths) = 1
paths[0] = '/Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc'
plugin_name = 'napari-boxmanager'
error = ReaderPluginError('Tried opening with napari-boxmanager, but failed.')
244 trans._(
245 "Tried to read {path_message} with plugin {plugin}, because it was associated with that file extension/because it is the only plugin capable of reading that path, but it gave an error. Try associating a different plugin or installing a different plugin for this kind of file.",
246 path_message=f"[{paths[0]}, ...]"
247 if len(paths) > 1
248 else paths[0],
249 plugin=plugin_name,
250 ),
251 plugin_name,
252 paths,
253 ) from error
255 return readers
ReaderPluginError: Tried to read /Volumes/tdendooven-1/Flagella/hsperm_20221019_Krios3/movies/reconstruction/tomotwin_all/locate_2/lmla04_ts_013.mrc_10.00Apx.mrc/located.tloc with plugin napari-boxmanager, because it was associated with that file extension/because it is the only plugin capable of reading that path, but it gave an error. Try associating a different plugin or installing a different plugin for this kind of file.```
I'm running napari Runs/000367_ProtImportTomograms/extra/emd_10439.mrc Runs/001103_ProtTomoTwinRefPicking/extra/emd_10439/locate/located.tloc -w napari-boxmanager with python 3.9
File "/home/gsharov/soft/miniconda3/envs/napari/lib/python3.9/site-packages/box_manager/__init__.py", line 6, in <module>
from ._reader import napari_get_reader
File "/home/gsharov/soft/miniconda3/envs/napari/lib/python3.9/site-packages/box_manager/_reader.py", line 7, in <module>
from . import io as bm_readers
File "/home/gsharov/soft/miniconda3/envs/napari/lib/python3.9/site-packages/box_manager/io/__init__.py", line 9, in <module>
from .interface import IOInterface
File "/home/gsharov/soft/miniconda3/envs/napari/lib/python3.9/site-packages/box_manager/io/interface.py", line 11, in <module>
class IOInterface(typing.Protocol):
File "/home/gsharov/soft/miniconda3/envs/napari/lib/python3.9/site-packages/box_manager/io/interface.py", line 14, in IOInterface
path: os.PathLike | list[os.PathLike],
TypeError: unsupported operand type(s) for |: 'ABCMeta' and 'types.GenericAlias'
Hello!
I am using TomoTwin to analyze my tomograms and I am in the step of Estimate UMAP manifold and Generate Embedding Mask.
When I run the command for that: tomotwin_tools.py umap -i .out/embed/tomo/tomo_embeddings.temb -o out/clustering
I receive the following error:
Traceback (most recent call last):
File "/mnt/nfs/clustersw/Debian/bullseye/python/3.11.4/bin/tomotwin_tools.py", line 5, in
from tomotwin.tools_main import main
File "/mnt/nfs/clustersw/Debian/bullseye/python/3.11.4/lib/python3.11/site-packages/tomotwin/tools_main.py", line 390, in
from tomotwin.modules.tools.umap import UmapTool
File "/mnt/nfs/clustersw/Debian/bullseye/python/3.11.4/lib/python3.11/site-packages/tomotwin/modules/tools/umap.py", line 7, in
import cuml
ModuleNotFoundError: No module named 'cuml'
We have updated TomoTwin to the latest version yesterday in our cluster, since this error was occurring before, but unfortunately it did not fix the issue.
Any suggestion to tackle this?
Thank you,
Francisco
tomotwin_tools.py will always load all tool dependencies regardless of which tools is used.
The effect of this is that if running on a cluster, you need to block a GPU node even if you don't use a gpu due to the cuml dependency for Umap, for example with calculating the median embedding or simply loading a help message from a tool.
Dear tomotwin team,
Thanks for developing this software.
I just installed to our workstation (centos) and following your tutorial.
In this step:
tomotwin_tools.py extractref --tomo TS_14.mrc --coords coords/reference.coords --out reference/ --filename protein_a
I got an error message of this:
ImportError: /data/miniconda3/envs/tomotwin/lib/python3.10/site-packages/cuml/internals/../../../.././libcublas.so.11: symbol cublasLtHSHMatmulAlgoInit, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference
When I went actual place of /data/miniconda3/envs/tomotwin/lib/, I couldn't list anything using "ls"command with the error message as this:
ls: relocation error: /lib64/libacl.so.1: symbol getxattr, version ATTR_1.0 not defined in file libattr.so.1 with link time reference
Could you please help me in this please?
Thanks a lot!
Joy
In issue #58 @azazellochg run into an issue that ldconfig was not in the path. This also happend on some other machines for me as well:
https://discuss.pytorch.org/t/dynamo-exceptions-with-distributeddataprallel-compile/186768/3
We should add a check if ldconfig is in the path before running tomotwin
With the last release we updated rapids to 23.10. Now I noticed that with 23.10 calculating a umap is much slower. With 23.12 its fast again.
Here are some stats:
23.10: 64.05s / it
23.12: 15s / it
So its more then 4x faster.
Hi Thorsten,
I found tomotwin_locate.py tries to allocate more memory than the capacity of our workstation and it failed to run with the following error:
OSError: [Errno 12] Cannot allocate memory
Is there an argument tag that I could add to make it use less memory and let it run?
Best wishes,
Joy
To reduce divergence
With the upcoming napari-tomotwin, the clustering workflow becomes more streamlined and powerful. This needs to be reflected in the documentation before I can release it.
Some points that needs to be reflected in the documentation:
This would allow multi gpu estimation of umaps.
https://docs.rapids.ai/api/cuml/stable/api/#cuml.dask.manifold.UMAP
This is because of a bug in napari:
napari/napari#6077
I changed the install instructions back to 0.4.17 until a solution is found:
#36
Hi @thorstenwagner
I'm checking the tutorial with the latest version. All steps are working except mask creation.
source /public/EM/Scipion/conda.rc&& conda activate tomotwin-0.6.1 && tomotwin_tools.py embedding_mask -i tomo.mrc -o ../extra/ fails on a single rtx2080ti GPU with:
00001: Traceback (most recent call last):
00002: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/bin/tomotwin_tools.py", line 33, in <module>
00003: sys.exit(load_entry_point('tomotwin-cryoet', 'console_scripts', 'tomotwin_tools.py')())
00004: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/bin/tomotwin_tools.py", line 25, in importlib_load_entry_point
00005: return next(matches).load()
00006: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
00007: module = import_module(match.group('module'))
00008: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/importlib/__init__.py", line 126, in import_module
00009: return _bootstrap._gcd_import(name[level:], package, level)
00010: File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
00011: File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
00012: File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
00013: File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
00014: File "<frozen importlib._bootstrap_external>", line 883, in exec_module
00015: File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
00016: File "/public/EM/Scipion/scipion-dev/software/em/tomotwin-0.6.1/tomotwin/tools_main.py", line 390, in <module>
00017: from tomotwin.modules.tools.umap import UmapTool
00018: File "/public/EM/Scipion/scipion-dev/software/em/tomotwin-0.6.1/tomotwin/modules/tools/umap.py", line 7, in <module>
00019: import cuml
00020: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/__init__.py", line 17, in <module>
00021: from cuml.internals.base import Base, UniversalBase
00022: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/__init__.py", line 17, in <module>
00023: from cuml.internals.base_helpers import BaseMetaClass, _tags_class_and_instance
00024: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/base_helpers.py", line 20, in <module>
00025: from cuml.internals.api_decorators import (
00026: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 24, in <module>
00027: from cuml.internals import input_utils as iu
00028: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/input_utils.py", line 19, in <module>
00029: from cuml.internals.array import CumlArray
00030: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/array.py", line 22, in <module>
00031: from cuml.internals.global_settings import GlobalSettings
00032: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/global_settings.py", line 19, in <module>
00033: from cuml.internals.available_devices import is_cuda_available
00034: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/available_devices.py", line 17, in <module>
00035: from cuml.internals.safe_imports import gpu_only_import_from, UnavailableError
00036: File "/public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/safe_imports.py", line 21, in <module>
00037: from cuml.internals import logger
00038: ImportError: /public/EM/Scipion/miniconda3/envs/tomotwin-0.6.1/lib/python3.10/site-packages/cuml/internals/../../../.././libcublas.so.11: undefined symbol: cublasLt_for_cublas_HSS, version libcublasLt.so.11
I'm happy to provide more information if required.
Dear Devs,
I am using tomotwin as part of napari 0.4.18. I am running in clustering mode. The pixel size of my tomogram is 11.06, note that I did try to use e2proc3d with fourier shrink as written in the tutorial. It appeared to execute the command, but when I use Imod's "header" Tool to examine the output I see that the pixel size is unchanged.
Nevertheless, I continued with the tomo at 11.06. It appeared to run through on the embedding without issue. Then it runs through the second part (Estimate UMAP manifold and Generate Embedding Mask) without issue either.
output:
Read data
Prepare data
Fit umap on 400000 samples
Transform complete dataset in 8 chunks with a chunksize of ~447678
Transform:
Write embeedings to disk
Write umap model to disk
Calculate label mask and write it to disk
Create embedding mask
Done
I then open napari 0.4.18 along with the tomogram. As per the instructions I use the plugin to open and point to the UMAP path. "...embeddings.tumap". I click LOAD. I then get the error "label mask is not specified".
I can see tomotwin label mask in the upper right. It is a dropdown but it is greyed out so I cannot select anything. The field is currently empty.
In the output folder (TOMOTWIN/tomograms/out/clustering) I have the following files:
ts_001_embeddings.tumap
ts_001_embeddings_umap_model.pkl
ts_001_embeddings_label_mask.mrci
thanks in advance!
Jesse
Hey TomoTwins!
@cmccaffe and I have been stumbling upon a strange error (see full error message below) when trying to embed a tomo:
tomotwin_embed.py tomogram -m /scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_model_p120_052022_loss.pth -v /scicore/home/engel0006/GROUP/pool-visprot/Caitie/TomoTwin/Tkiv/tomo9/9.rec -b 256 -o /scicore/home/engel0006/GROUP/pool-visprot/Caitie/TomoTwin/Tkiv/tomo9/embed/tomo/ -s 2
This has been happening since updating to v0.7.0. We believe this might be due to some recent update in pytorch, because that has broken membrain-seg too. The error messages seem cryptic, but according to this thread it could be a problem with the loss function:
https://stackoverflow.com/questions/68277801/extracting-meaningful-error-message-from-runtimeerror-cuda-error-device-side
I'm mentioning this because in membrain-seg we got error messages that mentioned the loss function explicitly, but I'm not sure if it's related. Pinging @LorenzLamm here because he managed to fix membrain-seg so maybe he has some advice.
Any ideas?
Thanks a lot!
Latest version of TomoTwin is installed :-)
Found 1 GPU(s). Start DDP + Compiling.
reading /scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_model_p120_052022_loss.pth
Model config:
{'identifier': 'SiameseNet', 'network_config': {'output_channels': 32, 'dropout': 0.2, 'repeat_layers': 0, 'norm_name': 'GroupNorm', 'norm_kwargs': {'num_groups': 64, 'num_channels': 1024}, 'gem_pooling_p': 0}, 'train_config': {'loss': 'TripletLoss', 'tl_margin': 0.539, 'miner': True, 'miner_margin': 0.734, 'learning_rate': 5.945e-05, 'optimizer': 'Adam', 'weight_decay': 0, 'batchsize': 35, 'patience': 50, 'aug_train_shift_distance': 2}, 'distance': 'COSINE', 'window_size': (37, 37, 37)}
Traceback (most recent call last):
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/bin/tomotwin_embed.py", line 8, in <module>
sys.exit(_main_())
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 630, in _main_
start(config)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 618, in start
run_distr(config, world_size)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 604, in run_distr
mp.spawn(
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
while not context.join():
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 163, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1084, in _record
static_outputs = model(inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 401, in _run_from_cache
return compiled_graph.compiled_artifact(inputs)
File "/scratch/torchinductor_diogori/ou/cou7qwba4awtz6sdrugswxxjpfrrzyvzysk4ckcmf6ftq3a52drp.py", line 481, in call
buf2 = extern_kernels.convolution(buf0, buf1, stride=(1, 1, 1), padding=(0, 0, 0), dilation=(1, 1, 1), transposed=False, output_padding=(0, 0, 0), groups=1, bias=None)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
fn(i, *args)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 584, in run
embed_tomogram(tomo, embedor, conf, window_size, mask)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 512, in embed_tomogram
embeddings = sliding_window_embedding(tomo=tomo, boxer=boxer, embedor=embedor)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/embed_main.py", line 409, in sliding_window_embedding
embeddings = embedor.embed(volume_data=boxes)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/modules/inference/embedor.py", line 600, in embed
subvolume = self.model.forward(subvolume).type(torch.HalfTensor)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1519, in forward
else self._run_ddp_forward(*inputs, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1355, in _run_ddp_forward
return self.module(*inputs, **kwargs) # type: ignore[index]
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
return fn(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/tomotwin/modules/networks/SiameseNet3D.py", line 522, in forward
def forward(self, inputtensor):
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
return fn(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/fx/graph_module.py", line 678, in call_wrapped
return self._wrapped_call(self, *args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/fx/graph_module.py", line 284, in __call__
raise e
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/fx/graph_module.py", line 274, in __call__
return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "<eval_with_key>.214", line 6, in forward
submod_0 = self.compiled_submod_0(l_inputtensor_); l_inputtensor_ = None
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 335, in forward
x = self.submod(*args)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
return fn(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
return fn(*args, **kwargs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3905, in forward
return compiled_fn(full_args)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1482, in g
return f(*args)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2533, in runtime_wrapper
all_outs = call_func_with_args(
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1506, in call_func_with_args
out = normalize_as_list(f(args))
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1594, in rng_functionalization_wrapper
return compiled_fw(args)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 374, in __call__
return self.get_current_callable()(inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 680, in run
return compiled_fn(new_inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 368, in deferred_cudagraphify
return fn(inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 628, in run
return model(new_inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1741, in run
out = self._run(new_inputs, function_id)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1815, in _run
return self.record_function(new_inputs, function_id)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1846, in record_function
node = CUDAGraphNode(
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 892, in __init__
] = self._record(wrapped_function.model, recording_inputs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1076, in _record
with preserve_rng_state(), torch.cuda.device(
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/cuda/graphs.py", line 197, in __exit__
self.cuda_graph.capture_end()
File "/scicore/home/engel0006/GROUP/pool-engel/soft/tomotwin/tomotwin_env/lib/python3.10/site-packages/torch/cuda/graphs.py", line 88, in capture_end
super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
@rdrighetto & co from @CellArchLab made the suggestion to allow the specification of ROI for embedding a tomogram using a binary mask. Should be straight forward to implement.
Hi @thorstenwagner ! I'm looking at https://tomotwin-cryoet.readthedocs.io/en/latest/tutorials/tutorials_overview.html#tutorial-1-reference-based-particle-picking. I understand this repo is still work in progress, but I got a few questions:
This can be in particular useful when using the clustering mode.
Hi team,
Thank you for developing tomotwin.
I’ve got a following warning/errors after the third step of embedding my tomogram:
/data/miniconda3/envs/tomotwin/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:347: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
....
Calculate embeddings (1): 6%|\u2588\u2589 | 56672/889366 [1:18:40<19:13:39, 12.03it/s]Traceback (most recent call last):
(then it stopped with these error messages:)
...
File "/data/miniconda3/envs/tomotwin/lib/python3.10/site-packages/torch/multiprocessing/reductions.py", line 360, in rebuild_storage_fd
storage = cls._new_shared_fd_cpu(fd, size)
RuntimeError: unable to mmap 512 bytes from file : Cannot allocate memory (12)
.....
/data/miniconda3/envs/tomotwin/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 21 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Do you think you could help in this please?
Thanks a lot,
Joy
Finally I found time to wrap my head around dask and to speed-up the locate command with it.
PR #89 contains the implementation. For my example, the locate time was reduced from 30 minutes to 3 minutes.
The @rdrighetto & co from @CellArchLab made the suggestion to add a tool that allows the merging of picks of a group of targets.
I think that suggestion came from Manon.
@rdrighetto Does she has a GitHub account to that I can keep her posted?
Hi there,
First of all, TomoTwin looks super impressive and useful and I look forward to trying it out.
Unfortunately, I'm encountering a couple of issues that I was hoping someone could help me with. My apologies if this is not the right place, or if these issues turn out to be super specific to my setup - I'm using a server PC to which I have limited access in terms of rights, so that may be part of the problem (but it doesn't look like this is the case just yet). I hope that this post can be mutually helpful, but if it turns out that these are indeed probably local problems I'll mark this post solved and see if our sysadmin can help me out instead.
I'm on Debian GNU/Linux 11 with kernel 5.10.0-22-amd64 and eight Nvidia GeForce RTX 3080's. TomoTwin was installed exactly as described here: https://tomotwin-cryoet.readthedocs.io/en/stable/installation.html
Problem 1
This one may be normal but just wanted to check: embedding one tomogram (10apix, 1.1 Gb, 1584 x 1125 pixels and 157 slices) takes seven hours when using two GPUs. Is this normal?
Log output:
(tomotwin) ...> CUDA_VISIBLE_DEVICES=0,1 tomotwin_embed.py tomogram -m tomotwin_model_p120_052022_loss.pth -v ali_tomo_1_corrected__rec_10A.mrc -b 256 -o out/embed/tomo/ -s 2
Latest version of TomoTwin is installed :-)
reading tomotwin_model_p120_052022_loss.pth
Model config:
{'identifier': 'SiameseNet', 'network_config': {'output_channels': 32, 'dropout': 0.2, 'repeat_layers': 0, 'norm_name': 'GroupNorm', 'norm_kwargs': {'num_groups': 64, 'num_channels': 1024}, 'gem_pooling_p': 0}, 'train_config': {'loss': 'TripletLoss', 'tl_margin': 0.539, 'miner': True, 'miner_margin': 0.734, 'learning_rate': 5.945e-05, 'optimizer': 'Adam', 'weight_decay': 0, 'batchsize': 35, 'patience': 50, 'aug_train_shift_distance': 2}, 'distance': 'COSINE', 'window_size': (37, 37, 37)}
Embeddings have shape: (25731630, 35)
Wrote embeddings to disk to out/embed/tomo/ali_tomo_1_corrected__rec_10A_embeddings.temb
Done.
Problem 2
After embedding, the generation of the umap crashes right upon starting due to some problem with numba.
(tomotwin) ...> tomotwin_tools.py umap -i out/embed/tomo/tomo_embedding.temb -o tomotwin
Traceback (most recent call last):
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/bin/tomotwin_tools.py", line 5, in <module>
from tomotwin.tools_main import _main_
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/tomotwin/tools_main.py", line 390, in <module>
from tomotwin.modules.tools.umap import UmapTool
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/tomotwin/modules/tools/umap.py", line 7, in <module>
import cuml
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/cuml/__init__.py", line 17, in <module>
from cuml.common.base import Base
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/cuml/common/__init__.py", line 17, in <module>
from cuml.common.array import CumlArray
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/cuml/common/array.py", line 25, in <module>
from cudf import DataFrame
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/cudf/__init__.py", line 5, in <module>
validate_setup()
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/cudf/utils/gpu_utils.py", line 20, in validate_setup
from rmm._cuda.gpu import (
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/rmm/__init__.py", line 16, in <module>
from rmm import mr
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/rmm/mr.py", line 14, in <module>
from rmm._lib.memory_resource import (
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/rmm/_lib/__init__.py", line 15, in <module>
from .device_buffer import DeviceBuffer
File "device_buffer.pyx", line 1, in init rmm._lib.device_buffer
File "stream.pyx", line 26, in init rmm._cuda.stream
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/numba/__init__.py", line 42, in <module>
from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/numba/np/ufunc/__init__.py", line 3, in <module>
from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
File "/scisoft/x86-64/anaconda/2021.05/envs/tomotwin/lib/python3.9/site-packages/numba/np/ufunc/decorators.py", line 3, in <module>
from numba.np.ufunc import _internal
SystemError: initialization of _internal failed without raising an exception
Log output:
Problem 3
In the napari-tomotwin environment, napari won't launch due to an issue with pyqt (some dependency missing in that environment I guess?)
(napari-tomotwin) ...> napari_boxmanager ali_tomo_1_corrected__rec_10A.mrc
Traceback (most recent call last):
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/__init__.py", line 9, in <module>
from qtpy import API_NAME, QtCore
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/qtpy/__init__.py", line 259, in <module>
raise QtBindingsNotFoundError from None
qtpy.QtBindingsNotFoundError: No Qt bindings could be found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/bin/napari", line 8, in <module>
sys.exit(main())
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/__main__.py", line 561, in main
_run()
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/__main__.py", line 218, in _run
from napari import Viewer, run
File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_lazy.py", line 48, in __getattr__
submod = import_module(
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_event_loop.py", line 2, in <module>
from ._qt.qt_event_loop import gui_qt, run
File "/scisoft/x86-64/anaconda/2021.05/envs/napari-tomotwin/lib/python3.10/site-packages/napari/_qt/__init__.py", line 12, in <module>
raise type(e)(
TypeError: QtBindingsNotFoundError.__init__() takes 1 positional argument but 2 were given
Log output:
Any help much appreciated, thanks in advance!
Hi,
While installing tomotwin as directed in the documentation, i.e., using the yml file, it seems mamba installs pytorch from conda-forge which by default is the non-GPU version. This is because conda-forge is on the top of the channels list. If I change the order of the channels, putting conda-forge below pytorch and nvidia, that correctly installs the CUDA enabled pytorch. For example:
name: tomotwin
channels:
- nvidia
- pytorch
- rapidsai
- conda-forge
- defaults
I discovered this while building a container for tomotwin in a completely vanilla environment (no ~/.condarc for example).
I installed napari_boxmanager at my centos without any error and warning.
However, after I sshed into my Centos with -X (even I confirmed with xclock
), I can't open mrc.
For example, when I tried
I see
.
When I tried to open mrc by GUI (file -> open file
), I see
On the other hand, the same napari-tomotwin installation trial on my macOS opened mrc file well.
Hi,
I'm trying to install tomotwin but encountered an error, when ran the command 'mamba create xxxx', I got the following error:
Encountered problems while solving:
This happens no matter I use cuda-10.1 or cuda-11.8 in my shell (checked by nvcc --version), Any idea how to solve this problem?
Thank you very much in advance!
Sincerely,
Linhua Tai
Running command: source /public/EM/Scipion/conda.rc&& conda activate napari-0.4.4 && napari_boxmanager Runs/000002_ProtImportTomograms/extra/tomo.mrc Runs/000247_ProtTomoTwinRefPicking/extra/tomo/locate/located.tloc
Traceback (most recent call last):
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/bin/napari", line 10, in <module>
sys.exit(main())
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/__main__.py", line 561, in main
_run()
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/__main__.py", line 341, in _run
viewer._window._qt_viewer._qt_open(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/_qt/qt_viewer.py", line 830, in _qt_open
self.viewer.open(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1014, in open
self._add_layers_with_plugins(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/components/viewer_model.py", line 1216, in _add_layers_with_plugins
layer_data, hookimpl = read_data_with_plugins(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/plugins/io.py", line 77, in read_data_with_plugins
res = _npe2.read(paths, plugin, stack=stack)
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/napari/plugins/_npe2.py", line 55, in read
layer_data, reader = io_utils.read_get_reader(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/npe2/io_utils.py", line 66, in read_get_reader
return _read(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/npe2/io_utils.py", line 172, in _read
if layer_data := read_func(paths, stack=stack):
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/npe2/manifest/contributions/_readers.py", line 60, in npe1_compat
return callable_(path) # type: ignore
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/box_manager/io/mrc.py", line 49, in to_napari
layer_data = to_napari_image(
File "/public/EM/Scipion/miniconda3/envs/napari-0.4.4/lib/python3.10/site-packages/box_manager/io/io_utils.py", line 265, in to_napari_image
metadata["pixel_spacing"] = get_pixel_size(path[0])
IndexError: list index out of range
Here is the tloc file:
https://mrclmbcamacuk.sharepoint.com/:u:/s/EMFacility/EfksdZpb-stJsukM2v4A9mUB1Xk_0tjaUPcqp4xNPdfZUw?e=vCfFkz
Is it corrupted?
During training, I do three independent forward passes through the network for the anchors, the positive and the negative volumes respectively (see
).It is probably more efficient to concatenate them into one batch and do it in one pass.
I use mamba 0.11.2 and CentOS Linux release 7.8.2003
I tried
mamba env create -n tomotwin -f https://raw.githubusercontent.com/MPI-Dortmund/tomotwin-cryoet/main/conda_env_tomotwin.yml
according to install instruction.
However, I see
[kimd999@deception04 ~]$ mamba env create -n tomotwin -f https://raw.githubusercontent.com/MPI-Dortmund/tomotwin-cryoet/main/conda_env_tomotwin.yml
usage: mamba [-h] [--version] [--slow SLOW] [--enable-coverage] [--coverage-file COVERAGE_FILE] [--format FORMAT] [--no-color]
[--tags TAGS]
[specs ...]
mamba: error: unrecognized arguments: -n tomotwin
conda env create -n tomotwin -f https://raw.githubusercontent.com/MPI-Dortmund/tomotwin-cryoet/main/conda_env_tomotwin.yml
takes more than few hrs now, and it is still trying to solve conflicts.
Maybe tomotwin can be installed at Ubuntu only?
@rdrighetto made me aware to provide update instructions for TomoTwin.
I think the best way to do it is to use conda env update . Need test it a bit.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.