lcsr-ciis/ambf_util_slicer_plugin

some utilities for quickly converting volumes, markups from 3DSlicer into format accepted by AMBF simulator

AMBF_Utils Module for 3D Slicer

This module is a collection of utilities for exporting volumes and markers from 3D Slicer to a format that can be read by the AMBF Simulator and various volumetric plugins for the AMBF Simulator.

Primarily, this module is expected to be used with continuum-manipulator-volumetric-drilling-plugin (https://github.com/htp2/continuum-manip-volumetric-drilling-plugin.git) and volumetric-drilling-plugin (https://github.com/LCSR-SICKKIDS/volumetric_drilling)

I would be remiss to not mention the work of Adnan Munawar et al., whose scripts in the referenced volumetric-drilling-plugin were used as a starting point for this module.

Tested with Slicer 5.2.1. Binary install is fine, no need for compilation from source.

Video Demonstrating Use:

abmf_slicer_demo.mp4

Exporting LabelMaps (Slicer Segmentations) To AMBF Volumes:

Volumes will be exported into a folder of png files sliced in the superior/inferior plane, which are then loaded into AMBF as a VOLUME using the images method in the ADF file. Future versions of AMBF likely will directly load e.g. nrrd files, and this plugin may be updated to include that functionality as well.

Along with the volume <volume_name>, a static body <volume_name>_anatomical_origin will be configured to load into AMBF. This is the Cartesian origin of the image (i.e. the `x=y=z=0' point on the image), which is typically somewhat arbitrarily defined by the imager. The volume will parent to this body to appear in the correct location in AMBF. This allows you to do all registrations and express all features relative to the same image coordinate system. The orientation of the volume within AMBF is such that its x,y,z axes correspond to the LPS axes of the image.

To assist with understanding coordinate systems in 3DSlicer, see the following link: https://www.slicer.org/wiki/Coordinate_systems

You may export a volume as a "grayscale" image, or it will be exported using the current color map of the segmentation.

Using Slicer's terminology, an AMBF "volume" really is generally a segmentation made from a volume. For the purposes of this plugin, you must must use a labelmap (which you can create by segmenting a volume in 3D Slicer). From that segmentation, you can right click on the node and "convert segmentation to labelmap"

Automatically Generate ADF files for AMBF Simulator

You can generate the accompanying AMBF ADF yaml file by checking the "Generate AMBF yaml" checkbox. This will generate a yaml file with the same name as the volume in the output directory which will have the size information, etc. setup so that the AMBF Simulator can read the volume.

In addition to the volume, this configuration file will also contain a body with [volume_name]_anatomical_origin which will be located at the anatomical or "space" origin of the volume. This is the "world" origin according to the imager the image was taken with (e.g. CT scanner, MRI machine, etc.). Usually it is located somewhere outside the image itself. For example, in 3D Slicer your markup coordinates are defined relative to this anatomical origin. The only difference is that the anatomical origin will be defined using the LPS = (x,y,z) convention in AMBF, whereas it is RAS = (x,y,z) in 3D slicer.

You can see this file within the Slicer plugin using the "YAML Output" Tab. If you press "Load/Refresh YAML File", it will update that text from the file into the text box in the plugin. Then, if you want you can make any hard-coded edits and save that out to the file using the "Overwrite YAML file". Note: By design, if you "Export LabelMap to PNGs for AMBF", the yaml file will be overridden to the default values given your data, so do any overwriting last if need be. This editing can also be done in your favorite text editor, this just allows you to do it in the same window.

If you want to regenerate the yaml file (e.g. if you changed a setting like the volume name or scale), and do not want to regenerate all of the image slices again, you may disable this with the "Generate Image Slices" checkbox.

Applying a non-identity offset to the volume

When you select a LabelMap, a transform called "AMBF_Pose" will be generated in the "Transforms" module. This will correspond to the initial pose that your volume will have (or more accurately that the anatomical_origin will have) in AMBF (i.e. what goes in the ADF yaml file). This is how you can specify your volume to have a non-identity pose. The anatomical origin's position will be updated in the ADF file accordingly, the volume is parented to this origin so it will move accordingly. Several other Transforms will appear, all having internal relations within 3DSlicer. This is done to allow the "AMBF_Pose" transform to act like an LPS transform (perhaps a bit controversial, but this mean the matrix that appears within 3D slicer will match that which appears in the ADF file).

A Note on the [volume_name]_anatomical_origin body

By default, this body is static in AMBF, but you can move it via the Python and ROS clients, or with your own plugin as you would any other body. It is imporant to note that you should never move the volume itself, ALWAYS move the anatomical_origin otherwise the relationship will not be maintained.

Exporting Markups

NOTE: This is left in for convenience, but since there is now direct parenting of the volume to the anatomical origin, you likely will have a good experience just use the builtin saving of Markups within 3D Slicer and parse them yourself.

Markups (for now markup point lists and markup curves are supported) will be written to a csv file, accounting for any scaling you choose to set.

Markups in 3D Slicer are given in "anatomical" / "world" coordinates (i.e. relative to some world origin defined by the imager). This tool will convert them into SI units, then scale them per the AMBF scale that is set, and finally convert them into the LPS coordinate convention that is used in AMBF. These values will again be in "anatomical" coordinates, i.e. relative to the "[volume_name]_anatomical_origin" body in the ADF file.

NOTE: You must make your markups at the volume's original location, not on the labelmap after it was moved to an inital AMBF pose

Example Usage of features

• Pick a volume of your choice, and make some segmentation on it (e.g. segment out bone from a CT volume)
• Convert the segmentation to a labelmap
• [Optional]: Change AMBF_Pose transform to show the labelmap volume in the position you would like relative to the AMBF origin
• Use this module to export the labelmap to a folder of png files, with a yaml file for the AMBF Simulator
• [Optional]: Draw a curve on the volume, and export it to a csv file
• Load the volume into the AMBF simulator (e.g. using the volumetric-drilling-plugin). If you generated a curve on the volume, use the ambf_trace_plugin (https://github.com/htp2/ambf_trace_plugin) to load the curve into the AMBF simulator using --csv_filename_static_traces <path_to_csv_file> and --static_trace_rel_body_name <volume_name>_anatomical_origin

Setup Instructions

After cloning this module:

• Open Slicer
• In the 'Welcome to Slicer' Module, click 'Customize Slicer'
• Press the Modules tab
• Under 'Paths' select 'Add'
• Select folder that contains this README document
• Restart Slicer as prompted
• The module should appear under the group name AMBF

ambf_util_slicer_plugin's Issues

"Export LabelMap to PNG for AMBF" does not generate file

Hello!

I have the plugin setup per the instructions. However, when I click the "Export LabelMap to PNG for AMBF" button the file does not generate in the selected directory. Do you know why this might be? Thanks!

Error: FileNotFoundError and Data Size does not match dimensions

Hi Henry,

I think there may be some confusion in the workflow of saving the ADF file. I completed all the steps as I previously did but somehow ended up with these errors.

1. FileNotFoundError
2. Data Size does not match dimensions

This is output in the Python console.

This is what my configuration looks like:

I checked and I do have the directory ~/Documents/adf_volume.

Another thing to note is that even though I only clicked Export LabelMap to PNGs for AMBF, the Python console shows the method onRefreshYamlButton was called. Is this the intended behavior? If so, would it not cause issues if a file does not already exist?

We should mention the required/tested 3D Slicer version

We should specify what 3D Slicer version is required for this addon. I tried with version 4.11.20210226 and got the following error with this line Resources.slicer_helper import slicer_helper as sh. I commented it out as sh was not being used in the rest of the script. Then I got the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/adnanmunawar/Slicer-4.11.20210226-linux-amd64/lib/Python/lib/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/adnanmunawar/ambf_util_slicer_plugin/AMBF_utils.py", line 14, in <module>
    import PIL.Image as Image
  File "/home/adnanmunawar/Slicer-4.11.20210226-linux-amd64/lib/Python/lib/python3.6/site-packages/PIL/Image.py", line 94, in <module>
    from . import _imaging as core
ImportError: /home/adnanmunawar/Slicer-4.11.20210226-linux-amd64/lib/Python/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned

Incorrect default parameters (Images Path & Shaders) in the generated ADF file

This is an example output from a volume I tested. Some fields are set to default (commented below) which will require manual editing for the volume to show up.

# AMBF Version: (0.1)
bodies:
- BODY volume_anatomical_origin
joints: []
volumes: [VOLUME volume]
high resolution path: meshes/high_res/
low resolution path: meshes/low_res/
ignore inter-collision: true
namespace: /ambf/env/
VOLUME volume:
  name: volume
  location:
    position: {x: 0.0 , y: 0.0, z: 0.0}
    orientation: {r: 0.0, p: 0.0, y: 0.0}
  scale: 1.0
  dimensions: {x: 0.512, y: 0.512, z: 0.505}
  images:
    path: ../resources/volume/    # <-- Default path and not where the PNGs were actually saved.
    prefix: plane000
    format: png
    count: 512
   shaders:    # <-- Default shader path and files, which isn't written by the module and thus the volume won't load. Can omit the entire field to use default rendering shaders or write them out.
     path: ../shaders/volume/
     vertex: shader.vs
     fragment: shader.fs
BODY volume_anatomical_origin: 
  name: volume_anatomical_origin
  mass: 0.0
  location:
      position:
        x: 0.029324594584528726
        y: 0.15771592665049528
        z: 0.17275107928340605
      orientation:
        r: 0.0
        p: 0.0
        y: 0.0

The Generated Volume is flipped by PI along the AMBF Z axis

Hi Henry,

I was testing a volume that I got from Jesse and found that the generated volume in AMBF is flipped by PI along the z-axis in AMBF. If I set the orientation of the "Volume" to be PI along the z-axis in the ADF the results look O.K.

Incorrect rotation directly generated by the plugin:

Corresponding ADF file:

# AMBF Version: (0.1)
bodies:
- BODY volume_anatomical_origin
joints: []
volumes: [VOLUME volume]
high resolution path: meshes/high_res/
low resolution path: meshes/low_res/
ignore inter-collision: true
namespace: /ambf/env/
VOLUME volume:
  name: volume
  parent: volume_anatomical_origin
  location:
    position: {x: 0.036698876469970686 , y: 0.01414718772534178, z: 0.027546614014801026}
    orientation: {r: 0.0, p: 0.0, y: 0.0}
  scale: 1.0
  dimensions: {x: 0.052761648, y: 0.061956624, z: 0.061518768}
  images:
    path: volume/
    prefix: plane000
    format: png
    count: 281
BODY volume_anatomical_origin: # This is a dummy body that can be used to represent the anatomical origin for easy reference
  name: volume_anatomical_origin
  mass: 0.0
  mesh: Cube.stl
  scale: 1.0
  location:
      position:
        x: 0.0
        y: 0.0
        z: 0.0
      orientation:
        r: 0.0
        p: 0.0
        y: 0.0

Manually adding a PI rotation to the "Volume" in the ADF:

Corresponding ADF file

# AMBF Version: (0.1)
bodies:
- BODY volume_anatomical_origin
joints: []
volumes: [VOLUME volume]
high resolution path: meshes/high_res/
low resolution path: meshes/low_res/
ignore inter-collision: true
namespace: /ambf/env/
VOLUME volume:
  name: volume
  parent: volume_anatomical_origin
  location:
    position: {x: 0.036698876469970686 , y: 0.01414718772534178, z: 0.027546614014801026}
    orientation: {r: 0.0, p: 0.0, y: 3.14} # <---- Added the PI here
  scale: 1.0
  dimensions: {x: 0.052761648, y: 0.061956624, z: 0.061518768}
  images:
    path: volume/
    prefix: plane000
    format: png
    count: 281
BODY volume_anatomical_origin: # This is a dummy body that can be used to represent the anatomical origin for easy reference
  name: volume_anatomical_origin
  mass: 0.0
  mesh: Cube.stl
  scale: 1.0
  location:
      position:
        x: 0.0
        y: 0.0
        z: 0.0
      orientation:
        r: 0.0
        p: 0.0
        y: 0.0

I also verified this by creating an STL mesh of the Segmentation in LPS coordinates and loading it alongside the edited (with PI rotation along z-axis) volume in AMBF. The orange thing is the mesh in this image.

Here is the NRRD file that I am using:
Trial1Scan0.zip
.

More control relative locations of ADF files and volume PNG slice files

Currently, will save ADF in the output directory and images in output_dir/volname/

I suppose this could be useful to generalize

Get rid of the length printed with the AMBF origin axes

Appears by default in 3D view

The calculated dimensions should take into account image spacing

For a volume the "dimensions" in 3D Slicer represent voxel count while in AMBF they represent simulator units (which we should consider as SI). Therefore we should take into account the Slicer "spacing" field and multiply it by the "dimensions" to get dimensions in mm and convert the to m before writing to ADF.