janeliascicomp/hortacloud

HortaCloud - 3D annotation platform for large microscopy data

HortaCloud

HortaCloud is a streaming 3D annotation platform for large microscopy data that runs entirely in the cloud. It is based on the Janelia Workstation software and was originally developed in support of the MouseLight Team Project.

It combines state-of-the-art volumetric visualization, advanced features for 3D neuronal annotation, and real-time multi-user collaboration with a set of enterprise-grade backend microservices for moving and processing large amounts of data rapidly and securely. HortaCloud takes advantage of cloud-based Virtual Desktop Infrastructure (VDI) to perform all 3D rendering in cloud-leased GPUs which are data-adjacent, and only transfer a high-fidelity interactive video stream to each annotator’s local compute platform through a web browser.

More information is available on the HortaCloud website.

Deployment

See the documentation to learn how to deploy this system to your AWS account.

Contributing

Read about how to contribute to this project.

hortacloud's Issues

Add "Launch in Client" and "Launch in Browser" buttons

On Windows you can install the AppStream client which makes AppStream applications look exactly like native apps: https://clients.amazonappstream.com/

Then, in the AppStream UI when you select "create streaming URL" it has an option to "Launch in Client" or "Launch in Browser". It would be nice to present both of these as options in our portal.

We should also include the "Copy link" button which copies the AppStream link to allow the user to connect manually.

Neurons are not shared when workspace is shared

When a workspace is shared, the neurons within the workspace are not shared. Usually this results in the target user being able to open the workspace but not see any neurons.

Internal number JW-49590.

Not possible to set keymap

Tools > Options > Keymap

Set the hotkey for the "Add/Edit Note" action to "N".
Press "Apply" and "OK".

Select a point in Horta 3D and press "N" - nothing happens. The expected behavior would be to see the "Add note" dialog pop up.

This also is the case for the other hotkeys.

Persisting filecache causing persistence issues

The filecache directory is in the user directory by default, and when we turned on AppStream user setting persistence, it tried to save all the cached files to S3. However the profile persistence has a max of 1GB, and they get corrupted when they are overfilled. We need to stop the filecache from persisting, and test to make sure user settings are persisted even after browsing many samples that fill up the file cache.

cannot run `npm run deploy`

I was just doing a full redeploy. I:

1. npm run destroy
2. git pull
3. npm run deploy

I received an error on this line:

hortacloud/scripts/credentials.js

Line 9 in 867887a

return mfaDeviceResponse.MFADevices[0].SerialNumber;

It indicated that the MFADevices array is empty.

To proceed, I edited this line to prevent looking for MFA devices:

hortacloud/scripts/credentials.js

Line 26 in 867887a

const mfaDevice = await getMFADevice(region);

add low disk space alarm for /data

When /data is full, usually because rabbitmq messages have backed up, a recovery.dats file becomes truncated and the rabbit server is unable to start. See:

https://www.ibm.com/support/pages/how-start-rabbitmq-when-it-crashes-due-insufficient-space

When this happens, the likelihood for data loss is high. Please add a cloudwatch alarm that will go off when /data is 80% full.

Move Horta-specific startup to its own showing hook

There are Horta options in org.janelia.workstation.core.options.ApplicationOptions and org.janelia.workstation.common.gui.lifecycle.ShowingHook which don't belong in the common GUI module. It should be very easy to move them into a Horta-specific ShowingHook.

Loading a large number of SWCs from s3 is slow

Using the "Load Linux SWC Folder into New Workspace on Sample" option, loading 2,354 SWC files from an S3 bucket takes ~2m. Assuming linear scaling, loading 2 million SWCs would take ~33hr . Clearly, loading data from S3 is expected to be slower than from a local or network drive, but it would be great if there were ways to speed it up (parallelize somehow?). It also seems tricky to get that number of files onto the EC2 instance's local disk via Temporary Files, OneDrive or Google Drive. Perhaps if there were a way to unzip files within the AppStream instance, that would help?

The swcs I used are here s3://aind-msma-morphology-data/test/from_google_exaSPIM_609281_2022-11-03_13-49-18-training-data_n5_whole-brain_consensus_1000/

Undo is not working after moving a point

After moving a neuron point the Edit->Undo option remains greyed out and the move operation cannot be undone.

Undoing neuron change makes neuron non-interactable

This started occurring with 9.14.1

Steps to reproduce:

1. Create a new neuron
2. Add 2 points
3. Press Ctrl+Z to undo 2nd point
4. Hover mouse over neuron - nodes not highlighted

The neuron restored from history has annotations with a null neuronId field, which causes the highlighting behavior to fail, see https://github.com/JaneliaSciComp/workstation/blob/430d670bd4edc7677d6bba7ef6a4d14824522028/modules/HortaTracer/src/main/java/org/janelia/horta/TracingInteractor.java#L706

The source of the issue appears to be with the updateNeuron function here https://github.com/JaneliaSciComp/workstation/blob/430d670bd4edc7677d6bba7ef6a4d14824522028/modules/ViewerController/src/main/java/org/janelia/workstation/controller/access/TiledMicroscopeRestClient.java#L311 , which returns a neuron with uninitialized data.

initializing the neuron data after update appears to fix the issue, see carshadi/workstation@c2fee44 . However, this might not be optimal since it requires a linear scan over all the nodes every time a neuron is updated, which could impact performance with larger neurons. I was considering doing the initialization only when the undo function is called, e.g., here https://github.com/JaneliaSciComp/workstation/blob/430d670bd4edc7677d6bba7ef6a4d14824522028/modules/ViewerController/src/main/java/org/janelia/workstation/controller/NeuronManager.java#L723

Maybe there's a better way to do this? thoughts @porterbot , @krokicki ?

Add persistence indicator

There should be an indicator which makes it obvious when user changes are finally persisted into the database. We can implement this using the async messaging service.

Published neurons should be read-only

Currently it's very easy to accidentally modify the published neuron workspaces. We need to add a feature to allow us to lock these down so that they cannot be edited by anyone, only viewed or copied.

Besides disallowing annotation operations, this should also disable SWC import and all other mutations.

Workstation client asking for login

Several users have reported having to log into the Workstation after already logging into HortaCloud. This shouldn't happen, login into Workstation should be automatic. The screenshot attached is from Jayaram, but Monet also reports the same behavior.

Clean up extra VPCs

There are currently two security groups created for the HortaCloud EC2 instance, and only one is used.
SG that is created and unused: https://github.com/JaneliaSciComp/hortacloud/blob/main/vpc_stack/src/hortacloud-vpc.ts#L20
SG that is actually used: https://github.com/JaneliaSciComp/hortacloud/blob/main/vpc_stack/src/hortacloud-jacs.ts#L45
You can see in the generated diagram that the one with all the Systems Manager access is never connected to anything.
We need to figure out which SG is needed, and eliminate any that are not needed.

Automatically start the stack after a workstation reboot

Case Insensitive usernames

Add content from workstation docs/Development to HortaCloud dev docs

Create a workspace for performance benchmarks

It would be useful to have a workspace with hundreds of thousands of reconstructions. (as in MouseLight "fragments"). A good candidate would be those generated for 2018-10-01

Brain Tile doesn't load correctly

I am running into some problems with the workstation that don't let me correctly use the Horta 3D viewer. The problem is that I always end up with a black screen and not being able to see the brain. The most common problem is that I am correctly following an axon, with lot of zoom in, and suddenly there are no more brain squares loaded and I just can see the traced neuron. I waited (5-10 minutes), I tried zooming in and out and moving the brain, but nothing changes. All the other interactive things of the workstation seem to work, but the brain doesn't load. All this time, I see the loading bar, but even if I wait for 15-20 minutes nothing changes, so I asume that something has freezed.

At some time a window pops up saying "Brain Raw Data Error":
Error loading brain specimen:java.lang.IllegalStateException: HTTP status 504 (not OK) from url https://ip-10-0-0-80.us-west-2.compute.internal/SCSW/JACS2SyncServices/v2/mouselight/samples/3087594671233310722/volumen_info

If I close the session and reload everything, then it works fine at first, until I run into the same problems.The same thing happens with different neurons (I tried it with AA0345 and AA0346). I am running them on the Tile Microscope Sample that I created, of the 2017-09-19 brain.

Synchronous neuron persistence

Refactor neuron persistence so that the client calls a synchronous remote API to make changes. This API will notify the other users using the RabbitMQ service. This will obviate the jacs-messaging piece and make it easier to reason about persistence issues.

The API calls will be serialized on a queue to prevent race conditions.

add "show other" neuron functions

Requested by local users. Added two variations of "show other neurons" to appropriate menus.

Fixed delayed update bug in "hide others" while in the code neighborhood.

Issues with neuron point selection in Horta 3D

There are two issues relating to point selection in Horta 3D

1. Occasionally it stops working entirely, and it is not possible to select any points. This can usually be fixed by closing the session and starting fresh.
2. When it does work, it is more difficult to select points compared to the desktop app. Hiding the image data does not help.

By point selection, I mean obtaining the state in the second image, where the nearest point to the cursor is highlighted.

issue with right-click not actually selecting a neuron in Horta3D, leading to errors in neuron deletes

Neuron ownership changes are not persisted

If you change neuron ownership in the UI, it appears to succeed. However, while the change is active in memory, it is not persisted to the db.

Internal number JW-49974.

Adding OmeZARR support for HortaCloud

The week of 4/7/2023, Patrick and I managed to get OmeZARR working in HortaCloud. There are some follow-up issues that need to be resolved to make it workable though.

• Support for initializing Zarr loading on Workspace open (having more messaging indicating load status for the Zarr) (David)
• Checking that you could load Levels 1-4 of the Zarr multi-level octree (Patrick)
• Making sure that the Zarr loading is performant compared to the KTX (chunk-size and multi-tile parallel loading) -
• Double-checking that the tile issues on rotation are still not occurring (artifacts) - Jayaram
• Setting Z-compression flag to uncompressed by default for Zarr samples

Google Drive folder disappears

Google Drive used to work but it has disappeared

Issue with Neurons disappearing from the workstation

Annotators are losing neuron work intermittently, where the neuron is deleted and doesn't show up upon workspace being reopened. We will need IDE logs to figure out exactly what's going on; in the meantime, annotators should periodically backup their work to SWC files

NPE when right-clicking in Horta 3D viewer

Occasionally, when first loading a workspace, I get this error when right-clicking in the 3D viewer to bring up the rendering options (not on a neuron)

java.lang.NullPointerException
	at org.janelia.horta.NeuronTracerTopComponent$13.createMenu(NeuronTracerTopComponent.java:1571)
	at org.janelia.horta.NeuronTracerTopComponent$13.showPopup(NeuronTracerTopComponent.java:1804)
	at org.openide.awt.MouseUtils$PopupMouseAdapter.maybePopup(MouseUtils.java:189)
	at org.openide.awt.MouseUtils$PopupMouseAdapter.mouseReleased(MouseUtils.java:184)
	at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:290)
	at java.awt.Component.processMouseEvent(Component.java:6539)
	at java.awt.Component.processEvent(Component.java:6304)
	at java.awt.Component.dispatchEventImpl(Component.java:4889)
	at java.awt.Component.dispatchEvent(Component.java:4711)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:760)
	at java.awt.EventQueue.access$500(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:709)
	at java.awt.EventQueue$3.run(EventQueue.java:703)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:84)
	at java.awt.EventQueue$4.run(EventQueue.java:733)
	at java.awt.EventQueue$4.run(EventQueue.java:731)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:730)
[catch] at org.janelia.workstation.common.logging.EDTExceptionInterceptor.dispatchEvent(EDTExceptionInterceptor.java:21)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

This is resolved by selecting an arbitrary point on a neuron, then right-clicking again away from the neuron to bring up the rendering options

Disable login screen

We should disable the Workstation login screen in HortaCloud and assume the user from the environment.

Fix email error reporting

Investigate why the service currently throws a HTTP 503 error for Don.

DropBox integration

Make it possible to mount DropBox (or OneDrive) on AppStream to be able to save/load SWC files (and other external files)

Demo instance

We should stand up a demo instance for users who do not have access to an institutional instance but want to try out the software.

Support importing/exporting .swc files directly to AWS s3

Proposal

Something that would be super useful is the ability to import and export .swc files directly from s3, using the top gear menu in the Horta Control Center, e.g., "Import SWC Files as One Neuron...". This would avoid having to go through the "Temporary Files" folder as a middle-man.

Suggest implementation

I would be happy to contribute this if someone could point me to the relevant classes and give a general outline of what needs to be accounted for / tested

Issue with Visibility toggling back on for a neuron in Horta3D even though the visibility for that neuron has been set to off.

This is working in Horta 2D so just need to find why this is happening the Horta3D

Automatic path tracing not working

If you enable the "Automatic path tracing" option in the top gear menu in Horta Control Center, then try to trace a path in Horta 2D, the following error occurs:

java.lang.NullPointerException
	at org.janelia.workstation.controller.tileimagery.TextureData2dGL.<init>(TextureData2dGL.java:31)
	at org.janelia.workstation.controller.tileimagery.Subvolume.fetchTileData(Subvolume.java:438)
	at org.janelia.workstation.controller.tileimagery.Subvolume.access$000(Subvolume.java:38)
	at org.janelia.workstation.controller.tileimagery.Subvolume$1.call(Subvolume.java:380)
	at org.janelia.workstation.controller.tileimagery.Subvolume$1.call(Subvolume.java:377)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Caused: java.util.concurrent.ExecutionException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[catch] at org.janelia.workstation.controller.tileimagery.Subvolume.multiThreadedFetch(Subvolume.java:395)
	at org.janelia.workstation.controller.tileimagery.Subvolume.initialize(Subvolume.java:231)
	at org.janelia.workstation.controller.tileimagery.Subvolume.<init>(Subvolume.java:84)
	at org.janelia.workstation.gui.large_volume_viewer.tracing.PathTraceToParentWorker.doStuff(PathTraceToParentWorker.java:82)
	at org.janelia.workstation.core.workers.SimpleWorker.createResult(SimpleWorker.java:14)
	at org.janelia.workstation.core.workers.SimpleWorker.createResult(SimpleWorker.java:10)
	at org.janelia.workstation.core.workers.ResultWorker.doInBackground(ResultWorker.java:50)
	at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at javax.swing.SwingWorker.run(SwingWorker.java:334)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

The error stops occurring when disabling the "Automatic path tracing" option

inadequate user feedback is neuron if not persisted

If a neuron is not saved, the user feedback is missing or inadequate. Currently the neuron name in the neuron list changes to italic if more than one neuron update sent to the server has not returned (leaving the neuron out of sync).

• This is easy to miss.
• It only refreshes when the table refreshes, which is often but is not guaranteed.
• If you only do one thing and it fails, the table may not refresh and never let you know there's a problem.

Improve this.

user with read-only perm to TmWorkspace can add neurons to it

Carveout feature for retrieving imagery in proximity to selected neuron

I'm in the process of finishing up a feature for Horta 2D that generates a zip archive of TIFF files in proximity to a selected neuron. It's accessible through right click and I'm retrieving the files through JADE.

Resolve issue with ClientRefresh backup queue for RabbitMQ

On the initial HortaCloud CDK, we should not configure a backup queue for RabbitMQ, which can causing out-of-space issues.

Rebooting daily to re-attach S3 mounts

We are rebooting the JACS EC2 instances nearly every day because the workstation loses access to the underlying S3 mounts. When I inspect the /s3data folder, sure enough the bucket in question appears to contain no data. We usually address this by rebooting the server, which is painful to users.

Can you please find a way to keep the underlying s3 mounts healthy?

Add neuron tags to indicate that a neuron is a fragment in Horta

Automated email error reporting

Currently, the Workstation's bug reporting relies on the client making an SMTP connection to a mail server. We need to get this working on AWS and expose it in the HortaCloud CDK installation.

See e.g. https://stackoverflow.com/questions/12122771/how-to-send-emails-from-amazon-ec2

Investigate why /data fills up

When /data is full, usually because rabbitmq messages have backed up, a recovery.dats file becomes truncated and the rabbit server is unable to start. See:

https://www.ibm.com/support/pages/how-start-rabbitmq-when-it-crashes-due-insufficient-space

When this happens, the likelihood for data loss is high.

Investigate why this happens. I have 300GB volume.

When I 1) grow the /data volume, 2) remove recovery.dets, and 3) bounce the stack, the message queue clears.

Test draft

REST client is being recreated for every operation

App state is not correctly cleared between sample loads

When you load a Sample or Workspace into Horta, it caches information in different objects through-out the app. If you open a new sample, sometimes the neurons are loaded in the wrong coordinate space, or there may be other issues. Many of these issues come down to clearing out the state through-out the application.
We need to ensure that every setting and state is reset before a new Sample or Workspace is loaded.

Fix link from Horta app to user manual

Currently using Help➝User Manual brings up an Internet Explorer window inside AppStream which never loads anything.

We need to find some way to display the user manual

If this isn't easy to do within AppStream, we could just popup a dialog box that says "Open hortacould.janelia.org in your browser to read the user manual."

Delete unused code from jacs-messaging

Verify all Open Data MouseLight samples

Hide terminal window

When Horta starts up there is a terminal window running the background that is distracting. Is there some way we can hide this window unless a debugging mode is enabled?