Observed:
Configured a workflow to run an example map built with MONAI Deploy App SDK v0.6, e.g., ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0, but workflow instance failed with "no output"
Expected:
MD Express should support the MAPs built with App SDK v0.6 and after; these MAPs are built differently than those with MONAI Deploy App SDK v0.5.x, and have additional requirements on the permission on the host folders (which are to be mapped as MAP output"
Preliminary Analysis:
Root cause is in the writer permission, or the lack thereof, on the output folder in the host file system.
Workaround attempted but did not working due to TM/Docker behavior:
MD Express TaskManager container itself maps host volume to its internal folder, and also uses them for MAP input and output, so tried to first create the host folder with the logged-on user's security context, e.g. mkdir -p .md/mdtm. However, when the TaskManager creates further sub-folder for a triggered task, the folder is created as owned by root, without sufficient permission for the MAP to write output to.
(.test) mqin@mingq-dt:~/md-express/deploy/monai-deploy-express$ ll .md
total 32
drwxrwxr-x 8 mqin mqin 4096 Oct 24 19:17 ./
drwxr-xr-x 8 mqin mqin 4096 Oct 24 19:16 ../
drwxr-xr-x 4 root root 4096 Oct 24 19:17 mdig/
drwxrwxr-x 2 mqin mqin 4096 Oct 24 19:16 mdtm/
drwxr-xr-x 4 root root 4096 Oct 24 19:16 minio/
drwxr-xr-x 4 systemd-coredump root 4096 Oct 24 19:17 mongodb/
drwxr-xr-x 231 root root 4096 Oct 24 19:18 orthanc/
drwxr-xr-x 4 systemd-coredump root 4096 Oct 24 19:17 rabbitmq/
(.test) mqin@mingq-dt:~/md-express/deploy/monai-deploy-express$ ll .md/mdtm
total 12
drwxrwxr-x 3 mqin mqin 4096 Oct 24 19:19 ./
drwxrwxr-x 8 mqin mqin 4096 Oct 24 19:17 ../
drwxr-xr-x 4 root root 4096 Oct 24 19:19 960636a5-d7be-415c-9fe3-1d93b0e783c6/
Step to reproduce:
Create an workflow definition as below (note, the attributes starting with env_ do not matter as the actual values are all correct and the MAP has default env vars. However, different error happened, see Additional Info)
Download the MD Express, docker compose up
Register the workflow, per instruction on MD Express Readme
Upload the a CT abdomen series, per MD Express Readme
Send the series to DICOM device destination, monai-deploy
Examine the console output of docker compose
{
"name": "ai-liver-seg-2",
"version": "1.0.0",
"description": "AI Liver Segmentation - 2",
"informatics_gateway": {
"ae_title": "MONAI-DEPLOY",
"data_origins": [
"ORTHANC"
],
"export_destinations": [
"ORTHANC"
]
},
"tasks": [
{
"id": "router",
"description": "Ensure series description contains liver",
"type": "router",
"task_destinations": [
{
"name": "liver",
"conditions": ["{{ context.dicom.series.any('0008','103E')}} == 'CT series for liver tumor from nii 014'"]
}
]
},
{
"id": "liver",
"description": "Execute Liver Segmentation MAP",
"type": "docker",
"args": {
"container_image": "ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0",
"server_url": "unix:///var/run/docker.sock",
"entrypoint": "/bin/bash,-c",
"command": "python3 -u /opt/holoscan/app/app.py",
"task_timeout_minutes": "5",
"temp_storage_container_path": "/var/lib/monai/",
"env_MONAI_INPUTPATH": "/var/holoscan/input/",
"env_MONAI_OUTPUTPATH": "/var/holoscan/output/",
"env_MONAI_MODELPATH": "/opt/holoscan/models/",
"env_MONAI_WORKDIR": "/var/holoscan/"
},
"artifacts": {
"input": [
{
"name": "env_MONAI_INPUTPATH",
"value": "{{ context.input.dicom }}"
}
],
"output": [
{
"name": "env_MONAI_OUTPUTPATH",
"mandatory": true
}
]
},
"task_destinations": [
{
"name": "export-liver-seg"
}
]
},
{
"id": "export-liver-seg",
"description": "Export Segmentation Storage Object",
"type": "export",
"export_destinations": [
{
"Name": "ORTHANC"
}
],
"artifacts": {
"input": [
{
"name": "export-dicom",
"value": "{{ context.executions.liver.artifacts.env_MONAI_OUTPUTPATH }}",
"mandatory": true
}
],
"output": []
}
}
]
}
Additional Info:
Used a config that replaces all monai with holoscan, but encountered a different error.
This one is because that in the docker-compose.yml file, the TaskManager container itself has the volume mapping set to its own /var/lib/monai/, so the workflow def cannot use "temp_storage_container_path": "/var/lib/holoscan/"
So, stick with "temp_storage_container_path": "/var/lib/monai/", and change it in the following config, to avoid this other binding error.
{
"name": "ai-liver-seg-HS",
"version": "1.0.0",
"description": "AI Liver Segmentation - HS",
"informatics_gateway": {
"ae_title": "MONAI-DEPLOY",
"data_origins": [
"ORTHANC"
],
"export_destinations": [
"ORTHANC"
]
},
"tasks": [
{
"id": "router",
"description": "Ensure series description contains liver",
"type": "router",
"task_destinations": [
{
"name": "liver",
"conditions": ["{{ context.dicom.series.any('0008','103E')}} == 'CT series for liver tumor from nii 014'"]
}
]
},
{
"id": "liver",
"description": "Execute Liver Segmentation MAP",
"type": "docker",
"args": {
"container_image": "ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0",
"server_url": "unix:///var/run/docker.sock",
"entrypoint": "/bin/bash,-c",
"command": "python3 -u /opt/holoscan/app/app.py",
"task_timeout_minutes": "5",
"temp_storage_container_path": "/var/lib/holoscan/",
"env_HOLOSCAN_INPUT_PATH": "/var/holoscan/input/",
"env_HOLOSCAN_OUTPUT_PATH": "/var/holoscan/output/",
"env_HOLOSCAN_MODEL_PATH": "/opt/holoscan/models/",
"env_HOLOSCAN_WORKDIR": "/var/holoscan/"
},
"artifacts": {
"input": [
{
"name": "env_HOLOSCAN_INPUT_PATH",
"value": "{{ context.input.dicom }}"
}
],
"output": [
{
"name": "env_HOLOSCAN_OUTPUT_PATH",
"mandatory": true
}
]
},
"task_destinations": [
{
"name": "export-liver-seg"
}
]
},
{
"id": "export-liver-seg",
"description": "Export Segmentation Storage Object",
"type": "export",
"export_destinations": [
{
"Name": "ORTHANC"
}
],
"artifacts": {
"input": [
{
"name": "export-dicom",
"value": "{{ context.executions.liver.artifacts.env_HOLOSCAN_OUTPUT_PATH }}",
"mandatory": true
}
],
"output": []
}
}
]
}
But got a different error from task manager on binding the volumes
mdl-tm | 2023-10-25 00: 52: 09.7745|1010|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|source=1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm, target=/var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm, EventId=1010, EventName=DownloadingArtifactFromStorageService, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Downloading artifact 1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm to /var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.
mdl-tm | 2023-10-25 00: 52: 09.7890|1010|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|source=1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json, target=/var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json, EventId=1010, EventName=DownloadingArtifactFromStorageService, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Downloading artifact 1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json to /var/lib/holoscan/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH/1.2.826.0.1.3680043.2.1125.1.19616861412188316212577695277886020/1.2.826.0.1.3680043.2.1125.1.34918616334750294149839565085991567/1.2.826.0.1.3680043.2.1125.1.99874075217122757899127966289752952.dcm.json.
mdl-tm | 2023-10-25 00: 52: 10.0870|1006|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|hostPath=/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH, containerPath=/var/holoscan/input/, EventId=1006, EventName=DockerInputMapped, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Input volume mapping host==/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH, container=/var/holoscan/input/.
mdl-tm | 2023-10-25 00: 52: 10.0872|1007|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|hostPath=/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/outputs/env_HOLOSCAN_OUTPUT_PATH, containerPath=/var/holoscan/output/, EventId=1007, EventName=DockerOutputMapped, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Output volume mapping host==/home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/outputs/env_HOLOSCAN_OUTPUT_PATH, container=/var/holoscan/output/.
mdl-tm | 2023-10-25 00: 52: 10.0873|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_INPUT_PATH, value=/var/holoscan/input/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_INPUT_PATH=/var/holoscan/input/.
mdl-tm | 2023-10-25 00: 52: 10.0874|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_OUTPUT_PATH, value=/var/holoscan/output/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_OUTPUT_PATH=/var/holoscan/output/.
mdl-tm | 2023-10-25 00: 52: 10.0874|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_MODEL_PATH, value=/opt/holoscan/models/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_MODEL_PATH=/opt/holoscan/models/.
mdl-tm | 2023-10-25 00: 52: 10.0875|1008|INFO|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|key=HOLOSCAN_WORKDIR, value=/var/holoscan/, EventId=1008, EventName=DockerEnvironmentVariableAdded, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Environment variabled added HOLOSCAN_WORKDIR=/var/holoscan/.
mdl-tm | 2023-10-25 00: 52: 10.0891|1002|ERROR|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|EventId=1002, EventName=ErrorDeployingContainer, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent, workflowInstanceId=12cb526e-6fe3-4518-a941-ca558be54d4f, taskId=liver, executionId=7e8816df-3fd4-4cae-b083-2fb2a074b8a5|Error deploying Container. Docker.DotNet.DockerApiException: Docker API responded with status code=BadRequest, response={
"message": "invalid mount config for type \"bind\": bind source path does not exist: /home/mqin/md-express/deploy/monai-deploy-express/sample-workflows/.md/mdtm/7e8816df-3fd4-4cae-b083-2fb2a074b8a5/inputs/env_HOLOSCAN_INPUT_PATH"
}
mdl-tm |
mdl-tm | at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
mdl-tm | at Docker.DotNet.DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
mdl-tm | at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken)
mdl-tm | at Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin.ExecuteTask(CancellationToken cancellationToken) in /app/src/TaskManager/Plug-ins/Docker/DockerPlugin.cs:line 205
mdl-tm | 2023-10-25 00: 52: 10.0906|106|DEBUG|Monai.Deploy.WorkflowManager.TaskManager.TaskManager|eventType=md.tasks.update, reason=PluginError, EventId=106, EventName=SendingTaskUpdateMessage, @messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, @applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent|Sending md.tasks.update, Status=PluginError .
mdl-tm | 2023-10-25 00: 52: 10.0911|10000|INFO|Monai.Deploy.Messaging.RabbitMQ.RabbitMQMessagePublisherService|endpoint=rabbitmq, virtualHost=monaideploy, exchange=monaideploy, topic=md.tasks.update, EventId=10000, EventName=PublshingRabbitMQ, @messageId=6da618d6-ada7-41b5-9427-46a775c7f8ce, @applicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, @correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, @recievedTime=10/25/2023 00: 52: 00, correlationId=edcb1deb-c6e2-4280-a096-2e8b29701f80, messageId=7c54a688-569e-47a5-a362-f2cdc4a5c2d8, messageType=TaskDispatchEvent|Publishing message to rabbitmq/monaideploy. Exchange: monaideploy, Topic: md.tasks.update.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent