MAP built with App SDK v0.6 fails when running on MONAI Deploy Express

[project-monai]

MAP built with App SDK v0.6 fails when running on MONAI Deploy Express

[SameerShanbhogue]

@mocsharp
@MMelQin
Yes v0.6 is there a work around till bug is fixed.

[SameerShanbhogue]

mdl-wm | 2023-10-31 09:18:13.2528|20001|ERROR|Monai.Deploy.Storage.MinIO.MinIoStorageService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|File '1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH' could not be found in 'monaideploy'. mdl-wm | 2023-10-31 09:18:13.2567|700000|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|Artifact Passed data Artifact {"name":"env_MONAI_OUTPUTPATH","value":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH","mandatory":true}, Path 1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc/env_MONAI_OUTPUTPATH, ArtifactType Post-Task Output Artifact, Exists False
mdl-wm | 2023-10-31 09:18:13.2601|700004|DEBUG|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|Mandatory output artefacts for task liver are missing. mdl-wm | 2023-10-31 09:18:13.2622|200015|INFO|Monai.Deploy.WorkflowManager.WorkfowExecuter.Services.WorkflowExecuterService|MessageId=860e66da-f16b-4ceb-8db5-580a43ec03ee, ApplicationId=4c9072a1-35f5-4d85-847d-dafca22244a8, CorrelationId=15d68444-fa6e-4c17-8fa8-e4318ca0ead7, RecievedTime=10/31/2023 09:18:13 System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object] System.Collections.Generic.Dictionary2[System.String,System.Object]|TaskFailed, Task {"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}, workflowInstance {"Version":"1.0.0","id":"6df274a9-49ab-4847-b1dc-69a8154a6612","ae_title":"MONAI-DEPLOY","workflow_name":"ai-liver-seg","workflow_id":"abdf887f-126d-4de6-bbb9-3e58a3ba81a8","payload_id":"1918d483-8609-4c64-b622-6413f43f9abb","start_time":"2023-10-31T09:18:02.967Z","status":0,"bucket_id":"monaideploy","input_metadata":{},"tasks":[{"execution_id":"8833f1c4-81c7-43d5-90b5-62a8a065aa15","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"router","task_start_time":"2023-10-31T09:18:02.969Z","task_end_time":"2023-10-31T09:18:03.3Z","execution_stats":{},"task_plugin_arguments":{"workflow_name":"ai-liver-seg"},"task_id":"router","previous_task_id":"","status":3,"reason":0,"input_artifacts":{},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/8833f1c4-81c7-43d5-90b5-62a8a065aa15","result":{},"next_timeout":"2023-10-31T10:18:02.969Z","timeout_interval":60.0,"acknowledged_task_errors":null},{"execution_id":"10edc066-56bc-43cf-8cf2-47291ebf3dfc","workflow_instance_id":"6df274a9-49ab-4847-b1dc-69a8154a6612","task_type":"docker","task_start_time":"2023-10-31T09:18:03.181Z","task_end_time":null,"execution_stats":{"IdentityKey":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e","workflowId":"6df274a9-49ab-4847-b1dc-69a8154a6612","duration":"1793.7903","startedAt":"2023-10-31T09:18:08","finishedAt":"2023-10-31T09:18:10"},"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","task_timeout_minutes":"30","temp_storage_container_path":"/var/lib/monai/","env_MONAI_INPUTPATH":"/var/monai/input/","env_MONAI_OUTPUTPATH":"/var/monai/output/","env_MONAI_MODELPATH":"/opt/monai/models/","env_MONAI_WORKDIR":"/var/monai/"},"task_id":"liver","previous_task_id":"router","status":2,"reason":0,"input_artifacts":{"env_MONAI_INPUTPATH":"1918d483-8609-4c64-b622-6413f43f9abb/dcm"},"output_artifacts":{},"output_directory":"1918d483-8609-4c64-b622-6413f43f9abb/workflows/6df274a9-49ab-4847-b1dc-69a8154a6612/10edc066-56bc-43cf-8cf2-47291ebf3dfc","result":{"JobIdentity":"8c2ed70e1b0acac04df8f916cd995ec8c13d9486294cf5eacf37e4a28346777e"},"next_timeout":"2023-10-31T10:18:03.181Z","timeout_interval":60.0,"acknowledged_task_errors":null}],"acknowledged_workflow_errors":null}, patientDetails {"patient_id":"C3N-00198","patient_name":"C3N-00198","patient_sex":"M","patient_dob":null,"patient_age":"055Y","patient_hospital_id":null}, correlationId 15d68444-fa6e-4c17-8fa8-e4318ca0ead7, taskStatus Failed

Is this error related to the bug

[mocsharp]

@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.

[MMelQin]

@SameerShanbhogue did you build your app using MONAI Deploy App SDK 0.5 or 0.6? If 0.6, then it is likely to be related. Thanks.

The log entry has the telltale sign that @SameerShanbhogue indeed built and tested the MAP with App SDK v0.6.
'''
"task_plugin_arguments":{"container_image":"sameershanbhogue/my_app-x64-workstation-dgpu-linux-amd64:1.0","server_url":"unix:///var/run/docker.sock","entrypoint":"/bin/bash,-c","command":"python3 -u /opt/monai/app/app.py","
'''

[mocsharp]

I have a fix coming soon!

[mocsharp]

@SameerShanbhogue Please try https://github.com/Project-MONAI/monai-deploy/releases/tag/monai-deploy-express-v0.5.0 and let me know if that works for you! Thanks!

[MMelQin]

@mocsharp @MMelQin Yes v0.6 is there a work around till bug is fixed.

As @mocsharp mentioned, the fix is known, and pull request already reviewed and merged. Please try MD Express v0.5.0 as mentioned above.

As for workaround with earlier version of MD Express, there is no good ones if the MAP (App SDK v0.6) is built with no-root user, because the Task Manager (and the Docker plugin), when run in container as root, will dynamically create a folder on the host FS to use as the output folder for the MAP container, and the dynamically created folder would not have "Write" permission for other users.

If you build the MAP with container user be root, then it would work. However, the Packager in v0.6 has a little issue where it does not fall back if GUID 0 in the container already exists, and would fail; this issue has also been addressed, but to be released in v1.0.

[SameerShanbhogue]

Thank You @mocsharp @MMelQin

MD Express v0.5.0 with Liver Tumor 2.0 workflow definition works !!