Comments (31)
To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.
1 = very difficult obstacle
5 = not an obstacle
e.g. Lack of resources | 1 2 3 4 5 N/A
from monai-deploy.
I would be interested to know whether the respondents anticipate that AI will be provided as predominantly as:
a) Cloud Services
b) On Premises
c) Hybrid
from monai-deploy.
Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ? Some of the options are
- Lack of Resources (Financial and Man Power)
- Lack of acceptance because of ethical issues
- Lack of knowhow.
- Not enough financial benefits from AI based deployment.
- Lack of trust.
Can we have some thoughts about the best set of 5 options we want to offer? @ristoh @laurencejackson @JHancox @dbericat @slbryson
I think the question should be something like :
A major challenge in the routine deployment of AI is:
- funding (1 | 2 | 3 |4 | 5)
- technical infrastructure (integration, hardware, software) (1 | 2 | 3 |4 | 5)
- viable products (return on investment, performance) (1 | 2 | 3 |4 | 5)
- local expertise (1 | 2 | 3 |4 | 5)
- business process change (1 | 2 | 3 |4 | 5)
from monai-deploy.
I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.
- local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)
Thanks, Laurence. I think we don't want to allow a general answer/option because it is of limited use when interpreting the results. I think what you're describing could be either local expertise or it could be viable products (i.e. there are no products that allay my fears around AI). I'll amend option 4 with your suggestion whilst awaiting other comments
from monai-deploy.
I like the set of questions posed. Definitely keep the format of allowing the response to comment on "All" the choices with 1|2|3|... adding that 5 is strongly impacted or whatever the desired meaning of the range.
from monai-deploy.
Survey looks good!
@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?
i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?
- Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
- Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
- Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
- Yes, other: (suggest ways to contribute)
- No, I'd rather focus on other things
@hshuaib90 if you think this is diluting the message, feel free to leave out.
I think this is a great idea. I think it would be good to follow up with respondents with this ask as opposed to putting it in the initial survey, just to keep the survey as focussed as possible.
from monai-deploy.
See draft here: https://docs.google.com/forms/d/1zHSS9exLLfivQlh9HxRUI3S8FLnznhlmShr5IIvARWo/edit
What's the best way to develop this? Happy for people to put comments here and I can make changes or can try convert to markdown (somehow?) and we can work on that until finalised and then convert back to Google Form.
from monai-deploy.
I like the survey. Do we have any members of the working group who haven't yet provided this info, who could dry run the survey and see if it fulfills the purpose?
from monai-deploy.
It's possible since we've had a few new members. We can ask during this Thursday's meeting.
from monai-deploy.
Is it possible to add a question to institutions that, If you wanted to deploy AI based solutions at you institutions and you have not deployed it, what was the reason ?
Some of the options are
- Lack of Resources (Financial and Man Power)
- Lack of acceptance because of ethical issues
- Lack of knowhow.
- Not enough financial benefits from AI based deployment.
- Lack of trust.
from monai-deploy.
Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.
from monai-deploy.
Is the Radiology focus deliberate? AI can obviously be a part of other MONAI Deploy pipelines such as pathology, endoscopy etc. I appreciate that Radiology is probably the simplest option - just don't want to miss a trick unless it's deliberate.
from monai-deploy.
I think it is deliberate. It would be a much bigger endeavour to try and map things like pathology , endoscopy etc. It might be worth adding to next year's audit.
I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.
from monai-deploy.
Some of the questions are asking about information that is probably publicly available (e.g. Number of studies per year etc.) We may want to consider sourcing this information independently, which will also make it consistent from different folks in the same institution, who also may not know all those figures.
I agree we should do this where available.
from monai-deploy.
To add to @vikashg's comment, how would people feel if the answers to those questions could be on a scale (1-5 + N/A) of how large an obstacle to deployment each factor is, then we could monitor improvements in each category in the case of this becoming an annual survey. e.g. could have evidence that in 2023 lack of trust was less of an obstacle to deployment than in 2022.
1 = very difficult obstacle 5 = not an obstacle e.g. Lack of resources | 1 2 3 4 5 N/A
I think this is a good idea, and seeing as it has some thumbs up, i'll add it
from monai-deploy.
Another interesting response might be: Insufficient Clinical Demand, but I think what you have @hshuaib90 is good
from monai-deploy.
I think what you've got here is great. The only thing I'd add is how do we capture general mistrust/resistance to AI from institutions? I think safety and liability concerns could be one of the primary obstacles to deployment, but not immediately clear which category this falls under? I think probably local expertise - in which case we could use some bracketed examples e.g.
- local expertise (technical, clinical and regulatory) (1 | 2 | 3 |4 | 5)
from monai-deploy.
I think on-premise question is a good one. Should we allow some fuzzyness in options (a) and (b) by adding something like (>90% of deployments)? Let me know and I'll add it to the questionnaire.
Yes, I think that would be fine @hshuaib90
from monai-deploy.
Survey looks good!
@hshuaib90 previously you mentioned, we could incentivize different ways to contribute to this effort. Should that be included in the survey?
i.e. Would you like to contribute more in the AI survey and MONAI Deploy WG effort?
- Yes, I could contribute as a reviewers of research [your name gets listed in the acknowledgement on the published paper]
- Yes, I could contribute a response and my opinion and insights on how my institution uses AI [your institution gets acknowlement on the published paper]
- Yes, I would like to join the MONAI Deploy WG as an active participant to define the future of AI deployment across institutions and industry
- Yes, other: (suggest ways to contribute)
- No, I'd rather focus on other things
@hshuaib90 if you think this is diluting the message, feel free to leave out.
from monai-deploy.
This is really cool effort! Thank you for this work! If I may, I have a few questions:
- Do you have aspirations for future surveys?
- How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
- Would you have interest in similar surveys aimed at infosec, IT, or IRBs?
from monai-deploy.
Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly.
And this would be research hospitals.
from monai-deploy.
This is really cool effort! Thank you for this work! If I may, I have a few questions:
- Do you have aspirations for future surveys?
- How are you recruiting respondents? Wondering if we could help at all, or if you've already covered the folks we might bring along (e.g. the MICCAI crowd).
- Would you have interest in similar surveys aimed at infosec, IT, or IRBs?
Thanks @msheller ! Yes the plan is to update it annually, potentially expanding the scope.
There's a list in the appendix of the Google doc , please add to that table! Currently just went through list from International Societies of Radiology.
Yes, I think it would definitely be worthwhile exploring other dimensions around AI deployment where we can either set a standard or report an analysis of the current state.
from monai-deploy.
Ralf Floca can elaborate here and on the issue #49 that we should be careful to clarify (if not already) on the survey they we are interested in responses for research workflows as well as clinical workflows. In some regions, when mentioning clinical workflows it may limit the response to Federally approved AI workflows (like FDA algorithms in the US) vs a broader category of research applied workflows please let me know if I captured correctly. And this would be research hospitals.
Yes, in the end it revolves around the question which kind of AI workload you want to cover with the survey. If you also want to
get things that are done with AI, but not in prospective manner in the regulatory sense (e.g. because it's the research and not yet in a regulatory compliant product), you should stated more explicitly in the introduction.
And you should ask yourself about the granularity you want for those use cases. Currently you only have "retrospective study" in your list, which automatically could imply that everything else is prospectively. But then you could not differentiate between different types of projects in a retrospective setting (not sure if this is of interest for you though); e.g. in context of the research project you also could run "emergency diagnostic" in a "retrospective" setting (thus not impacting clinical decision) to see how it performs.
Hope that I got the point across.
from monai-deploy.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
from monai-deploy.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
from monai-deploy.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming.
My 2 cents.
from monai-deploy.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.
Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?
from monai-deploy.
one more thing. I would say you're missing the use case of image-based therapy planning (e.g. for image guided surgery or radiotherapy). I would this be out of scope for the survey?
Yes I think this would be more in the MONAI Stream WG remit. @ristoh @dbericat may be able to confirm. I think for future survey we might want to think about including that scope
For ig surgery that might be the case. But I think typical RT planning use cases do not fall into that regime. Only MR guided RT or gating/tracking use cases based on fluroscopy, US or alike are about "streaming". In these are in numbers not the majority compared to "classical" planning use cases. Even things like "plan of the day" is closer to "emergency diagnostic" than to streaming. My 2 cents.
Sorry @rfloca of course you are correct, I would consider RT planning in scope - i guess i see that as a medical imaging workflow (even though it does not happen in Radiology). Can you think of a good general way to describe that work so we can add as an option?
@rfloca We can add it within the Workloads document itself and we already have an open issue to add an example "streaming" use case to the Workloads document. Do you think it fits in that category?
#37
I can help with where we want to add it into the document if you can draft and initial description.
from monai-deploy.
Just wanted to share this article about information-blocking rules:
https://www.sciencedirect.com/science/article/pii/S0363018822000123
In particular, and concerning MONAI, some related articles have suggested that in order for providers to handle the volume of requests and comply with a reduced embargo period, AI might assist in generating patient-facing reports (from an AI perspective, the target being patients probably adds an extra layer of complexity, beyond the ethical issues mentioned in the article)
This goes beyond the diagnostics stages mentioned in the survey, but maybe is a good idea to keep track of it, since providers could want to add this functionality to their clinical workflows in the future.
from monai-deploy.
Approved from my end! Great team work everyone! :D
from monai-deploy.
Can you think of a good general way to describe that work so we can add as an option?
It might be a bit naive, but in the survey I would have just added the option "Image based intervention/therapie planning (e.g. for radiation therapy treatment)". Assuming that people who are using such kind of techniques know what it means and the other would not care anyways.
from monai-deploy.
Related Issues (20)
- DICOM data upload
- e2e test scenarios: KeyError: 'nifti_affine_transform' in liver_seg HOT 1
- Workflow Request Message change request HOT 2
- Rebuild of Monai Deploy Express HOT 5
- MONAI Deploy Express Workflow Manager http://localhost:5001/ gives HTTP ERROR 404 HOT 8
- Workflow won't trigger/complete on Linux server(with no GPU) HOT 13
- Minio Storage Service error HOT 3
- MD Express: Issues with .env and docker-compose file
- MD Express - Hello World - No dicom list printed with curl - MAPs not getting launched
- HL7 support HOT 6
- Deploying MONAI in Azure HOT 6
- Remove deployed workflow definition from MONAI Deploy Workflow Manager HOT 3
- MAP built with App SDK v0.6 fails when running on MONAI Deploy Express HOT 8
- Automated functional e2e test
- MD Express – Hello World example – no file list output HOT 8
- MD Express error when starting from within a docker HOT 5
- MD Express dicom seg is flipped when displayed on OHIF viewer HOT 7
- MD Express Request a way to debug track jobs HOT 2
- MD Express: Request a way to clean up data from incoming requests HOT 2
- MDExpress: Orthanc Configurations HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from monai-deploy.