gersteinlab/medagents

[ACL 2024 Findings] MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning

📖 Paper

We propose a Multi-disciplinary Collaboration (MC) framework. The framework works in five stages: (i) expert gathering: gather experts from distinct disciplines according to the clinical question; (ii) analysis proposition: domain experts put forward their own analysis with their expertise; (iii) report summarization: compose a summarized report on the basis of a previous series of analyses; (iv) collaborative consultation: engage the experts in discussions over the summarized report. The report will be revised iteratively until an agreement from all the experts is reached; (v) decision making: derive a final decision from the unanimous report.

Requirements

Install all required python dependencies:

pip install -r requirements.txt

Data

We evaluate our MC framework on two benchmark datasets MedQA, MedMCQA, and PubMedQA, as well as six subtasks most relevant to the medical domain from MMLU datasets including anatomy, clinical knowledge, college medicine, medical genetics, professional medicine, and college biology.

Please check our Google Drive: https://drive.google.com/file/d/11qNzDYIlimGGJ1fhQn2ux6w_rfFgJbyo/view?usp=sharing

Implementations

Input your own openai api key in api_utils.py.

sh inference.sh

Cite Us

If you find this project useful, please cite the following paper:

@article{tang2023MedAgents,
      title={ML-MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning}, 
      author={Xiangru Tang and Anni Zou and Zhuosheng Zhang and Yilun Zhao and Xingyao Zhang and Arman Cohan and Mark Gerstein},
      year={2023},
      journal={arXiv preprint arXiv:2311.10537},
}

medagents's Issues

What did you do for the evaluation?

hi,What did you do for the evaluation?

no `requirements.txt` file?

How long did it take you to do these evaluation?

Hello, I am running MedAgents' code and find it slow to get one answer (One question takes about 2~3 minutes). So I wonder if there is any method to speed it up. And how do you evaluate your methods on those datasets? How much time did it take?

实验不足

为什么实验结果没有与gpt4进行对比，是因为MC framework的推理能力并不如GPT4的zero-shot, zero-shot cot, few-shot,few-shot cot 和 few- shot cot + sc吗？