This repository is the implementation of ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning. Before running the code, please install the prerequisite libraries, and follow Step 0, Step 1, and Step 2 to replicate the experiments.
- [2023/11/15] Checkout our EMNLP'23 Findings paper about progression modeling: RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
- [2023/10/31] Pretained checkpoint for the IU X-ray dataset is available at Google Drive
This paper explores the task of radiology report generation, which aims at generating free-text descriptions for a set of radiographs. One significant challenge of this task is how to correctly maintain the consistency between the images and the lengthy report. Previous research explored solving this issue through planning-based methods, which generate reports only based on high-level plans. However, these plans usually only contain the major observations from the radiographs (e.g., lung opacity), lacking much necessary information, such as the observation characteristics and preliminary clinical diagnoses. To address this problem, the system should also take the image information into account together with the textual plan and perform stronger reasoning during the generation process. In this paper, we propose an Observation-guided radiology Report Generation framework (ORGan). It first produces an observation plan and then feeds both the plan and radiographs for report generation, where an observation graph and a tree reasoning mechanism are adopted to precisely enrich the plan information by capturing the multi-formats of each observation. Experimental results demonstrate that our framework outperforms previous state-of-the-art methods regarding text quality and clinical efficacy.
torch==1.9.1torchvision==0.10.1transformers==4.15.0
Pretrained model weight for the IU X-ray dataset: Google Drive
Trained model weights on the two datasets are available at:
- IU X-ray: Google Drive
- MIMIC-CXR: Google Drive
Please download the two datasets: IU X-ray and MIMIC-CXR. For observation preprocessing, we use CheXbert to extract relevant observation information. Please follow the instruction to extract the observation tags.
chmod +x ./src/graph_construction/run_iu_xray.sh
./src/graph_construction/run_iu_xray.sh
cd ORGan
python ./src/plan_extraction.py
There are two parameters required to run the code of the planner:
debug: whether debugging the code (0 for debugging and 1 for running)checkpoint_name: indicating the location for the pre-trained visual model, mainly for IU X-ray dataset.
chmod +x ./script_plan/run_iu_xray.sh
./script_plan/run_iu_xray.sh debug checkpoint_name
There are four parameters required to run the code of the generator:
debug: whether debugging the code (0 for debugging and 1 for running)checkpoint_name: indicating the location for the pre-trained visual model, mainly for the IU－Xray dataset, same as the setting of the plannerplan_model_name_or_path: indicating the location of the trained planner (from Step 1)plan_eval_file: indicating the file name of generated plans for the validation set (from Step 1)
chmod +x ./script/run_iu_xray.sh
./script/run_iu_xray.sh debug checkpoint_name plan_model_name_or_path plan_eval_file
If you use the ORGan, please cite our paper:
@inproceedings{hou-etal-2023-organ,
title = "{ORGAN}: Observation-Guided Radiology Report Generation via Tree Reasoning",
author = "Hou, Wenjun and Xu, Kaishuai and Cheng, Yi and Li, Wenjie and Liu, Jiang",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-long.451",
doi = "10.18653/v1/2023.acl-long.451",
pages = "8108--8122",
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent