iriscxy/genfair

GenFair

Overview

This repository contains multiple components related to medical report summarization and evaluation. It includes both baseline and proposed methods for generating and evaluating reports, as well as test datasets.

Directory Structure

• BART-summarization

  • This directory contains the code for the baseline report summarization using BART. It focuses on generating summaries from medical reports.

• BART-our-proposed

  • This directory includes our proposed method for report summarization, which is based on the BART-large model. It contains enhancements over the baseline to improve the quality of the generated summaries.

• R2Gen-main

  • Contains the code for the baseline report generation approach named R2Gen. This serves as a baseline method for generating medical reports from the given data.

• R2Gen-our-proposed

  • Includes the code for our proposed improvements to the R2Gen model, aimed at enhancing the quality and accuracy of the generated reports.

• report_generation_baseline_results

  • This folder contains the results from the baseline report generation methods. It includes the generated reports which can be compared against the ground truth or other models.

• report_summarization_baseline_results

  • This folder contains the results from the baseline report summarization methods. It includes the generated reports which can be compared against the ground truth or other models.

• chexpert-evaluation

  • Contains the evaluation scripts for assessing the Chexpert score of the generated medical reports. The Chexpert score is used to evaluate the clinical relevance and correctness of the generated content.

• test_image.json

  • This file is the test dataset used in the evaluation. Each line in the file is a JSON object containing the input findings and the corresponding output impression generated by the models.

How to Use

Report Generation: For R2Gen-main and R2Gen-our-proposed, the train code is

python main_train.py \
--image_dir no_use \
--ann_path annotation.json \
--dataset_name mimic_cxr \
--max_seq_length 100 \
--threshold 10 \
--batch_size 16 \
--epochs 30 \
--save_dir results/RGen14 \
--step_size 1 \
--gamma 0.8 

Evaluation: To decode and evaluate the model, use the following command:

python main_test.py \
        --image_dir data/mimic_cxr/images/ \
        --ann_path annotation.json \
        --dataset_name mimic_cxr \
        --max_seq_length 100 \
        --threshold 10 \
        --batch_size 16 \
        --epochs 30 \
        --save_dir results/RGen \
        --step_size 1 \
        --gamma 0.8 \
        --load checkpoint.pth

Report Summarization: For BART-summarization and BART-our-proposed, the train code is

python3 run_mybart.py --model_name_or_path facebook/bart-base \
                      --do_train --output_dir das \
                      --exp_name exp_name \
                      --max_source_length 1024 --max_target_length 300 \
                      --save_dataset_path withlabel\
                       --num_train_epochs 100 \
                      --per_device_train_batch_size 8 --save_strategy epoch  \
                      --label_smoothing_factor 0.1 --weight_decay 0.01 \
                      --max_grad_norm 0.1 --warmup_steps 500\
                      --gradient_accumulation_steps 8 \
                      --lr_scheduler_type polynomial --learning_rate 3e-05 

Evaluation: To decode and evaluate the model, use the following command:

CUDA_VISIBLE_DEVICES=0 python3 run_mybart.py \
    --per_device_eval_batch_size 64 \
    --log_root ./log \
    --save_dataset_path $dataset \
    --exp_name $proj_name \
    --do_predict --predict_with_generate True \
    --output_dir das \
    --val_max_target_length 100 \
    --model_name_or_path checkpoint

Use the chexpert-evaluation directory to evaluate the Chexpert score of the generated reports, which helps in assessing their clinical relevance.