omar-mohamed/x-ray-report-generation

This is the implementation of the 'VSGRU' model mentioned in our paper 'Automated Radiology Report Generation using Conditioned Transformers'.

X-Ray-Report-Generation (VSGRU)

This is the implementation of the 'VSGRU' model mentioned in our paper 'Automated Radiology Report Generation using Conditioned Transformers'.

Paper link here.

We automatically generate full radiology reports given chest X-ray images from the IU-X-Ray dataset by conditioning a recurrent neural net on the visual and semantic features of the image.

Installation & Usage

The project was tested on a virtual environment of python 3.7, pip 23.2.1, and MacOS

• pip install -r full_requirements.txt (or pip install -r requirements.txt if there are errors because of using a different operating system, as requirements.txt only contains the main dependencies and pip will fetch the compatible sub-dependencies, but it will be slower)
• nlg-eval --setup
• python get_iu_xray.py (to download the dataset)
• python train.py

Related Repositories

• CDGPT2 repository (main paper repo) here.
• Finetuned Chexnet repository here.

Citation

To cite this paper, please use:

@article{ALFARGHALY2021100557,
title = {Automated radiology report generation using conditioned transformers},
journal = {Informatics in Medicine Unlocked},
volume = {24},
pages = {100557},
year = {2021},
issn = {2352-9148},
doi = {https://doi.org/10.1016/j.imu.2021.100557},
url = {https://www.sciencedirect.com/science/article/pii/S2352914821000472},
author = {Omar Alfarghaly and Rana Khaled and Abeer Elkorany and Maha Helal and Aly Fahmy}
}

x-ray-report-generation's Issues

Getting error RNNDecoder class in call(self, x, features, hidden)

Hi,

Getting error "InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [90,1,400] vs. shape[1] = [1,1,400] [Op:ConcatV2] name: concat" in RNNDecoder.ipynb in call(self, x, features, hidden) probably giving the error on this line x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1) but not sure why there is difference in shapes of embedding and features which is giving error and rest all code has executed fine

This is happening when I call below to train my traning corpus from train.py
evaluate_enqueuer(train_enqueuer, train_steps, FLAGS, encoder, decoder, tokenizer_wrapper, chexnet)

Why are we not first training the model rather evaluating on a test set ?

Additionally, when I use augmenter in the below statement
train_enqueuer, train_steps = get_enqueuer(FLAGS.train_csv, FLAGS.batch_size, FLAGS, tokenizer_wrapper, augmenter)
I get the below error in Generator.ipynb in transform_batch_images(self, batch_x)
"Image Correction methods work correctly only on images with non-negative values. Use skimage.exposure.rescale_intensity"

Unable to locate the 'config.ini' file

Running the predict.py but is giving me the error as such

Traceback (most recent call last):
  File "/root/BE_project2/X-Ray-Report-Generation/predict.py", line 84, in <module>
    main()
  File "/root/BE_project2/X-Ray-Report-Generation/predict.py", line 19, in main
    chexnet_class_names = cp["DEFAULT"].get("chexnet_class_names").split(",")
AttributeError: 'NoneType' object has no attribute 'split'

Can you provide the config.ini file location or the file itself.

JSON file not found

FileNotFoundError: [Errno 2] No such file or directory: 'pretrained_models\fine_tuned_chexnet.json'

getting this error i have tried creating a dummy json file then i get this

Traceback (most recent call last):
  File "train.py", line 91, in <module>
    chexnet = ChexnetWrapper('pretrained_models', FLAGS.visual_model_name, FLAGS.visual_model_pop_layers)
  File "F:\imagecaption\X-Ray-Report-Generation\chexnet_wrapper.py", line 7, in __init__
    model = load_model(model_path, model_name)
  File "F:\imagecaption\X-Ray-Report-Generation\utility.py", line 85, in load_model
    loaded_model = model_from_json(loaded_model_json)
  File "F:\imagecaption\X-Ray-Report-Generation\x-rayenv\lib\site-packages\tensorflow_core\python\keras\saving\model_config.py", line 94, in model_from_json
    config = json.loads(json_string)
  File "c:\users\work\appdata\local\programs\python\python37\Lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "c:\users\work\appdata\local\programs\python\python37\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "c:\users\work\appdata\local\programs\python\python37\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

please help me resolve this!

thanks in advance