gregwchase/eyenet

Identifying diabetic retinopathy using convolutional neural networks

EyeNet

Detecting Diabetic Retinopathy With Deep Learning

Objective

Diabetic retinopathy is the leading cause of blindness in the working-age population of the developed world. The condition is estimated to affect over 93 million people.

The need for a comprehensive and automated method of diabetic retinopathy screening has long been recognized, and previous efforts have made good progress using image classification, pattern recognition, and machine learning. With photos of eyes as input, the goal of this capstone is to create a new model, ideally resulting in realistic clinical potential.

The motivations for this project are twofold:

• Image classification has been a personal interest for years, in addition to classification on a large scale data set.

• Time is lost between patients getting their eyes scanned (shown below), having their images analyzed by doctors, and scheduling a follow-up appointment. By processing images in real-time, EyeNet would allow people to seek & schedule treatment the same day.

Table of Contents

1. Data
2. Exploratory Data Analysis
3. Preprocessing
   • Download Images to EC2
   • Crop & Resize Images
   • Rotate and Mirror All Images
4. CNN Architecture
5. Results
6. Next Steps
7. References

Data

The data originates from a 2015 Kaggle competition. However, is an atypical Kaggle dataset. In most Kaggle competitions, the data has already been cleaned, giving the data scientist very little to preprocess. With this dataset, this isn't the case.

All images are taken of different people, using different cameras, and of different sizes. Pertaining to the preprocessing section, this data is extremely noisy, and requires multiple preprocessing steps to get all images to a useable format for training a model.

The training data is comprised of 35,126 images, which are augmented during preprocessing.

Exploratory Data Analysis

The very first item analyzed was the training labels. While there are five categories to predict against, the plot below shows the severe class imbalance in the original dataset.

Of the original training data, 25,810 images are classified as not having retinopathy, while 9,316 are classified as having retinopathy.

Due to the class imbalance, steps taken during preprocessing in order to rectify the imbalance, and when training the model.

Furthermore, the variance between images of the eyes is extremely high. The first two rows of images show class 0 (no retinopathy); the second two rows show class 4 (proliferative retinopathy).

Preprocessing

The preprocessing pipeline is the following:

1. Download all images to EC2 using the download script.
2. Crop & resize all images using the resizing script and the preprocessing script.
3. Rotate & mirror all images using the rotation script.
4. Convert all images to array of NumPy arrays, using the conversion script.

Download All Images to EC2

The images were downloaded using the Kaggle CLI. Running this on an EC2 instance allows you to download the images in about 30 minutes. All images are then placed in their respective folders, and expanded from their compressed files. In total, the original dataset totals 35 gigabytes.

Crop and Resize All Images

All images were scaled down to 256 by 256. Despite taking longer to train, the detail present in photos of this size is much greater then at 128 by 128.

Additionally, 403 images were dropped from the training set. Scikit-Image raised multiple warnings during resizing, due to these images having no color space. Because of this, any images that were completely black were removed from the training data.

Rotate and Mirror All Images

All images were rotated and mirrored.Images without retinopathy were mirrored; images that had retinopathy were mirrored, and rotated 90, 120, 180, and 270 degrees.

The first images show two pairs of eyes, along with the black borders. Notice in the cropping and rotations how the majority of noise is removed.

After rotations and mirroring, the class imbalance is rectified, with a few thousand more images having retinopathy. In total, there are 106,386 images being processed by the neural network.

Neural Network Architecture

The model is built using Keras, utilizing TensorFlow as the backend. TensorFlow was chosen as the backend due to better performance over Theano, and the ability to visualize the neural network using TensorBoard.

For predicting two categories, EyeNet utilizes three convolutional layers, each having a depth of 32. A Max Pooling layer is applied after all three convolutional layers with size (2,2).

After pooling, the data is fed through a single dense layer of size 128, and finally to the output layer, consisting of 2 softmax nodes.

Results

The EyeNet classifier was created to determine if a patient has retinopathy. The current model returns the following scores.

Metric	Value
Accuracy (Train)	82%
Accuracy (Test)	80%
Precision	88%
Recall	77%

So, why does the neural network perform this way? Besides the class imbalance, the cropping is definitely helping in the network's performance. By not having extra black parts in the images, the network is able to process only the eye itself.

Next Steps

1. Program the neural network to retrain with new photos. This is a common practice, and only serves to optimize the model. Checks would be put in place to validate the images before being added to the classifier, in order to prevent low quality images from altering the classifier too drastically.

2. Port the Keras model to CoreML, and deploy to an EyeNet iOS application. CoreML is a framework designed by Apple for adding machine learning to iOS devices. This allows the ability of Python developers to export their models, convert the file to a .mlmodel file, and add the file to the iOS development cycle.

Furthermore, the model is able to perform classification on the local device. There is no need for an internet connection for the application to work. Because of this, the ability to use EyeNet in remote areas is further justified, and that much easier.

References

1. What is Diabetic Retinopathy?

2. Diabetic Retinopathy Winners' Interview: 4th place, Julian & Daniel

3. TensorFlow: Machine Learning For Everyone

Tech Stack

eyenet's Issues

Files missing 'X_train_256_v2.npy'

Hi Greg, i'm trying eyenet-2020, but when i run cnn.py, i'm getting this error:

No such file or directory: X_train_256_v2.npy. Where can i found or generate it?.
First I downloaded the data, then preprocess the images, now i'm trying to run cnn.py, am I in the correct order?. Thanks.

`X_train_256_v2.npy` saved?

In image_to_array.py I see you savedX_train.npy, but I don't see where you saved X_train_256_v2.npy even though you end up loading it as the training data in all of your CNN models. Is the code to save X_train_256_v2.npy missing?

I feel like you meant to save X_train.npy as X_train_256_v2.npy since you ended up using the labels from trainLabels_master_256_v2.csv when you called convert_images_to_arrays_train, but I am not sure.

Layer conv2d_51 was called with an input that isn't a symbolic tensor

cnn.py is showing the value error. while running the main, it showing the following error

Splitting data into test/ train datasets
Reshaping Data
X_train Shape: (422, 256, 256, 3)
X_test Shape: (106, 256, 256, 3)
Normalizing Data
y_train Shape: (422, 2)
y_test Shape: (106, 2)
Training Model
Traceback (most recent call last):

File "", line 45, in
nb_classes, nb_gpus=8)

File "", line 24, in cnn_model
input_shape=(img_rows, img_cols, channels), activation="relu"))

File "C:\Users\naveenn\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\models.py", line 467, in add
layer(x)

File "C:\Users\naveenn\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 575, in call
self.assert_input_compatibility(inputs)

File "C:\Users\naveenn\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py", line 448, in assert_input_compatibility
str(inputs) + '. All inputs to the layer '

ValueError: Layer conv2d_51 was called with an input that isn't a symbolic tensor. Received type: <class 'tensorflow.python.framework.ops.Tensor'>. Full input: [<tf.Tensor 'conv2d_51_input:0' shape=(?, 256, 256, 3) dtype=float32>]. All inputs to the layer should be tensors.

System Requirements for Model Training

Hi Greg,
I have a few queries, Please do answer

1. I am running your code and I always get Insufficient Memory or other error.I wish to know at what Configuration will your Model codes work.I am Opting for Google Cloud. So I need Exact Configuration like RAM,GPU, Processor, HDD?
2. I am running it in My laptop with 8GB RAM, 4GB Nvidia 940MX when executing cnn.py It show CUDA Allocation Error Insufficient Memory.Is it possible to run in this Laptop?
3. cnn.py,cnn_class.py,cnn_multi.py. What are the difference? What are their Individual System Requirements?How many GPU is minimum for running CNN Model?
4. Is a Confusion Matrix Implementaion Possible in your Code?
5. Is it possible to give input sample data image after Training the model completely and get predict and predict_classes?How to Do it?
6. After Training the model and saving it can we Copy the model and Run it in a Lower Configuration Laptop like Above?Can you Add Trained_Complete_Model.h5 to Repo?

Please Do reply.

file missing the file X_train_256_v2.npy can't be build?

sir,I'am trying to run eyenet-2018,here is my running series:

resize_images.py → preprocess_images.py → rotate_images.py→ reconcile_labels.py → image_to_array.py

when i finished running image_to_array.py, it build a file--X_train.npy,

but when I turn to try cnn.py, I found that it needs X_train_256_v2.npy insted of X_train.npy,

but I can't find any .py file to build this file. How can I build this file?

X_train_256_v2.npy

Hi Greg,
The X_train_256_v2.npy was implemented in all three classification files. But it was never created in the code that you provided. I only created X_train.npy file while preprocessing. By using that in cnn.py, the model is over fitting to the training data. The different implementation like categorical cnn_class.py and cnn_multi.py, but we never separated the images into 5 classes. We only did it for two classes. I am so confused Greg. Can you please help me.
Thanks,
Vamsi.

Some questions about rotate_images.py

Hi! I find that the DR dataset has been rotated 90,120,180,270 degrees, but the DM dataset hasn't been rotated. Is it matter whether to rotate or not?
Or some ways like to rotate randomly will help more for training.
I would appreciate it if you could answer my questions. Thanks a lot!

Next Steps

Sir , Can you help me out with the commands to run after running the conversion script please?

the trained model

Can you provide the .ckpt or .h5 file for this model?

GPU problem

ValueError: To call multi_gpu_model with gpus=8, we expect the following devices to be available: ['/cpu:0', '/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3', '/gpu:4', '/gpu:5', '/gpu:6', '/gpu:7']. However this machine only has: ['/cpu:0', '/xla_cpu:0', '/xla_gpu:0', '/gpu:0']. Try reducing gpus

Not getting enough accuracy

When I tried to train on the whole dataset which of almost 20Gb's then I ran out of Memory! so I split the dataset into 4 batches each contain 26600 images i.e 26600 + 26600 + 26600 + 26586 = 106,386. And so I have to split the dataset into 4 batches and make a slight adjustment in the code.

To load the save model with trained weights from the previous batch I use the keas load_weight() method. Here, I'm working on cnn.py for all 5 classes

model.add(Dense(nb_classes, activation = 'softmax'))
model.summary()

model.load_weights(model_name + '.h5' )

model.compile(loss = 'categorical_crossentropy', optimizer = 'adam',
                    metrics = ['accuracy'])

When I train on the first batch which contain 26600 images I got

loss: 1.0042 - acc: 0.6248 - val_loss: 1.0625 - val_acc: 0.6029

For the second batch of 26600 images I got

loss: 0.9026 - acc: 0.6563 - val_loss: 1.1008 - val_acc: 0.6114

For the third batch of 26600 images I got

loss: 0.8860 - acc: 0.6666 - val_loss: 0.9988 - val_acc: 0.6330

For the fourth batch of 26586 images I got

loss: 0.8227 - acc: 0.6888 - val_loss: 1.0289 - val_acc: 0.6356

Question 1: If you see there is not a much significant change in the score. Can you identify where's the problem is occurring? If you want then I can provide you the code which I have slightly altered from the original.

Question 2: As I have split the dataset into individual .npy arrays could this be a reason for not seeing much improvement in the score?

Question 3; As you mentioned in previous issues that you train on p2.8x large AWS instance. If I train on the same instance how much time does it takes to train the whole network?

Question 4: You have also mentioned that you use the VGG arch but the VGG contain more layers then you have used in cnn.py OR cnn_multi.py could it be the reason that model is not extracting enough feature to learn?

Question 5: When I try to train the cnn.py for binary classification on the first batch which contains 26600 images then I got the 99% accuracy after epoch which shows that model is obviously overfitting. Again, As I have split the dataset into individual arrays could this be the reason for getting 99% accuracy?

O/p after first epoch using binary classification :

loss: 0.0088 - acc: 0.9934 - val_loss: 8.1185e-05 - val_acc: 1.0000

Thanks! Please do answer Sir! :)

rotation script

Hello gregwchase, can you please tell me which data is stored in trainLabels_master.csv?

IOError: [Errno 2] No such file or directory: '../data/train-resized-256/19_left.jpeg'

Hi I am running the !python preprocess_images.py
I used training.zip.001 and test.zip.001 only as per the script "resize_images.py"
Should i have to process / rotate all the train and test images before running the pre-process ?
TRACE of running preprocess_images.py

Started Image preprocess
Traceback (most recent call last):
File "preprocess_images.py", line 31, in
trainLabels['black'] = find_black_images('../data/train-resized-256/', trainLabels)
File "preprocess_images.py", line 21, in find_black_images
return [1 if np.mean(np.array(Image.open(file_path + img))) == 0 else 0 for img in lst_imgs]
File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2312, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: '../data/train-resized-256/19_left.jpeg'
Sat Oct 20 08:42:36 UTC 2018
Ended Image preprocess

MemoryError

Hi,my configure is 👍
ubuntu10.04
python3
but,when I run the scipt:image_to_array.py,
the error occur.
Writing Train Array Traceback (most recent call last): File "image_to_array.py", line 62, in <module> X_train = convert_images_to_arrays_train('../data/train-resized-256/', labels) File "image_to_array.py", line 38, in convert_images_to_arrays_train return np.array([np.array(Image.open(file_path + img)) for img in lst_imgs]) MemoryError

Thanks in advance.

question

Which algorithm can be used for detecting Diabetic Retinopathy using image processing algorithms? in python