jeffreydf/kaggle_diabetic_retinopathy

Fifth place solution of the Kaggle Diabetic Retinopathy competition.

kaggle_diabetic_retinopathy's Issues

training data

Hello Jeffrey
Thank you for your sharing in this project. I want to learn your method but the training data is unavailable. Could you provide the data ? Thanks !!!

How to make single predictions?

Hi Dear Jeffery

In your IPython notebook, you have given instructions for predicting a set of images using a GPU implementation. However, I have used the weights and loaded the data into the network and want to make predictions for images one at a time, since I do not have access to GPUs.

I think this might be possible by the following code, but it is not! Could you please help!

Thanks in advance

img = np.array(Image.open(r'/home/ali/Desktop/data/train/1099_right.jpeg')).T / 255.0
xs_shared[0].set_value(img[np.newaxis, :, :, :])
xs_shared[1].set_value(np.matrix([[512, 512]]))
compute_output(0)

questions about the datasets provided by the Kaggle

Sorry to bother, and maybe too late, but I did encounter a question about the datasets, not sure how you labeld the train datasets? cause what I got for train datasets look like this :

and train.zip.001 contains 35126 images , while
train.zip.002 8394
train.zip.003 8482
train.zip.004 8415
train.zip.001 1427
I didn't figure out how the data is labeld ?
any help will be hihgly appreciated.

How to load weights of the model for a single CPU?

Hi there,

I have modified the code, so it can run on a single CPU.

In file basic_model.py, comment line 7

from lasagne.layers import dnn

and change lines 136-137 as follows

    Conv2DLayer = nn.layers.Conv2DLayer #dnn.Conv2DDNNLayer
    MaxPool2DLayer = nn.layers.MaxPool2DLayer #dnn.MaxPool2DDNNLayer

Then, the following code is working without any errors, however, produces a set non-related and random weights! Could you please help to load the weights?

# coding: utf-8

# In[1]:

import sys
sys.path.append('../')


import cPickle as pickle
import re
import glob
import os

import time

import theano
import theano.tensor as T
import numpy as np
import pandas as p
import lasagne as nn

from PIL import Image

from utils import hms, architecture_string, get_img_ids_from_iter


# In[2]:

get_ipython().magic(u'pylab inline')
rcParams['figure.figsize'] = 16, 6
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)


# In[3]:

dump_path = '../dumps/2015_07_17_123003_PARAMSDUMP.pkl'


# In[4]:

model_data = pickle.load(open(dump_path, 'r'))


# In[5]:

from models import basic_model as model


# In[6]:

LEARNING_RATE_SCHEDULE = model.LEARNING_RATE_SCHEDULE
prefix_train = model.prefix_train if hasattr(model, 'prefix_train') else     '/run/shm/train_ds2_crop/'
prefix_test = model.prefix_test if hasattr(model, 'prefix_test') else     '/run/shm/test_ds2_crop/'
SEED = model.SEED if hasattr(model, 'SEED') else 11111

id_train, y_train = model.id_train, model.y_train
id_valid, y_valid = model.id_valid, model.y_valid
id_train_oversample = model.id_train_oversample,
labels_train_oversample = model.labels_train_oversample

sample_coefs = model.sample_coefs if hasattr(model, 'sample_coefs')     else [0, 7, 3, 22, 25]

l_out, l_ins = model.build_model()


# In[7]:

params = nn.layers.get_all_param_values(l_out)
for p, v in zip(params, model_data):
    p = v


# In[8]:

chunk_size = 64
batch_size = 256


# In[9]:

output = nn.layers.get_output(l_out, deterministic=True)
input_ndims = [len(nn.layers.get_output_shape(l_in))
               for l_in in l_ins]
xs_shared = [nn.utils.shared_empty(dim=ndim)
             for ndim in input_ndims]


# In[10]:

import pandas as pd
OriginalLabels = pd.read_csv(r'../data/trainLabels.csv', sep=',')


# In[11]:

import glob
temp1 = np.zeros((chunk_size, 3, 512, 512), dtype='float64')
fileList = sorted(glob.glob(r'/home/ali/Desktop/kaggle_diabetic_retinopathy-master/data/64/*.tiff'))
labels64 = []
for i, f in enumerate(fileList):
    temp1[i,:,:,:] = np.array(Image.open(f)).T / 255.0
    fname = f.split('/')[-1].split('.')[0]
    lbl = OriginalLabels.loc[OriginalLabels['image'] == fname]['level'].values.item(0)
#     print lbl
    labels64.append(lbl)
xs_shared[0].set_value(temp1)
print temp1.shape


# In[12]:

temp2 = np.ones((chunk_size, 2), dtype='float64') * 512
xs_shared[1].set_value(temp2)
temp2.shape


# In[13]:

idx = T.lscalar('idx')

givens = {}
for l_in, x_shared in zip(l_ins, xs_shared):
    givens[l_in.input_var] = x_shared[idx * batch_size:(idx + 1) * batch_size]

compute_output = theano.function(
    [idx],
    output,
    givens=givens,
    on_unused_input='ignore'
)


# In[14]:

# import pandas as pd
# OriginalLabels = pd.read_csv(r'../data/trainLabels.csv', sep=',')


# In[15]:

# import glob
# temp1 = np.zeros((chunk_size, 3, 512, 512), dtype='float64')
# fileList = glob.glob(r'/home/ali/Desktop/kaggle_diabetic_retinopathy-master/data/64/*.tiff')
# labels64 = []
# for i, f in enumerate(fileList):
#     temp1[i,:,:,:] = np.array(Image.open(f)).T / 255.0
#     fname = f.split('/')[-1].split('.')[0]
#     lbl = OriginalLabels.loc[OriginalLabels['image'] == fname]['level'].values.item(0)
# #     print lbl
#     labels64.append(lbl)
# xs_shared[0].set_value(temp1)
# print temp1.shape


# In[16]:

# temp2 = np.ones((chunk_size, 2), dtype='float64') * 512
# xs_shared[1].set_value(temp2)
# temp2.shape


# In[17]:

get_ipython().magic(u'time predictions = compute_output(0)')


# In[18]:

results = []
for i, p in enumerate(predictions):
    resul = dict()
    resul['fileName'] = fileList[i].split('/')[-1].split('.')[0]
    resul['original'] = labels64[i]
    resul['pred'] = p
    results.append(resul)


# In[19]:

for i in results:
    print i


# In[20]:

len(model_data)

No file in the directory data/train_ds5_crop

Dear Jeffrey:
I'm so sorry to bother you.I download the file from this website, and follow the instruction in your ipython notebook, but get some problem. The generators.py include some code about prefix:

def patches_gen_pairs_pseudolabel(images, labels, p_x=80, p_y=80, num_channels=3, chunk_size=1024, num_chunks=100, rng=np.random, prefix_train='data/train_ds5_crop/', prefix_test='data/test_ds5_crop/', transfo_params=None, paired_transfos=False): num_patients = len(images)

but there is no file named train_ds5_crop and test_ds5_crop, and only tranlabel.

what should i do?

I'm new to Deep Learning .Thanks a lot if you can help me , sincerely!

Shawn