selimsef/dsb2018_topcoders

DSB2018 [ods.ai] topcoders

DSB2018 [ods.ai] topcoders 1st place solution

Model weights/annotated data

You can download the whole package (7.5G) that contains training data, nn models weights, GBT models.

See also solution description on Kaggle

You need to setup your environment first. Please install latest nvidia drivers, cuda 9 and cudnn 7. After that run setup_env.sh script

How to run predict

unzip test data into data_test folder and

./predict_test.sh

Submission files will be in predictions folder (submission_0.csv, submission_1.csv). Individual model predictions will be also in predictions folder.

How to run training

Before training please remove models from:

• albu/weights
• selim/nn_models
• victor/nn_models
• victor/lgbm_models

after it run:

./train_all.sh
./tune_all.sh
./predict_oof_trees.sh

We use two stage training because we want to tune models on the stage1 data released 11.04. Every script goes into every folder and runs scripts to train models.

dsb2018_topcoders's Issues

Which Keras version are you using for the models in folder Selim?

How to mark nucleus with rectangles?

Hi @selimsef
Thanks for your help, Now I can run the project completely. But I wish all nucleus in output images can be marked with rectangles! Can you give me some ideas? Waiting your reply!

unable to reproduce the masks

Hi, I'm trying to generate the masks for training data using your code below. Just to be sure that I am doing everything correctly I compared the mask generated from this code and the mask present in the already existing dataset that you provided via the dropbox link. They aren't quire right. The separation between nuclei is not fully done. My major suspicion is if the changes in the versions of the libraries used has affected this or I'm making a silly mistake somewhere. Could you please let me know where I'm going wrong ?

`

def create_mask(img_id):
    labels = cv2.imread(path.join(labels_folder, '{0}.tif'.format(img_id)), cv2.IMREAD_UNCHANGED)
    tmp = dilation(labels > 0, square(9))    
    tmp2 = watershed(tmp, labels, mask=tmp, watershed_line=True) > 0
    tmp = tmp ^ tmp2
    tmp = dilation(tmp, square(7))
    msk = (255 * tmp).astype('uint8')

    props = measure.regionprops(labels)
    msk0 = 255 * (labels > 0)
    msk0 = msk0.astype('uint8')

    msk1 = np.zeros_like(labels, dtype='bool')

    max_area = np.max([p.area for p in props])

    for y0 in range(labels.shape[0]):
        for x0 in range(labels.shape[1]):
            if not tmp[y0, x0]:
                continue
            if labels[y0, x0] == 0:
                if max_area > 4000:
                    sz = 6
                else:
                    sz = 3
            else:
                sz = 3
                if props[labels[y0, x0] - 1].area < 300:
                    sz = 1
                elif props[labels[y0, x0] - 1].area < 2000:
                    sz = 2
            uniq = np.unique(labels[max(0, y0-sz):min(labels.shape[0], y0+sz+1), max(0, x0-sz):min(labels.shape[1], x0+sz+1)])
            if len(uniq[uniq > 0]) > 1:
                msk1[y0, x0] = True
                msk0[y0, x0] = 0

    msk1 = 255 * msk1
    msk1 = msk1.astype('uint8')

    msk2 = np.zeros_like(labels, dtype='uint8')
    msk = np.stack((msk0, msk1, msk2))
    msk = np.rollaxis(msk, 0, 3)

    cv2.imwrite(path.join(masks_out_folder, '{0}.png'.format(img_id)), msk, [cv2.IMWRITE_PNG_COMPRESSION, 9])
    return 0

`

id : 00ae65c1c6631ae6f2be1a449902976e6eb8483bf6b0740d00530220832c6d3e
platform : google colab

After running the code:

Directly from the dataset :

how to set "dirs"?

like "train_dir", "mask_dir" and "label_dir" in train.py (selim)
it seems only stage1_train and stage1_test.

License for this repo

Hi, thanks for sharing the code and I find it's helpful to reproduce the results. However, I cannot find a license, could you please add one?

where are the tif format labels?

I'm quite confused about your code lbl = cv2.imread(os.path.join(labels_dir, '{0}.tif'.format(img_id)), cv2.IMREAD_UNCHANGED) in dsb_binary.py, because I can't find any tif format labels in the data that Kaggle provided, could you please tell me how did you preprocess the data?

ImportError: cannot import name 'inception_resnet_block'

hi, thank you very much for you codes, when I predict on the test data, get a ImportError:
from keras.applications.inception_resnet_v2 import InceptionResNetV2, inception_resnet_block, conv2d_bn
ImportError: cannot import name 'inception_resnet_block'

Loss calculation has an index error

Hello, I am very happy to see your code, I am trying to implement it, I downloaded the cells and data through Kaggle, and got the mask and label, I executed /albu/src/train_all.bash according to your instructions, but I got the following error:

Traceback (most recent call last):
  File "bowl_train.py", line 120, in <module>
    train_bowl()
  File "bowl_train.py", line 97, in train_bowl
    train(ds, val_ds, fold, train_idx, val_idx, config, num_workers=num_workers, transforms=aug_victor(.97))
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 345, in train
    trainer.fit(train_loader, val_loader, config.nb_epoch)
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 290, in fit
    self.metrics_collection.train_metrics = self._run_one_epoch(epoch, train_loader, training=True)
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 256, in _run_one_epoch
    meter, ypreds = self._make_step(data, training)
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 276, in _make_step
    meter, ypreds = self.estimator.make_step_itersize(images, ytrues, training)
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 207, in make_step_itersize
    meter = self.calculate_loss_softmax(output, target, meter, training, iter_size)
  File "/scratch2/hzhou/DPNUnet/dsb2018_topcoders/albu/src/pytorch_utils/train.py", line 135, in calculate_loss_softmax
    meter['loss'] += loss.data.cpu().numpy()[0]
IndexError: too many indices for array

I output a loss.data.cpu().numpy(), I get a number instead of an array or list:

Why is this reason? Can you guide me in detail?

AttributeError: 'NoneType' object has no attribute 'Conv2D'

while i am trying to execute predict_test.sh in victor folder
I am getting issue :
in predict_inception.py
---> model = get_inception_resnet_v2_unet_softmax((None,None), weights=None)

in models.py
def get_inception_resnet_v2_unet_softmax(input_shape, weights='imagenet'):
inp = Input(input_shape + (4,))
print("inp",inp)
# Stem block: 35 x 35 x 192
---> x = conv2d_bn(inp, 32, 3, strides=2, padding='same')
x = conv2d_bn(x, 32, 3, padding='same')

auxiliary files: data/folds.csv

Can we also get auxiliary files like data/folds.csv needed to run training? Thank you for supporting this repo!

The dropbox link was broken?

Thank you for a great job.
I tried to download the dropbox link that you mentioned README.

I just want to see what the dataset (input masks, and their channel) looks like.
Can you share a few samples of the dataset?
Thank you.

lgbm_models

Hello,

i saw the explanation about lgbm model (2nd model) in the link in kaggle. However, i still dont understand it well. And there is no lgbm_model_{0}_{1}.txt. So if you could provide the file or elaborate how to make this file, it will be great. What is the train and label like? Thanks.

Empty nn_models

selim# ./predict_test.sh, got error with build model:.... "Building model resnet101_2 from weights nn_models/best_resnet101_2_fold0.h5"

Check on nn_models, there is nothing...

My question: would you point where can I find the download link?

Unused BCEDiceJaccardLoss in loss.py

The class BCEDiceJaccardLoss defined in dsb2018_topcoders/albu/src/pytorch_utils/loss.py is never used. Eventhough, it, perhaps, contains a mistake: it estimates nn.BCEWithLogitsLoss which combines a Sigmoid layer and the BCELoss in one single class. In the same way BCEDiceJaccardLoss computes JaccardLoss and DiceLoss which expects an input to be processed with sigmoid.
A simple solution could be to change this lines of code on the following:

sigmoid_input = torch.sigmoid(input)
for k, v in self.weights.items():
    if not v: 
        continue
    val = self.mapping[k](input if k == 'bce' else sigmoid_input, target)

permission denied error

permission denied error when i run the code from google colab after clonning repo in the colab notebooks.please help

question

what is your keras version?

what is folds.csv?

Hello,

i was looking at your code. There is one file, folds.csv. What is this like? Can you upload one example here? Thank you. I saw there are four information inside the file. 'img_id',` 'fold', 'cluster', 'source'
i can understand the first one, but not the rest of them. If you can provide one example, it will be great. Thanks

How do the three input folders correspond to each other?

In this code, i found there have three input folders: images_all, masks_all and labels_all,
could U please help me in detail how to set the original images and masks to these folders?
because i found that a image correspond many masks.