vqdang/hover_net

Simultaneous Nuclear Instance Segmentation and Classification in H&E Histology Images.

HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images

A multiple branch network that performs nuclear instance segmentation and classification within a single network. The network leverages the horizontal and vertical distances of nuclear pixels to their centres of mass to separate clustered cells. A dedicated up-sampling branch is used to classify the nuclear type for each segmented instance.

Link to Medical Image Analysis paper.

This is the official PyTorch implementation of HoVer-Net. For the original TensorFlow version of this code, please refer to this branch. The repository can be used for training HoVer-Net and to process image tiles or whole-slide images. As part of this repository, we supply model weights trained on the following datasets:

• CoNSeP
• PanNuke
• MoNuSAC
• Kumar
• CPM17

Links to the checkpoints can be found in the inference description below.

Set Up Environment

conda env create -f environment.yml
conda activate hovernet
pip install torch==1.6.0 torchvision==0.7.0

Above, we install PyTorch version 1.6 with CUDA 10.2.

Repository Structure

Below are the main directories in the repository:

• dataloader/: the data loader and augmentation pipeline
• docs/: figures/GIFs used in the repo
• metrics/: scripts for metric calculation
• misc/: utils that are
• models/: model definition, along with the main run step and hyperparameter settings
• run_utils/: defines the train/validation loop and callbacks

Below are the main executable scripts in the repository:

• config.py: configuration file
• dataset.py: defines the dataset classes
• extract_patches.py: extracts patches from original images
• compute_stats.py: main metric computation script
• run_train.py: main training script
• run_infer.py: main inference script for tile and WSI processing
• convert_chkpt_tf2pytorch: convert tensorflow .npz model trained in original repository to pytorch supported .tar format.

Running the Code

Training

Data Format

For training, patches must be extracted using extract_patches.py. For instance segmentation, patches are stored as a 4 dimensional numpy array with channels [RGB, inst]. Here, inst is the instance segmentation ground truth. I.e pixels range from 0 to N, where 0 is background and N is the number of nuclear instances for that particular image.

For simultaneous instance segmentation and classification, patches are stored as a 5 dimensional numpy array with channels [RGB, inst, type]. Here, type is the ground truth of the nuclear type. I.e every pixel ranges from 0-K, where 0 is background and K is the number of classes.

Before training:

• Set path to the data directories in config.py
• Set path where checkpoints will be saved in config.py
• Set path to pretrained Preact-ResNet50 weights in models/hovernet/opt.py. Download the weights here.
• Modify hyperparameters, including number of epochs and learning rate in models/hovernet/opt.py.

Usage and Options

Usage:

  python run_train.py [--gpu=<id>] [--view=<dset>]
  python run_train.py (-h | --help)
  python run_train.py --version

Options:

  -h --help       Show this string.
  --version       Show version.
  --gpu=<id>      Comma separated GPU list.  
  --view=<dset>   Visualise images after augmentation. Choose 'train' or 'valid'.

Examples:

To visualise the training dataset as a sanity check before training use:

python run_train.py --view='train'

To initialise the training script with GPUs 0 and 1, the command is:

python run_train.py --gpu='0,1' 

Inference

Data Format

Input:

• Standard images files, including png, jpg and tiff.
• WSIs supported by OpenSlide, including svs, tif, ndpi and mrxs.

Output:

• Both image tiles and whole-slide images output a json file with keys:
  • 'bbox': bounding box coordinates for each nucleus
  • 'centroid': centroid coordinates for each nucleus
  • 'contour': contour coordinates for each nucleus
  • 'type_prob': per class probabilities for each nucleus (default configuration doesn't output this)
  • 'type': prediction of category for each nucleus
• Image tiles output a mat file, with keys:
  • 'raw': raw output of network (default configuration doesn't output this)
  • 'inst_map': instance map containing values from 0 to N, where N is the number of nuclei
  • 'inst_type': list of length N containing predictions for each nucleus
• Image tiles output a png overlay of nuclear boundaries on top of original RGB image

Model Weights

Model weights obtained from training HoVer-Net as a result of the above instructions can be supplied to process input images / WSIs. Alternatively, any of the below pre-trained model weights can be used to process the data. These checkpoints were initially trained using TensorFlow and were converted using convert_chkpt_tf2pytorch.py. Provided checkpoints either are either trained for segmentation alone or for simultaneous segmentation and classification. Note, we do not provide a segmentation and classification model for CPM17 and Kumar because classification labels aren't available.

IMPORTANT: CoNSeP, Kumar and CPM17 checkpoints use the original model mode, whereas PanNuke and MoNuSAC use the fast model mode. Refer to the inference instructions below for more information.

Segmentation and Classification:

• CoNSeP checkpoint
• PanNuke checkpoint
• MoNuSAC checkpoint

Segmentation Only:

• CoNSeP checkpoint
• Kumar checkpoint
• CPM17 checkpoint

Access the entire checkpoint directory, along with a README on the filename description here.

If any of the above checkpoints are used, please ensure to cite the corresponding paper.

Usage and Options

Usage:

  run_infer.py [options] [--help] <command> [<args>...]
  run_infer.py --version
  run_infer.py (-h | --help)

Options:

  -h --help                   Show this string.
  --version                   Show version.

  --gpu=<id>                  GPU list. [default: 0]
  --nr_types=<n>              Number of nuclei types to predict. [default: 0]
  --type_info_path=<path>     Path to a json define mapping between type id, type name, 
                              and expected overlay color. [default: '']

  --model_path=<path>         Path to saved checkpoint.
  --model_mode=<mode>         Original HoVer-Net or the reduced version used in PanNuke / MoNuSAC, 'original' or 'fast'. [default: fast]
  --nr_inference_workers=<n>  Number of workers during inference. [default: 8]
  --nr_post_proc_workers=<n>  Number of workers during post-processing. [default: 16]
  --batch_size=<n>            Batch size. [default: 128]

Tile Processing Options:

   --input_dir=<path>     Path to input data directory. Assumes the files are not nested within directory.
   --output_dir=<path>    Path to output directory..

   --draw_dot             To draw nuclei centroid on overlay. [default: False]
   --save_qupath          To optionally output QuPath v0.2.3 compatible format. [default: False]
   --save_raw_map         To save raw prediction or not. [default: False]

WSI Processing Options:

    --input_dir=<path>      Path to input data directory. Assumes the files are not nested within directory.
    --output_dir=<path>     Path to output directory.
    --cache_path=<path>     Path for cache. Should be placed on SSD with at least 100GB. [default: cache]
    --mask_dir=<path>       Path to directory containing tissue masks. 
                            Should have the same name as corresponding WSIs. [default: '']

    --proc_mag=<n>          Magnification level (objective power) used for WSI processing. [default: 40]
    --ambiguous_size=<int>  Define ambiguous region along tiling grid to perform re-post processing. [default: 128]
    --chunk_shape=<n>       Shape of chunk for processing. [default: 10000]
    --tile_shape=<n>        Shape of tiles for processing. [default: 2048]
    --save_thumb            To save thumb. [default: False]
    --save_mask             To save mask. [default: False]

The above command can be used from the command line or via an executable script. We supply two example executable scripts: one for tile processing and one for WSI processing. To run the scripts, first make them executable by using chmod +x run_tile.sh and chmod +x run_tile.sh. Then run by using ./run_tile.sh and ./run_wsi.sh.

Intermediate results are stored in cache. Therefore ensure that the specified cache location has enough space! Preferably ensure that the cache location is SSD.

Note, it is important to select the correct model mode when running inference. 'original' model mode refers to the method described in the original medical image analysis paper with a 270x270 patch input and 80x80 patch output. 'fast' model mode uses a 256x256 patch input and 164x164 patch output. Model checkpoints trained on Kumar, CPM17 and CoNSeP are from our original publication and therefore the 'original' mode must be used. For PanNuke and MoNuSAC, the 'fast' mode must be selected. The model mode for each checkpoint that we provide is given in the filename. Also, if using a model trained only for segmentation, nr_types must be set to 0.

type_info.json is used to specify what RGB colours are used in the overlay. Make sure to modify this for different datasets and if you would like to generally control overlay boundary colours.

As part of our tile processing implementation, we add an option to save the output in a form compatible with QuPath.

Take a look on how to utilise the output in examples/usage.ipynb.

Overlaid Segmentation and Classification Prediction

Overlaid results of HoVer-Net trained on the CoNSeP dataset. The colour of the nuclear boundary denotes the type of nucleus.
Blue: epithelial
Red: inflammatory
Green: spindle-shaped
Cyan: miscellaneous

Datasets

Download the CoNSeP dataset as used in our paper from this link.
Download the Kumar, CPM-15, CPM-17 and TNBC datsets from this link.
Down

Ground truth files are in .mat format, refer to the README included with the datasets for further information.

Comparison to Original TensorFlow Implementation

Below we report the difference in segmentation results trained using this repository (PyTorch) and the results reported in the original manuscript (TensorFlow).

Segmentation results on the Kumar dataset:

Platform	DICE	PQ	AJI
TensorFlow	0.8258	0.5971	0.6412
PyTorch	0.8211	0.5904	0.6321

Segmentation results on CoNSeP dataset:

Platform	DICE	PQ	AJI
TensorFlow	0.8525	0.5477	0.5995
PyTorch	0.8504	0.5464	0.6009

Checkpoints to reproduce the above results can be found here.

Simultaneous Segmentation and Classification results on CoNSeP dataset:

Platform	F1d	F1e	F1i	F1s	F1m
TensorFlow	0.748	0.635	0.631	0.566	0.426
PyTorch	0.756	0.636	0.559	0.557	0.348

Citation

If any part of this code is used, please give appropriate citation to our paper.

BibTex entry:

@article{graham2019hover,
  title={Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images},
  author={Graham, Simon and Vu, Quoc Dang and Raza, Shan E Ahmed and Azam, Ayesha and Tsang, Yee Wah and Kwak, Jin Tae and Rajpoot, Nasir},
  journal={Medical Image Analysis},
  pages={101563},
  year={2019},
  publisher={Elsevier}
}

Authors

• Quoc Dang Vu
• Simon Graham

License

This project is licensed under the MIT License - see the LICENSE file for details.

Note that the PanNuke dataset is licensed under Attribution-NonCommercial-ShareAlike 4.0 International, therefore the derived weights for HoVer-Net are also shared under the same license. Please consider the implications of using the weights under this license on your work and it's licensing.

hover_net's Issues

Use Mask-RCNN to train CoNSeP dataset

Hi, I am reproducing your work in your paper. And now, I have almost the results using the model hover_net. However, when I use Mask RCNN(just modify the anchor size and detection instances number) to train these datasets(CoNSeP/Kumar/CPM17)，I get the much worse result compared with yours. Could you help me with more details about how did you implement Mask RCNN to train these three datasets?
Thanks~

Question of dense block's kernel size

In the paper, we can see that the kernel size of the second convolutional layer in dense block is 3 but in the code it's 5. I just wonder is it a mistake or it will be better.

Google Colab usage problem

i run into this error while trying to run this project on google colab.any idea to resolve this

tensorflow-gpu==1.12
tensorpack==0.9.0.1
python 3.6.9
CUDA 10.0.130

resampling code

Excellent work - if i wanted to change the resampling techniques from NN to interpolation, please advise where is this part of the code? Also, have you trained on the PanNuke dataset? If so are any changes are needed? Thank-you.

what is the validation data for Kumar comparative experiments

what is the strategy for your hyperparameter-tuning? For Kumar comparative experiments, if your network has been adjusted according to Kumar test set, the test result might be unreliable.

get_fast_AJI

When we use the get_fast_AJI to calculate the original AJI, I find that this function just add the missed GT to the denominator.This is a little difference from the original AJI. Does this have an impact on the results？

Issue running code

hey can you please explain how can we run this code on our ConSep dataset once we have dataset downloaded.im facing various problems running this project related to assigning directories.

Can I run the model on MoNuSAC dataset

Can I run the model on MoNuSAC dataset? Since MoNuSAC dataset images have varied shapes and some are smaller than input patch 270*270, what pre-process is needed?

Has PQ been tested?

hover_net/src/metrics/stats_utils.py

Line 152 in 909ef03

def get_fast_pq(true, pred, match_iou=0.5):

Hi @vqdang @simongraham

When I feed this image (just random polygons with unique digits per polygon) for example into get_fast_pq function as both prediction and truth

I get an error:

----> 1 compute_fast_pq(y_t1, y_t2)

~/ds-field-boundaries/segmentation/utils/score.py in compute_fast_pq(true, pred, match_iou)
     55             if pred_id == 0: # ignore
     56                 continue # overlaping background
---> 57             p_mask = pred_masks[pred_id]
     58             total = (t_mask + p_mask).sum()
     59             inter = (t_mask * p_mask).sum()

IndexError: list index out of range

When it should produce a score of 1.

Could you please clarify? Maybe I am missing something obvious here

Patch Extraction

I downloaded the dataset of MoNuSeg. The Annotations directory and Tissue_images directory are under the root dataset directory. Inside Annotations directory, XML file is provided. But I notice it read .npy file inside your code. SO I want to know how did you convert the XML file to npy? Or Can you share the .npy file for annotations?
One more thing, Before Training , I need to extract patches , right? Also, In config.py file, There are too many Path String , Could you add more explanation for these different Path?
Thanks! Looking forward to your reply.

About src/loader/augs.py

Thanks for your nice code!
I have two questions here:

1. The operation that expanding the box seems not considering the situation in which the instance is on the boundary of the image. (where -2 may be negative and +2 may be out of bounds).Code Here
2. And another question is that, after expanding the box, is there any need to check the size of the height and width of the bounding box? (Unless the instance is on the boundary, the minimal size of the bounding box would be 4x4)Code Here

pre-trained model not found

Hi,

I'm wanted to experiment with the model presented in your paper. Unfortunately however, I'm not capable of running the model out of box because the pre-trained model is missing.

Also the link on the Warwick website seems to be broken, could you help me?

Thanks in advance.

Version compatibility

Hi, what's the tensorflow and tensorpack version are you using?

Train my own data

When i train my data,which is only two kind of nuclei,i find that:

how can i solve it

errors running inference

i run into this error when running inference

The model of contrast

This work is amazing, I have a little question, do the other models you compare also use imagNet for pre-training?

Please provide docstrings and param specifications

hover_net/src/model/graph.py

Line 210 in 38bf1e8

def get_gradient_hv(l, h_ch, v_ch):

hey @simongraham and @vqdang, I am trying to reproduce hover-net. Could you please update the docstring of get_gradient_hv function. Specifically, what are the h_ch, v_ch here, and what are their dimensions.

@navidstuv, this might be of interest to you

you have done an excellent code but i am facing some issues while reproducing your paper. i am using the packages and libraries that you mentioned in requirement.txt

stats for classification error

While running compute_stats for type classification the console produces no this error.
The ground_truth labels in ConSep dataset are in npy format , but the code is reading ground_truth as .mat files. ?

Low F1 score for the miscellaneous class

Thanks for your excellent work first!
I got a result on the CoNSeP dataset as follows:

Metrics	Fm
Mine	0.042
In Paper	0.426

I find that the F1 score for the miscellaneous class is significantly lower than the record in your paper. I have tried a lot to find out the reasons but failed...Could you help me with this problem?
P.S. I should expect a lower F1 score for the miscellaneous class because there are significantly fewer nuclei represented, but it seems something is wrong...

Some other results

Metrics	Fi	Fe	Fs
Mine	0.556	0.628	0.532
In Paper	0.631	0.635	0.566

About get centroids from training,Calculate the loss function related to the distance from the center of mass to the horizontal and vertical

Hi，Thanks for your excellent papaer and code！
Learned from the paper 'HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images', that leverages the instance-rich information encoded within the vertical and horizontal distances of nuclear pixels to their centres of mass。I know how to predicted hv-map (pred_hv) ,but I don‘t know how to predicted centroid in training, because the loss funtion is related to the distance from the center of mass to the horizontal and vertical。

How to get centroid? As shown in the black part . Where should I find the relevant predicted centroid distance codes.
Looking forward to your reply, thank you.

Running inference on CoNSeP

Hi, I'm trying to run the inference on ConSeP using the pretrained model on CoNSeP for instance segmentation. I put the CoNSep data folder as it is ( from the link given in Readme )

Here are the changes I did in my config.py

i'm getting an error like this -

Any ideas on what's going wrong ?
Your help is really appreciated. Thanks.

Not working with windows

The model does not work with windows.

I get the following error:

[270, 270] [80, 80]
Traceback (most recent call last):
File "train.py", line 268, in
trainer.run()
File "train.py", line 236, in run
self.run_once(opt, sess_init=init_weights, save_dir=log_dir)
File "train.py", line 170, in run_once
train_datagen = self.get_datagen(opt['train_batch_size'], mode='train')
File "train.py", line 158, in get_datagen
nr_procs=nr_procs)
File "C:\Users\Administrator\hover_net\src\loader\loader.py", line 73, in train_generator
ds = PrefetchDataZMQ(ds, nr_procs)
File "C:\ProgramData\Anaconda3\envs\tf_gpu\lib\site-packages\tensorpack\dataflow\parallel.py", line 320, in init
super(MultiProcessRunnerZMQ, self).init()
File "C:\ProgramData\Anaconda3\envs\tf_gpu\lib\site-packages\tensorpack\dataflow\parallel.py", line 90, in init
assert os.name != 'nt', "ZMQ IPC doesn't support windows!"
AssertionError: ZMQ IPC doesn't support windows!

bh
Christian

on MoNuSAC dataset

Hi! Sorry to bother again. I trained the hovernet on MoNuSAC dataset and I encountered this problem. Where might go wrong?

Revision: Improving codebase clarity and usages

Basing on recent created issues from other users. There are following problems need to be tidied up.

• inconsistency between .mat and .npy files
• hard coding of the dice loss number of nuclei types in graph
• hard coding of nuclei types for stats collector in train.py

Any proposal and revision should be posted here for tracking.

Issue running code in Kumar dataset

Hi，Excuse me! Thank you so much for your excellent code and paper :‘HoVer-Net’.

I want to train HoVer-Net using Kumar，but i have some problems:

1. I have changed the Monseg dataset(Kumar) . xml file into .npy file by [https://github.com/vqdang/hover_net/blob/master/src/misc/proc_kumar_ann.py].For a start, I want use https://github.com/vqdang/hover_net/blob/master/src/extract_patches.py to generate training and validation patches.How to choose 'extract_type'?I have used “extract_type = 'mirror' "generate training patches?How to generate validation patches?

2. When training , [https://github.com/vqdang/hover_net/blob/master/src/train.py],How to set self.train_dir = ['../../../CoNSeP/train/%s/' % data_code_dict[self.model_type]]
   self.valid_dir = ['../../../CoNSeP/valid/%s/' % data_code_dict[self.model_type]] in config.py .

Sorry to bother you, you can reply in your free time. Thank you.

crcHisto Dataset usage

can i ask how you used the crcHisto dataset mentioned in the paper .The format of that dataset is much different from others.can you provide some helper code for setting up that dataset for HoverNet pipeline .

Class Imbalance

Hello, I notice there is a low miscellaneous F1 score due to class imbalance - How was class imbalance dealt with in this paper - did augmentation help? Also where and how can i change each of the loss functions in this code?

About CoNSeP Labels

Hi, thanks for your excellent work!
I'm reproducing your work, but I have a problem here:
in src/compute_stats.py
true_info = sio.loadmat(true_dir + basename + '.mat')
the labels should be '.mat' format, however, the dataset I downloaded from the drive.google.com is '.npy' format, so my question is how to change the format, could you provide the source code?

About data augmentation

Hi! I have a question about augmentation. Performing augmentation like rotation or affine to the images and masks always comes with interpolation, which I find will corrupt the instance map and class map of the GT masks. I think maybe this will cause problem to the training. How do you solve this problem?
Thank you so much for your time!!

Extraction of vertical and horizontal density maps

Hi

Thank you once more for the nice code.
I cannot see how you precompute the vertical and horizontal maps.
Can you guide me to the relevant part of the code :-) ?

best regards
Christian

Extract patches

Hi. Thank you for awsome code.

To make extract_patches work for the cpm dataset (and any other of the posted datasets) I had to revert the code to inlcude
ann = np.expand_dims(ann, axis=-1)
which was included prior to v2.00 and comment out
# # merge class for CoNSeP # ann_type = ann[...,1] # ann_type[(ann_type == 3) | (ann_type == 4)] = 3 # ann_type[(ann_type == 5) | (ann_type == 6)] = 4 # assert np.max(ann[...,1]) <= 4, np.max(ann[...,1])

Hover net decoder

Hi

I am trying to reimplement your implementation in pytorch.
In Line 91 you do 1x1 convolution. However it looks like the output of the dense block with input 256 and growth rate 32 for 8 layers gives 512 already. ( u3 = Conv2D('convf', u3, 512, 1, strides=1) )

Why do you do 1x1 convolution in line 91 when the volume already has dimension equal to 512?
Normally you do 1x1 convolution to do a linear combination of features.

best regard
Christian

PQ calculation issue.

Hello!
I have a problem with the PQ value. I just run the code with the given samples and the PQ is 0 and not that one present in the readme file. I also tried with my samples but the results is the same.
Could you please help me?
Thank you in advance!

How to train the hover-net on pre-trained weights?

Hi! Thank you for your work. However, I don't know how to train the model on the pre-trained weights of imagenet?

cell annotation

Hello, your work inspires me a lot.I'd like to ask you a quick question about cell annotation.
I now want to annotate my collection of cells, including the Outlines and types of cells.Your generosity in sharing data in your program is admirable.I want to annotate HEs cells the same way you do. Can you provide software and experience for manually annotating cells?Your help will be greatly appreciated.I wish you all the best.

I

config.py

Hi,
Thank you for your study. I could not understand how to update the following lines in config.py (line 74 and 75 of config.py). Can you indicate which data will be given to train_dir and valid_dir?

    self.data_ext = '.npy' 
    # list of directories containing validation patches. 
    # For both train and valid directories, a comma separated list of directories can be used
    self.train_dir = ['../../../CoNSeP/train/%s/'  % data_code_dict[self.model_type]]
    self.valid_dir = ['../../../CoNSeP/valid/%s/' % data_code_dict[self.model_type]]

Thank you

error running train.py

Hi! When I ran train.py, I ran into this. I wonder where might be the problem?

Issues testing data with certain images

I am successfully able to test the image 'test_14.png' into the network, but for 'test_13.png' after I use infer.py I get a warning saying the instance type is of background type and the image does not load. Also when i enter compute_stats.py I just get the output in #13 (comment) even though the directories are correct and I followed the steps.

This implies the input is corrupt somehow. Either way, it shouldn't produce this for test_14 right as I got the correct output image from this and there were no errors in inference?

So just to check, as the paths are definitely right, can I confirm how many test images should there be in the CoNSeP dataset? I had troubles unzipping the file so i'm not sure if it has become corrupt, i only get test_13 & test_14, perhaps causing the issue. Also, the output should give both .mat and .png files out right?

It could also be an error with Numpy as when I run infer.py I get this:

FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)])

UPDATE: Although now I can't even seem to get test_14.png to work...

Add code [resolved]

Your paper says:

Model code available at: https://github.com/vqdang/xy_net

I look forward to it. Thanks.

About proc_kumar_ann.py

Hi. Thank you for the awesome code.
I have a problem to understand this code on the 63rd line of proc_kumar_ann.py :

for idx, inst_map in enumerate(insts_list):
    ann[inst_map > 0] = idx + 1

Why the values of annotation are given the idx of insts_list? Doesn't this lose the segmentation information?

CPU compatibilty?

Hi
thanks for your work. I have no access to a GPU, can i train this model with CPU?

Error running the code

Hi

Thanks again again for the awesome code.
I have now switched to a working Ubuntu machine. However, I still have problems running your code.
As I am a bit new to the model tensorpack I will appreciate some help here.

$python train.py --gpu='0'

OpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py
[270, 270] [80, 80]
[0811 16:53:46 @parallel.py:327] [MultiProcessRunnerZMQ] Will fork a dataflow more than one times. This assumes the datapoints are i.i.d.
[270, 270] [80, 80]
[0811 16:53:46 @logger.py:90] Argv: train.py --gpu=0
2019-08-11 16:53:46.244932: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-08-11 16:53:46.273391: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.273857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2019-08-11 16:53:46.273996: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-08-11 16:53:46.274974: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-08-11 16:53:46.275938: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-08-11 16:53:46.276112: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-08-11 16:53:46.277188: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-08-11 16:53:46.277829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-08-11 16:53:46.280153: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-11 16:53:46.280315: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.280833: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-11 16:53:46.281246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[0811 16:53:46 @interface.py:31] Automatically applying QueueInput on the DataFlow.
Traceback (most recent call last):
  File "train.py", line 268, in <module>
    trainer.run()
  File "train.py", line 236, in run
    self.run_once(opt, sess_init=init_weights, save_dir=log_dir)
  File "train.py", line 208, in run_once
    launch_train_with_config(config, SyncMultiGPUTrainerParameterServer(nr_gpus))
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/train/interface.py", line 90, in launch_train_with_config
    model.get_input_signature(), input,
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/utils/argtools.py", line 200, in wrapper
    value = func(*args, **kwargs)
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/model_desc.py", line 86, in get_input_signature
    inputs = self.inputs()
  File "/home/gandalf/.conda/envs/tf/lib/python3.7/site-packages/tensorpack/graph_builder/model_desc.py", line 118, in inputs
    raise NotImplementedError()
NotImplementedError

Can you provide the version numbers of all packages in the code

I want to run this code, but I don't know the version number of the package involved in the code.Can you provide the version numbers of all packages in the code？Thank you。

Infer

Hi

I have successfully used your repository for training the network.

I could use some help to run infer.py

What exactly should I use for
self.inf_data_list = [
['../../../data/NUC_Kumar/train-set/orig_split/', 'XXXX', 'valid_diff'],?=

CPM15 and CPM17 dataset

Could you please update the CPM15 and CPM17 dataset on Github?

Problem of using GPU in Hover-Net

As I am using co-lab to train the model and it only has one physical GPU which makes the progress very slow.
I am trying to set up two logical GPUs using TensorFlow, however, the error always comes up and the picture below shows the error.

So, my question is can I use tensorpack.train.launch_train_with_config(config, SyncMultiGPUTrainerParameterServer(nr_gpus)) with logical gpus?

Here is my approach:

gpus = tf.config.experimental.list_physical_devices(‘GPU’)
    if gpus:
        # Create 2 virtual GPUs with 1GB memory each
        try:
            tf.config.experimental.set_virtual_device_configuration(
                gpus[0],
                [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024),
                tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
            logical_gpus = tf.config.experimental.list_logical_devices(‘GPU’)
            print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs”)
        except RuntimeError as e:
            # Virtual devices must be set before GPUs have been initialized
            print(e)

tf.device(logical_gpus) #original is os.environ['CUDA_VISIBLE_DEVICES'] = logical_gpus

The Error shows:

Thanks,
Chang

about grouping the labels

Hi @vqdang, thanks for the great work.
I reproduced your results with the code, looks really nice.
I have a question about the labels. in the paper, you mentioned

For our experiments, we grouped the normal and malignant/dysplastic epithelial nuclei
into a single class and we grouped the fibroblast, muscle and
endothelial nuclei into a class named spindle-shaped nuclei.

I didn't see the discussion about the motivation behind it.
I was wondering if there are any reasons to do so.
One reason I could imagine is that some of the nucleus share common features, so we could treat them as one category, the training can be easier.
The thing is, we may want to keep the fine-grained labels for more precise analysis.
Based on your experience, do you think there will be some obstacles to stop us from using the fine-grained labels i.e. 7 classes? such as it may be difficult to train such a model etc. ?
Thank you in advance! Again, great job, well done!

Errors with Train.py

I've extracted image patches successfully, however I get the following error when running train.py. Any ideas?

File "train.py", line 184, in run_once    logger.set_logger_dir(save_dir)  File "../tensorpack/utils/logger.py", line 131, in set_logger_dir    action = input("Select Action: k (keep) / d (delete) / q (quit):").lower().strip()EOFError: EOF when reading a line

Any idea here? This is inside my log file:

^[[32m[0303 01:56:20 @logger.py:90]^[[0m Argv: train.py --gpu=0,1^[[32m[0303 01:56:39 @training.py:50]^[[0m [DataParallel] Training a model of 2 towers.^[[32m[0303 01:56:39 @interface.py:31]^[[0m Automatically applying QueueInput on the DataFlow.^[[32m[0303 01:56:39 @interface.py:43]^[[0m Automatically applying StagingInput on the DataFlow.