dhammack/dsb2017

Code for 2nd place solution to the 2017 National Data Science Bowl

dsb2017's Issues

about the data

Hi Daniel,

When I run the file DSB2017-master/training_code/FLung_nodule_models/create_nodules_from_modelv29.py, I find the line 230:

train_files = [f for f in os.listdir(r'E:\lung\data_raw\data')]

I want to know if the data in the directory 'E:\lung\data_raw\data' is the data provided by kaggle ?
If this is the case, kaggle provides the stage1 data is not like LUNA. Do I need to convert the .dcm file provided by kaggle into a .raw file and a .mhd file? How can I do it?

How to interpret the output .csv?

csv_file

Hello daniel:
if I want to change your code for luna16,how can i create my own stage1_masses_predictions.csv?

Thanks

slice indices must be integers or Noe

when I run your code,I got this error slice indices must be integers
detail:
--> 174 voxel = img[z_start:z_end,y_start:y_end,x_start:x_end]
voxel = undefined
img = array([[[-2048, -2048, -2048, ..., -2048, -2048,... -2048, ..., -2048, -2048, -2048]]], dtype=int16)
z_start = 68.5
z_end = 81.5
y_start = 268.5
y_end = 311.5
x_start = 75.5
x_end = 118.5

build_luna_model_v29.py

Please give me your email. can share simpler code like in Actually using this you mentioned

Hello Mr Hammack. I am a university student from Vietnam, currently I have a subject project on the topic of using this LUNA16 set. Can you give me your email information and how to simplify this code?, sorry my english is not good. Thank you very much, looking forward to your reply.

Does this project use GPU? If not, can it be added?

Start file?

Hi @dhammack ,

Thanks for the uploading. I have seen several different scripts for nodule detection. However, which file should I start with? Or are they all parallel? Thanks

load_model

when i load your trained model,i got some error like :
ValueError: Improper config format: {u'l2': 9.999999747378752e-05, u'name': u'L1L2Regularizer', u'l1': 0.0}

already tried all the keras version,still cant load your mode

data generator failure

Hi Daniel,

Last time I sent an email to you to report the problem. I tried to run the /DSB2017-master/training_code/aws/nodule_des_v37b.py file. But there is an error like this:
File "nodule_des_v37b.py", line 121 ,in get_generator_static
ixs1 = np.random.choice(range(X1.shape[0]),size=n1,replace=False)
ValueError: Cannot take a larger sample than population when 'replace=False'

You gave me the suggestion that the data didn't generate correctly.
But about the file /DSB2017-master/training_code/aws/data_generator_fn3.py, I just modified the corresponding directory . I guess if the file being used has any problem.
annotations_enhanced.csv
I use the file in DSB2017-master/training_code/DLung
candidates_V2.csv
Because I did not find this file in your folder, I used this candidates_V2.csv file in the sources provided by Julian. And I don't know whether there is a problem.
LUNA
I just use the whole data set of LUNA16.

Secondly, I have some confusion about these ‘None‘.

Thirdly, about the error :
File "nodule_des_v37b.py", line 121 ,in get_generator_static
ixs1 = np.random.choice(range(X1.shape[0]),size=n1,replace=False)
ValueError: Cannot take a larger sample than population when 'replace=False'

When I change the replace=True, it gets following results:

There are also some 'None' and 'MetaImage: M_ReadElementsData: data not read completely'.
I don't kow what's wrong.

How to generate annotations_enhanced.csv? and how do you plot Figure 3: Feature importance plot. in pdf thanks

Hi, dhammack. Your work is great ! Thank you for sharing!
I have two questions to ask you.

1. how to generate annotations_enhanced.csv? which code can generate this file?
2. how do you know the Feature importance mentioned in pdf file? How to calculate that? Could you provide and share the relevant code?
   Looking forward to a reply.Thank you very much!

In what order should I run the scripts?

License for the project

Hi!
I would like to try out your project/reproduce it, but as it stands right now I have no right to do so since there is no license attached.
Would you mind adding one?

DSB

Hi @dhammack
Lung Cancer detection is my final year project. So i have to work seriously on that. So can you help me for that ?

what is branching?

Could you elaborate what do you mean by "branching":

It was also found that ‘branching’ the model earlier produced better results when training with multiple objectives. If branching is done too late, the model outputs are too correlated (as they share too many parameters) and thus they provide overall less information in the next stage of the pipeline.

My understanding, that you are running a few iterations adding (?) blocks and checking whether the network started predicting, is it correct?

about the file DSB2017-master\scoring_code\create_preds_from_model_outputs.py???

Hi Daniel,

When I run the file DSB2017-master\scoring_code\create_preds_from_model_outputs.py, I find the line 264:

labels = pd.read_csv(r"F:\Flung\stage2\stage1plus2_labels.csv")

In the DSB2017-master \ scoring_code folder I can not find this file stage1plus2_labels.csv, I can only find stage1plus2_masses_predictions.csv. When I use stage1plus2_masses_predictions.csv instead of the stage1plus2_labels.csv, code produces an error. The error should come from the file format is wrong, so I guess stage1plus2_masses_predictions.csv file can not replace the above documents.

I hope you can give me some advice, thanks

Question about lung segmentation U-net

in your description,
use U-net for segmentation lung.

• http://juliandewit.github.io/kaggle-ndsb2017/

how to get lung segmentation mask?
there are some other data or manually segment lung?

Generation of annotations_enhanced.csv

Hi Daniel,

Had a question regarding the generation of annotations_enhanced.csv. The first few columns are fairly straightforward(from luna16 annotations.csv and the mhd file itself). But how was the 'margin', 'lobulation', 'spiculation', 'malignancy' values generated? In LIDC xml, these features have integral values from 1 - 5 but in annotations_enhanced.csv, these features have fractional parts. Can you please explain?

seriesuid 1.3.6.1.4.1.14519.5.2.1.6279.6001.100398138793540579077826395208
margin 3.66667
lobulation 1.33333
spiculation 1.33333
malignancy 2.66667

Thanks.

trained network

Hi Daniel,

Would it be possible to get access to your already trained network without the need to re-do the trainig from scratch?

Thank you!
Laleh