Add support for multiple and missing modalities

[fepegar]

Add support for multiple and missing modalities

[fepegar]

Idea:

subject_1 = {
    'image': {
        'T1': path,
        'T2': path,
    },
    'label': path,
}
subject_2 = {
    'image': {
        'T1': path,
        'T2': path,
    },
    'label': path,
}
paths = [subject_1, subject_2]

And then

CHANNELS_DIMENSION = 1
t1_tensor = batch['image']['T1']
t2_tensor = batch['image']['T2']
model_input = torch.cat((t1_tensor, t2_tensor), dim=CHANNELS_DIMENSION)

[fepegar]

@romainVala

[romainVala]

yes sound good, I see the point
may be the code could understand the 2 possibles data structure, (just by testing if subject_1['image'] is a dict, so that for the case of single modality one could stay with the previous simpler definition.
but we can also force the user to use this description.... as you want

[romainVala]

Actually an other possibility is to just use on "dictionary" level but with specific keyword: (containing the "image" string)
for instance:
suj1={
'image_1' : path,
'image_2' : path,
'label' : path,}

the in the code, you can do something like
image_keys = [ll for ll in batch if 'image' in ll]
all_images = [batch[ik].squeeze() for ik in image_keys]
model_input = np.stack( all_images)

the code is more vebose, but the input structure more simple ... it is just a convention choice

[romainVala]

Actually I need something very similar for the label definition. In the case I want to learn from probability labels (and not binairy ones) I need to have the label concatenate in the channel dimension

this is what I added in the getitem from ImagesDataset :
after line 56 (when the sample dict is filled)

    label_name = [kk for kk in sample if ('label' in kk) ]
    if len(label_name) > 1:
        list_label = [sample[kkk].squeeze() for kkk in label_name]
        for kkk in label_name : 
            del sample[kkk]  #remove label_1 label_2 ... entery
        sample['label'] = np.stack(list_label)

[fepegar]

I ended up using this design: https://github.com/fepegar/torchio/blob/master/examples/example_multimodal.py#L37-L54

I think it's clear, generic and not too hard-coded. Please let me know what you think.