eisen-ai/eisen-core

Core functionality of Eisen

eisen-core's Issues

Testing and exporting commands

Dear all,
I run the colab tutorial but I miss something. I have this questions:

1. How can test the model after running?
2. How can save checkpoint and the model?
   thanks so much

convert_output_dict_to_cpu removes metrics if they return scalars

the last line assumes that if the value is not a torch tensor or numpy array it should be removed, so it silently eliminates any scalar values returned from losses, metrics, or inputs/outputs

                if isinstance(output_dict[typ][i][key], torch.Tensor):
                    output_dict[typ][i][key] = (
                        output_dict[typ][i][key].cpu().data.numpy()
                    )
                elif isinstance(output_dict[typ][i][key], np.ndarray):
                    pass
                else:
                    output_dict[typ][i].pop(key, None)

the python "numbers" package can be used to check if a value is a number of any kind.

import numbers

if isinstance(x, numbers.Number):
    ....

Question on transformers models

I'm attempting to find a good model parallel library for large NLP transformer models. Transformers by HuggingFace is the main library that makes pretrained models accessible. These models can be 1.5 billion parameters or more. Using ModelParallel from eisen on models like T5, DialoGPT, BERT and GPT2 (here's a link as an example) would be simply amazing. The improvements from using large transformer models are convincing. See also here.

The current PyTorch implementation from the transformers library passes a set of tensors to the model to combine (input ids, an attention mask and sometimes token types). As a result, using ModelParallel on a model from the transformers library will expectedly raise NotImplementedError: Support for modules with more than one input is not yet implemented. Similarly, a single forward pass through the model will yield either a dictionary or tuple, so there are multiple outputs as well as multiple inputs.

Is there any way to use Eisen with this, given that support for multiple inputs and multiple outputs seems to have been removed?

How to setup for non-binary segmentation (background, organ and tumor)

Hi @eisen-ai ,

Thanks for the amazing package!

I am experimenting with Eisen to perform segmentations on both organ and tumor and I tried to get example in the documentation on how to work with a segmentation that as two labels (0s - background; 1s - organ; 2s - tumor), but didn't find anything.

I am following the tutorial 'Eisen_MSD_Demo.ipynb' but I don't know what I shall put in the cell that contains the following code:

resample_tform = ResampleNiftiVolumes(['image', 'label'], [1.0, 1.0, 1.0], 'linear')

to_numpy_tform = NiftiToNumpy(['image', 'label'])

crop = CropCenteredSubVolumes(fields=['image', 'label'], size=[64, 64, 64])

add_channel = AddChannelDimension(['image', 'label'])

map_intensities = MapValues(['image'], min_value=0.0, max_value=1.0)

threshold_labels = ThresholdValues(['label'], threshold=0.5)

should I transform
threshold_labels = ThresholdValues(['label'], threshold=0.5)
into
threshold_organ_labels = ThresholdValues(['label'], threshold=0.5)

threshold_tumor_labels = ThresholdValues(['label'], threshold=1.5)
?

Thanks in advance?

Add random spatial and intensity transforms

It would be very helpful to have support for various spatial and intensity transforms, similar to what is already available in torchio transforms and MONAI. Supporting torchio in eisen-extras may be the most sensible way to go, imo.

The specific transforms I am interested in include:

• random affine (3D, 4D)
• random elastic deformation (3D, 4D)
• random shift and crop (3D, 4D)
• random noise, ideally with specified distribution (3D, 4D)
• random flips (3D, 4D)

Note: 4D in my definition means supporting multichannel data (e.g. T1 and T2 channel) and applying the same 3D transform to each channel.

Eisen trasformation

I would like to save my input after normalization and resize as numpy. What is the procedure?
thanks so much!!

Support Running Metric Computation

The current behavior of eisen is to take the mean of metrics over batches regardless of if this makes sense or not. It also has the minor inaccuracy of adding additional weight to the final batch, e.g. if batch size is 128 and the last batch is only 10 examples.

I propose an Metric object modeled after pytorch ignite with three functions, naively:

class Metric:

    def reset():
        raise NotImplemented

    def update(*args, **kwargs):
        raise NotImplemented

    def compute():
        raise NotImplemented

class Accuracy(Metric):
    def __init__(self):
        self.num_correct = 0
        self.total = 0

    def reset():
        self.num_correct = 0
        self.total = 0

    def update(y_pred, y_true):
        if y_pred.ndim == y_true.ndim == 1:
            self.num_correct += np.equal(y_pred, y_true).astype(int).sum()
        elif y_pred.ndim == 2 and y_true.ndim == 1:
            y_pred = np.argmax(y_pred, dim=1)
            self.num_correct += np.equal(y_pred, y_true).astype(int).sum()
        else:
            ValueError("multilabel accuracy not supported, y_true must have ndim=1 to prevent ambiguity")
        self.total += y_true.shape[0]

    def compute():
        return self.num_correct / self.total
        

perhaps an EisenMetricWrapper could take metrics and run simple reductions on them along these lines to convert an arbitrary function into a Metric class which keeps track of a running average.

Critically, this allows for the ever-important metrics such as Precision or Recall, even PR_curve to be computed across epochs by eisen.

tensorboard summary confusion matrix looks bad for binary labels

scaling is pretty poor in this case, upper left box number is not rendering.

Eisen Pytorch and CUDA version problem！

Hi，My PC have the CUDA Version is 10.0. so i check online find that : pytorch version must be 1.2.
but if i change the torch version to 1.2, the Eisen will be prompt ModuleNotFoundError: No module named 'torch.cuda.amp', i find that reason is Eisen and Pytorch version not match!!!

so, how can i Configuration these ones version？

My configuration：
ubuntu 18.04LST
CUDA 10.0
torch 1.2
Eisen 0.1.10

Tk u

TrainingAMP issues

Hey there,

Thank you for open-sourcing this project. In order to decrease the burden of batch size, I'm trying to use eisen.utils.workflows.TrainingAMP class instead of eisen.utils.workflows.Training as a choice of workflow. However, I measured that there is no significant change in training and inference time, and there is literally no change for the memory footprint.

I analyzed the training.py file where all of these classes are implemented and noted that the lines, that correspond to the torch.cuda.amp.autocast module, have been commented. After uncommenting the lines 9 and 251, this time I faced with a Runtime Error that advice me to use the combination of BCE and Sigmoid, instead of placing the Sigmoid function at the end and using BCE for the loss function. How should we approach to this problem when we would like to use more than 1 loss function term?

GPU: GV100 (32 GB)
OS: Ubuntu 20.04
Model: U-Net3D
Batch Size: 4 (~31 GB of memory footprint)
PyTorch: 1.6.0 + CUDA Toolkit 10.2 (conda)
Eisen: 0.1.10
Python: 3.7

workflow is taking up more memory and time than it needs to

eisen-core/eisen/utils/workflows/workflows.py

Lines 200 to 202 in a265a06

	for loss in losses:
	for key in loss.keys():
	loss[key].backward(retain_graph=True)

This is more memory and computationally efficient than sending each loss back individually:

sum([loss for loss_outputs in losses for loss in loss_outputs.values()]).backward()