gcastro-98/synthetic-medical-images

Using a GAN to synthetically generate medical images for DL purposes

Encantat 👋, I'm Gerard!

A little bit about me

Amongst other things: I love maths, physics, modelling & computers. No wonder why we started Nebbo, right? But did you know...

• 🔭 is a seasonal weather forecasting company, a Vortex spin-off, in which I am working as Technical Director
• 🌱 I studied Maths & Physics in UAB, and Data Science in UB
• 💬 Ask me about diving, nature and sport and I will not shut up
• 📫 Reach me: messaging through my LinkedIn
• 😄 Pronouns: he/him

synthetic-medical-images's Issues

Assess interpretability

Enhance the decoder CNN by explaining its decisions by:

• Layer Wise Relevance Propagation: there are several pytorch implementations. For instance:
  • Tutorial to apply them to VGG-16 and other arquitectures
  • A different flavour also appliable tto pre-trained large VGG models of torchvision
  • Other different suggestions: note residual connections (e.g. those of ResNet) may give continuity problems
• Gradient-based localization (Grad-CAM):
  • There are plenty of guides out there but OmniXAI provides some interesting implementations
• Other explainability flavours different than pixel-attribution methods:
  • Concept Activation Vectors (CAV): check, for instance, captum implementation
  • Counterfactual explanations
  • Contrastive explanations
  • LIME for image classification

Deploying web app

Create a simple fronted-backend with streamlit and deploy it as Heroku web app.

• Similary to what I did for this movie-recommender web app

Implement GAN flavours

Apart from the vanilla conditional cGAN, the following seems promising for CV:

• Deep Convolutional cGAN (cDCGAN):
  • Note a DCGAN can be first constructed, and then conditioned to the labels
• Self-attention GAN
  • Note this tensorflow implementation could be adapted to pytorch
• SynGAN

Or we could even try other generative models such as diffusion ones:

• Class-conditional diffusion models: with this interesting tutorial

Uncertainty quantification

Implement model-agnostic uncertainity quantification procedures, such as according to this survey:

• Conformal Prediction (CP), it has been succesfully applied to:
  • Explain 99% of variance in predicted versus actual error for medical imaging classification
  • Improve audio-visual emotion classification for a semi-supervised GAN (compared with a similar network using the classifier alone)
• Monte Carlo dropout, as Bayesian approx. for Gaussian probabilistic processes (MC), it has been applied to:
  • To correlate uncertainity with observed error when estimating brain and cerebrospinal fluid intracellular volume applying CNN Qin et al.
  • To identify 96% of all predictions with high-risk for error (when applying a CNN for differentiating among glioma, multiple sclerosis, and healthy brain), Tanno et al.