tensor46/tensormonk

A collection of deep learning models (PyTorch implemtation)

TensorMONK

A collection of deep learning architectures (a PyTorch implementation).

Dependencies

• python >= 3.6
• PyTorch > 1.3
• torchvision
• visdom

Training models on 2012 ImageNet recognition task

If you have more nvidia cards & cores available, adjust the batch size (BSZ), number of GPUs (gpus), & number of threads (cpus) accordingly in the ./ImageNet.sh. Next, select an available architecture and update your train & validation folder location (trainDataPath and testDataPath). Finally, run ./ImageNet.sh.

Training CapsuleNet on MNIST

To replicate Hinton's paper on MNIST, run the following:

python Capsule.py -A capsule -B 256 -E 500 --optimizer adam --gpus 2 --cpus 6 --trainDataPath ./data --testDataPath ./data --replicate_paper

Ignore the replicate_paper argument to create a deep architecture (with few residual blocks before primary capsule). You can essentially add any block available in NeuralLayers to create a deeper architecture, which is followed by a primary capsule and secondary capsule. However, do consider two things 1. if you do reconstruction, update the reconstruction network relative to tensor_size, 2. capsule nets do require a good amount of gpu ram.

Generative Adversarial Networks

Progressive Growing of GANs

Trained on CIFAR10 (pggan-cifar10.py) -- requires more training (more gpus)!

References

Activation Functions

• Maxout Networks
• SWISH: A SELF-GATED ACTIVATION FUNCTION
• Mish: Self Regularized Non-Monotonic Activation Function

Classification

• Deep Neural Decision Forests

Generative Models

• Auto-Encoding Variational Bayes
• Progressive Growing of GANs
• Self-Attention Generative Adversarial Networks

Image Recognition Models

• Aggregated Residual Transformations for Deep Neural Networks
• Deep Residual Learning for Image Recognition
• Densely Connected Convolutional Networks
• Dynamic Routing Between Capsules
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Squeeze-and-Excitation Networks

Image Segmentation Models

• AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• U-Net: Convolutional Networks for Biomedical Image Segmentation

Local Features

• Learning Discriminative and Transformation Covariant Local Feature Detectors

Loss Functions

• AN EXPLORATION OF SOFTMAX ALTERNATIVES BELONGING TO THE SPHERICAL LOSS FAMILY
• Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
• ArcFace: Additive Angular Margin Loss for Deep Face Recognition
• CosFace: Large Margin Cosine Loss for Deep Face Recognition
• Rethinking Feature Distribution for Loss Functions in Image Classification
• Tversky loss function for image segmentation using 3D fully convolutional deep networks

Object Detection Models

• SSD: Single Shot MultiBox Detector (MobileNetV2SSD320 and TinySSD320)

Optimizer

• Lookahead Optimizer: k steps forward, 1 step back
• On the Variance of the Adaptive Learning Rate and Beyond

Regularizations

• DropBlock: A regularization method for convolutional networks

tensormonk's Issues

How to use condconv

Thanks for your great work
Could you tell me how to use Condconv in my network ？

Mish Implementation

According to the code file, the implementation of Mish is as follows:

def mish(tensor: torch.Tensor) -> torch.Tensor:
    return tensor * tensor.exp().add(1).log().tanh()

In my repository Mish implementation is like:

def mish(tensor: torch.Tensor) -> torch.Tensor:
    return tensor * torch.tanh(F.softplus(tensor))

Is your implementation faster than mine??

Also, would appreciate if you included the paper link of Mish in the main Readme and in the activations readme, in the list of available activations, Mish is not present.

The paper link is in my repository here - https://github.com/digantamisra98/Mish

Request for a folder structure for models and the corresponding files created

Issue: While training multiple models, all the models and the corresponding files generated are all saved in the same folder. This makes it hard to search for files generated by a particular model.

Recommendation: To generate a folder for the model being created with the same name and all the corresponding files get dumped inside it.