alykhantejani/nninit

Weight initialization schemes for PyTorch nn.Modules

nninit

Weight initialization schemes for PyTorch nn.Modules. This is a port of the popular nninit for Torch7 by @kaixhin.

##Update

This repo has been merged into PyTorch's nn module, I recommend you use that version going forward.

###PyTorch Example

import nninit
from torch import nn
import torch.nn.init as init
import numpy as np

class Net(nn.Module):
  def __init__(self):
     super(Net, self).__init__()
     self.conv1 = nn.Conv2d(5, 10, (3, 3))
     init.xavier_uniform(self.conv1.weight, gain=np.sqrt(2))
     init.constant(self.conv1.bias, 0.1)

network = Net()

##Installation Clone the repo and run python setup install

##Usage

import nninit
from torch import nn
import numpy as np

class Net(nn.Module):
  def __init__(self):
     super(Net, self).__init__()
     self.conv1 = nn.Conv2d(5, 10, (3, 3))
     nninit.xavier_uniform(self.conv1.weight, gain=np.sqrt(2))
     nninit.constant(self.conv1.bias, 0.1)

network = Net()

##Supported Schemes

• nninit.uniform(tensor, a=0, b=1) - Fills tensor with values from a uniform, U(a,b)
• nninit.normal(tensor, mean=0, std=1) - Fills tensor with values drawn from a normal distribution with the given mean and std
• nninit.constant(tensor, val) - Fills tensor with the constant val
• nninit.xavier_uniform(tensor, gain=1) - Fills tensor with values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. and Bengio, Y., using a uniform distribution.
• nninit.xavier_normal(tensor, gain=1) - Fills tensor with values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. and Bengio, Y., using a normal distribution.
• nninit.kaiming_uniform(tensor, gain=1) - Fills tensor with values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. using a uniform distribution.
• nninit.kaiming_normal(tensor, gain=1) - Fills tensor with values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. using a normal distribution.
• nninit.orthogonal(tensor, gain=1) - Fills the tensor with a (semi) orthogonal matrix. Reference: "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks" - Saxe, A. et al.
• nninit.sparse(tensor, sparsity, std=0.01) - Fills the 2D tensor as a sparse matrix, where the non-zero elements will be drawn from a normal distribution with mean=0 and std=std.

nninit's Issues

set seed in weight init

Hi ;

I am implementing a fully conv network which has conv and transposed convolution modules in them. I was wondering if it possible to set the seed so that I would be able to reproduce my results . Below is the code I wrote for weight init.

def init_weights(m):
if type(m) == nn.Conv2d :
init.xavier_uniform(m.weight, gain=np.sqrt(2))
if type(m)==nn.ConvTranspose2d:
init.xavier_uniform(m.weight, gain=np.sqrt(2))

model.apply(init_weight)

Add ability to pass in Parameters as well as Tensors

right now the functions need to be passed weight.data, it would be nice to pass in just weight

usage problem

Should we use it like:
w = torch.Tensor(3, 5)
nninit.uniform(w)
OR:
w = torch.Tensor(3, 5)
w = nninit.uniform(w)
I suppose they give the same results ? But your function always return values ...

license?

Given that it is derived from nninit, I'm assuming that the license is MIT; still, it would be advisable to include a license file in the repo.