weight initializations for conv2d and linear

[pytorch]

weight initializations for conv2d and linear

[fariasfc]

I thought to add another class called Initializers with a function that we could pass the weights and a string with the initialization name. Also add a string argument to every layer called "initialization".

In a Linear layer the init should be something like this:

def __init__(self, in_features, out_features, bias=True, initializer=None):
    super(Linear, self).__init__()
    self.in_features = in_features
    self.out_features = out_features
    self.weight = Parameter(torch.Tensor(out_features, in_features))
    Initializers.initialize(self.weight, initializer)
    if bias:
        self.bias = Parameter(torch.Tensor(out_features))
    else:
        self.register_parameter('bias', None)
    self.reset_parameters()

Do you think that this is the best approach? If not, what would be the best way to do this?

[apaszke]

I think it's better to pass in a function as the initializer and call it with weights as argument. It's clean, verbose, and you don't have to do error checking for invalid/incompatible strings. For example:

nn.Linear(20, 20, weight_init=nn.init.orthogonal)

Do you want to implement it, or was your comment only a suggestion?

[jekbradbury]

Chainer has a nice approach, where the initializer argument is a little polymorphic (can be a callable or a scalar/matrix value). Looks like this:

super().__init__(
    maxout=Maxout(maxout_size, maxout_size, 2, initialW=initializers.GlorotUniform()),
    softmax_linear=Linear(maxout_size, task.num_classes, initialW=0))

[fariasfc]

@apaszke, I am thinking to implement it.

I think that String-like is cleaner. But we could go the way Chainer implements it, as shown by @jekbradbury. With strings or class names being accepted.

What do you guys think?