ProgDenseBlock, numLaterals and number of columns

[zyzhang1130]

ProgDenseBlock, numLaterals and number of columns

[arcosin]

numLaterals is the number of lateral layers that will be added for finetuning of previous columns. Usually, when using a column generator, you can just set it to len(parentCols). Here is an example:

  class FFGenerator(ProgColumnGenerator):
      def __init__(self):
          self.ids = 0

      def generateColumn(self, parentCols, msg = None):
          cols = []
          cols.append(ProgDenseBlock(8, 20, len(parentCols)))
          cols.append(ProgDenseBlock(20, 50, len(parentCols)))
          cols.append(ProgDenseBlock(50, 10, len(parentCols)))
          cols.append(ProgDenseBlock(10, 2, len(parentCols)))
          return ProgColumn(self.__genID(), cols, parentCols = parentCols)

      def __genID(self):
          id = self.ids
          self.ids += 1
          return id

[zyzhang1130]

Thank you for your reply. That means this lateral connection must always be dense (fully connected)? Also by 'finetuning of previous columns' do you mean the subsequent columns since usually the previous columns' weights will be frozen?

[arcosin]

Happy to help :)

For that particular example, yes. But the only necessary thing is that the columns share the same architecture. You could have something like 2 conv2Ds, a flatten (in an inert block), then a fully connected in the generate column part. These sections of code may make things easier to understand.

dense block init:

def __init__(self, inSize, outSize, numLaterals, ...):
       ...
       self.module = nn.Linear(inSize, outSize)
       self.laterals = nn.ModuleList([nn.Linear(inSize, outSize) for _ in range(numLaterals)])
       ...

conv block init:

def __init__(self, inSize, outSize, kernelSize, numLaterals, ...):
       ...
       self.module = nn.Conv2d(inSize, outSize, kernelSize, **layerArgs)
       self.laterals = nn.ModuleList([nn.Conv2d(inSize, outSize, kernelSize, **layerArgs) for _ in range(numLaterals)])
       ...

Prognet laterals part:

currOutput = block.runBlock(x)
...
for c, col in enumerate(self.parentCols):
    currOutput += block.runLateral(c, col.lastOutputList[row - 1])
    y = block.runActivation(currOutput)
return y

As to finetuning the previous column, I did mean previous though strictly speaking 'finetuning' might have not been the best word. The laterals finetune the representations of the frozen columns through themselves as the laterals are not frozen. The original column keeps all parameters the same while finetuning is learned through the laterals.

[zyzhang1130]

Thank you for your clarification. That really helps a lot.

[arcosin]

No problem!

[zyzhang1130]

Hi, sorry I think I still need a bit clarification:

1. regarding the lateral connection. From this figure, I think it is only those newly added lateral connections will be trainable (i.e. in this case those connected to the 3rd column) while the rest laterals are still frozen right?

2. even if my question 1 is the case, that means when training the newly added columns, there is forward pass through the previously trained columns but the backprop only updates some laterals. It seems quite weird to me. A comparable example I can think of is dropout. In dropout if u dont want to update a certain path of weights u dont even let it be a part of the forward pass in the first place (set to 0). What do you think?