I've been stuck on the first epoch in a tiny model for 8+ hours using 300 5s spectrograms for training. Is this normal?

[yuangongnd]

I've been stuck on the first epoch in a tiny model for 8+ hours using 300 5s spectrograms for training. Is this normal?

[michaelschwob]

I think I see the issue. In ast_models.py, in the pretrain_stage==True loop, we compute

self.p_f_dim, self.p_t_dim = self.get_shape(fstride, tstride, input_fdim, input_tdim, fshape, tshape)
num_patches = self.p_f_dim * self.p_t_dim
self.num_patches = num_patches

which results in self.num_patches=24, which is then passed to gen_maskid_patch() as sequence_len. Because mask_size=400 in run.sh, the following loop is infinite:

while len(list(set(mask_id))) <= mask_size:
            start_id = randrange(sequence_len)

            # this improves the efficiency, but might change the pretrained model
            # while start_id in mask_id:
            #     start_id = randrange(sequence_len)

            cur_mask = []
            for i in range(0, cur_clus):
                for j in range(0, cur_clus):
                    mask_cand = start_id + self.p_t_dim * i + j
                    if mask_cand > 0 and mask_cand < sequence_len:
                        cur_mask.append(mask_cand)
            mask_id = mask_id + cur_mask

So, is the solution to simply reduce mask_patch in run.sh file? What exactly are the consequences in reducing the mask_patch to (for example) 10?

[YuanGongND]

please check the paper. The masked patch should be something 50%-80% of the total #patch.

You are running it on CPU (Intel UHD and AMD GPUs do not support cuda). So it will be slow anyways. Maybe training on colab (GPU) is a better idea, even with the free plan.

-Yuan

[michaelschwob]

This answers my question; thank you so much for your help, Yuan! I suggest mentioning that mask_patch should also be modified in the src/pretrain/run_mask_{frame,patch}.sh files. At the moment, in your "Pretrain on custom dataset" section, you do not include this as a parameter to be modified. Additionally, we would need to modify the --n_class and --label-csv arguments in the terminal command.