Bug with input_dim and pure inference

[yuangongnd]

Bug with input_dim and pure inference

[YuanGongND]

Hi,

Can you elaborate on which argument you are referring to, is that

ssast/src/finetune/esc50/run_esc_patch.sh

Line 41 in a1a3eec

target_length=512

Thanks!

-Yuan

[fanOfJava]

yes

[YuanGongND]

Can you explain why the value would be 1024?

It seems to me that it changes

ssast/src/run.py

Lines 97 to 101 in a1a3eec

	audio_conf = {'num_mel_bins': args.num_mel_bins, 'target_length': args.target_length, 'freqm': args.freqm, 'timem': args.timem, 'mixup': args.mixup, 'dataset': args.dataset,
	'mode':'train', 'mean':args.dataset_mean, 'std':args.dataset_std, 'noise':args.noise}
	val_audio_conf = {'num_mel_bins': args.num_mel_bins, 'target_length': args.target_length, 'freqm': 0, 'timem': 0, 'mixup': 0, 'dataset': args.dataset,
	'mode': 'evaluation', 'mean': args.dataset_mean, 'std': args.dataset_std, 'noise': False}

and

ssast/src/run.py

Lines 132 to 138 in a1a3eec

	audio_model = ASTModel(fshape=args.fshape, tshape=args.tshape, fstride=args.fshape, tstride=args.tshape,
	input_fdim=args.num_mel_bins, input_tdim=args.target_length, model_size=args.model_size, pretrain_stage=True)
	# in the fine-tuning stage
	else:
	audio_model = ASTModel(label_dim=args.n_class, fshape=args.fshape, tshape=args.tshape, fstride=args.fstride, tstride=args.tstride,
	input_fdim=args.num_mel_bins, input_tdim=args.target_length, model_size=args.model_size, pretrain_stage=False,
	load_pretrained_mdl_path=args.pretrained_mdl_path)

for both dataloading and model instantiation.

[fanOfJava]

because the process of loading the model file ssast-base-patch-400.pth changes the target_length, the code is shown as below
try:
p_fshape, p_tshape = sd['module.v.patch_embed.proj.weight'].shape[2],sd['module.v.patch_embed.proj.weight'].shape[3]
p_input_fdim, p_input_tdim = sd['module.p_input_fdim'].item(), sd['module.p_input_tdim'].item()

[fanOfJava]

我猜测这也是为什么finetune完之后，做纯推理时load model file会报错的原因。不知道我理解的是否对

[YuanGongND]

can you paste the error code here?

[fanOfJava]

can you paste the error code here?

you can print the p_input_tdim before 156 line of ast_model，you will find the error

[YuanGongND]

I don't have enough time to run it again. The code is a cleaned up version from the development version. It went through a brief test and I guess I did take care of this. So if you already have a error message, that would be very helpful. It might due to something else.

[fanOfJava]

我相信很多人都有同样的问题。因为finetune之后保存的模型，根本没法load进来做纯推理，我也不知道该如何测试训练好的模型的真实性能

[YuanGongND]

Oh I see, yes, that is a known problem. It should be fine if you finetune a pretrained model that has different target_length, but if you want to take the finetuned model for deployment, you will get an error.

For checking the performance, once you finetune a pretrained model, the script will print out the accuracy (or mAP) and also save the result on disk.

For deploy the model for inference, you will need to fix the bug.

[YuanGongND]

Can you check this: #4