Have you tried to finetune only a few parameters?

[hoyden]

Have you tried to finetune only a few parameters?

[SungFeng-Huang]

Hi! Since the primary goal of our paper is to improve the adaptation efficiency, fine-tuning more parameters would better suit our goal, so we do not dig into experiments of fine-tuning fewer parameters.
For very few trainable parameters, we've only tried fine-tuning the speaker embedding with all other parameters frozen, which means there are only 256 trainable parameters for each speaker adaptation task, and the results are in the paper.
But still, it's a great idea only to fine-tune the last few layers of the decoder, which might gain better adaptation results; however, we're not sure about the adaptation efficiency compared to fine-tuning more parameters.