Finetuning on custom dataset for ASR

[samuelleecong]

Finetuning on custom dataset for ASR

[mavlyutovr]

@Incarceronn I would suggest to implement a lightweight alternative to Speech2SpeechFleursDatasetBuilder and use it in scripts/m4t/finetune/dataset.py. The ASR specific will be that source and target in LangPairSample will be equal (pointing to the same MultimodalSample), and each MultimodalSample will have the following field values:

id - uniq identifier of the sample
lang - language code of the dialect
text - transcription of the audio sample
audio_local_path - local path to the audio sample
waveform = None
sampling_rate = None
units: List[int] - list of units produced by UnitSpeechTokenizer from the audio sample

[Gxhappiness]

Hi,
I have the same problem. Have u addressed the issue? Such as the data preparation and the finetuning process.