Looking through the code in your fork, and the original, it seems like when using data with the format {"input": "some text", "output": "some output"} the training code only uses the "input" to generate the prompt and then passes these to the model. Is this fine-tuning using CLM on just the inputs? I don't see the outputs being tokenized or fed to the model as labels anywhere to calculate loss and backpropagate.
I may have been confused in thinking that this was for supervised fine-tuning, is this not the case?
On every push pick up any changed trainer_states.json and produce some pretty plots.
Need a way to specify which trainer_states.json to use, how to label the plots accordingly, and whether or not to scale the plots to account for different models and/or fine-tuning data sets.