Do you mean those methods do not accelerate as they claims?

[Sapium59]

Do you mean those methods do not accelerate as they claims?

[JeanKaddour]

Thanks for your comment and interest in our work!
In order for me to better understand how I can help you, can you please specify more about what remains unclear after reading our paper?

[Sapium59]

In my case, I observed behavior of Sophia on both downstream tasks (GLUE & SuperGLUE) in the table 1 in your paper. It seems the baseline (with its learning rate being fully decayed? I am bnot sure) could outperform Sophia acceleration by performance after trained for the same time budget in the GLUE task, while the two methods made a tie in SuperGLUE task. I suppose this observation lead to the conclusion that Sophia cannot accelerate training; it slows down training in fact. However, I am also aware of my hypothesis that validation loss of this training procedure meets the down stream task performance, i.e. baseline's better performance should imply a smaller loss value than Sophia. This data intepretation problem is probably where I need your help most :)
A similar case is the Figure 13 in appendix A.6. (Really similar enough!) I suppose you find $\rho = 0.01$ is the best hyper parameter in practice, which still cannot reach baseline level.
Hope my (maybe silly) questions don't bother you too much. Gratitute again.

[JeanKaddour]

Thanks for your response. So your question is whether Sophia's pre-training loss is also worse than the baseline?
You can find these results in Figure 5. We report the training loss because we use each example only once throughout training; hence, there is no need to use a validation set.

[Sapium59]

Thank you. The loss curves in Figure 5 did report Sophia worse than baseline in most budget cases. It helps a lot.
Meanwhile, the Sophia paper gave the contrast conclusion (2x speed-up compared with AdamW in the number of
steps, total compute, and wall-clock time, as you say). The most difference seems to be time measurement, as you introduced RST. May I regard it as a crutial amendment for Sophia paper and its conclusion, by revealing the acceleration does not necessarily happen in practise as they expected?

[JeanKaddour]

Sorry for the late response. Yes, indeed, your suggestion makes sense!