jeankaddour/notrainnogain

Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)

No Train No Gain

Code for the paper "No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models"; Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, Matt J. Kusner .

Running the code

See the README for the:

• BERT experiments
• T5 experiments

Citation and license

We use two excellent open source codebases to implement our experiments:

• The BERT experiments are forked of Cramming
• The T5 experiments are forked of NanoT5

If you find this repository useful, please consider citing both our work and these original codebases.

To cite our work, we suggest the following BibTeX:

@misc{kaddourNoTrainNo2023,
	title = {No {Train} {No} {Gain}: {Revisiting} {Efficient} {Training} {Algorithms} {For} {Transformer}-based {Language} {Models}},
	url = {http://arxiv.org/abs/2307.06440},
	doi = {10.48550/arXiv.2307.06440},
	urldate = {2023-07-17},
	publisher = {arXiv},
	author = {Kaddour, Jean and Key, Oscar and Nawrot, Piotr and Minervini, Pasquale and Kusner, Matt J.},
	month = jul,
	year = {2023},
	note = {arXiv:2307.06440 [cs]},
}

We provide separate licenses for the BERT experiments and the T5 experiments.

Contact

Feel free to open an issue, or email us, with any questions.

notrainnogain's Issues

Virtual environment already activated

Hi, Thanks for the amazing repository.
I'm trying to setup the bert environment as per instruction, but the following error raises when I run poetry shell
'''
Virtual environment already activated: /localdisk/home/user/miniconda3/envs/ntng_bert
'''
I found a workaround here which is to run the command source $(poetry env info --path)/bin/activate however the following error raises
'''
source: no such file or directory: /localdisk/home/user/miniconda3/envs/ntng_bert/bin/activate
'''

Any idea why this error happens?
Thanks

Do you mean those methods do not accelerate as they claims?

According to your experiments in the paper, I am almost comvinced that 5 out of 6 sample acceleration methods fail to accelerate LLM models, when measured by your unique RST.
but I also studied Sophia for a while, found its theory and experiments being fantastic and well-grounded. It is hard to believe simply applying RST and change the model could lead to contrast conclusion on whether acceleration can work.
Would you like make a brief intuitive introduction to help me understand the critical value of your work? I will appreciate it very much.