Comments (5)
Sorry for the late reply.
The BabyLM dataset can be downloaded from the webpage of the challenge: https://babylm.github.io/
We haven’t evaluated the model on other datasets, but since we rely on the HuggingFace Transformers library, I would expect it to be widely-compatible with eval frameworks.
Thanks for your reply! That truly helps!
from babyllama.
Hello! This is expected. In the distillation pertaining, we use the loss that combines the usual cross-entropy with a KL divergence between the student's and teachers' probability distributions
from babyllama.
ello! This is expected. In the distillation pertaining, we use the loss that combines the usual cross-entropy with a KL divergence between the student's and teachers' probability distributions L=αLCE+(1−α)LKL. This second term is term is pretty large, so the resulting loss is completely off compared to the cross-entropy one. Yet, the model trains. Ideally, we should have computed and logged the cross-entropy on eval. Alternatively, one can load the trained model and eval it on the same evaluation dataset.
Okay, thank you for your reply. I will try their actual test level on the evaluation dataset later.
from babyllama.
Hi,
How can I get the babyllama dataset?
Is there any code for evaluation on other datasets as reported in the paper?
Thanks
from babyllama.
Sorry for the late reply.
The BabyLM dataset can be downloaded from the webpage of the challenge: https://babylm.github.io/
We haven’t evaluated the model on other datasets, but since we rely on the HuggingFace Transformers library, I would expect it to be widely-compatible with eval frameworks.
from babyllama.
Related Issues (5)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from babyllama.