Comments (6)
Hi, we have noticed slightly lower MMLU scores for declare-lab/flan-alpaca-xl compared to google/flan-t5-xl. This may be due to the zero-shot format of the Alpaca data compared to few-shot for MMLU, and we are benchmarking multiple models here:
https://github.com/declare-lab/flan-eval
from flan-alpaca.
Thanks a lot for the efforts
from flan-alpaca.
Are they evaluated using CoT prompting?
from flan-alpaca.
Hi, the evaluation is using direct prompting for MMLU
from flan-alpaca.
Btw, It seems you are doing few-shot prompting, am I right?
from flan-alpaca.
Yes, we used 5-shot prompting for MMLU based on the Flan-T5 paper
from flan-alpaca.
Related Issues (20)
- LoRA + FSDP -- issue HOT 2
- In ShareGPT, why the conversation from human is accumulated? HOT 5
- Flan Data HOT 2
- wget https://raw.githubusercontent.com/tloen/alpaca-lora/main/alpaca_data_cleaned.json -O data/alpaca_clean.json HOT 3
- OMP: Error #100: Fatal system error detected. HOT 1
- Unable to train on 4-5 gtx 1070s HOT 10
- Usage example in readme doesn't work HOT 2
- Loss value is NaN HOT 3
- Quantized HOT 1
- unable to use new flan-alpaca-gpt4-xl in pipeline HOT 2
- Any plan to support trl-peft load_in_8bit for training.py ? HOT 1
- Performance of the model on gsm8k/SVAMP/MultiArith. HOT 1
- Commercial Use? HOT 2
- RuntimeError: Trying to resize storage that is not resizable HOT 1
- Trouble training HOT 2
- I dont see any progress logs HOT 1
- use gpt4all dataset HOT 4
- issue in training declare-lab/flan-alpaca-base model HOT 2
- is there any plan for flan-ul2? HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flan-alpaca.