Comments (13)
Update: I checked it #106 and edit script
DATA_FILE=/Micla/Project/Aquila2/data/alpaca_data_train.jsonl
be ineffective
from aquila2.
All training logs are saved in the log dir.
from aquila2.
OMG!This is throwing the same error that Aquila_v1 had.
from aquila2.
大概率是环境问题。如果硬件适配的话,可以使用我们提供的镜像文件。
from aquila2.
大概率是环境问题。如果硬件适配的话,可以使用我们提供的镜像文件。
ok, I'll give it a go. My hunch is it's probably something with NCCL.
I need to confirm whether this is right or not. Give me 2-3 days to work on it more and I'll swing by with a status update.
PS: The image envs: Aquila_v1
from aquila2.
先试试这个例子能运行吗?
https://github.com/FlagAI-Open/Aquila2/blob/main/examples/predict_chat.py
from aquila2.
Update: edit finetune.sh ---->> python -V
run ssh $ip
,
The ssh connected me to a weird environment I didn't recognize, unlike my usual ssh experience.
I've identified the issue, but don't have a fix yet. The sample code has been running fine without any issues.
from aquila2.
Update:
ssh $ip \ python -V #OK
ssh $ip eval "python -V" #python command not found
torchrun command not found because of "eval
"
update:
add
ssh $ip eval \
"export PATH=$PATH;
I fixed the earlier problem, though a fresh environment setup error has arisen.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
RuntimeErrortorch._C._cuda_setDevice(device)
: RuntimeError
RuntimeErrorCUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
I wanna take a crack at figuring it out solo before anything else.
from aquila2.
Update: run ok, maby
expecthostfile: slots={num}
interacts with finetume.sh: --nproc_per_node={num}
Now:
The training data is too small. Emerge ValueError: num_samples should be a positive integer value, but got num_samples=0
Edit finetume.sh : DATA_FILE=/Micla/Data/GPT_Data/WuDaoCorpus2.0_base_200G/part-202101281a.json
TypeError: AquilaPreTrainedModel._set_gradient_checkpointing() got an unexpected keyword argument 'enable'
I'd check the dataset user guide for pointers on fetching and remaking the data.
from aquila2.
please use enough samples.
please try transformers 4.31.0 version. thanks.
from aquila2.
please use enough samples.
please try transformers 4.31.0 version. thanks.
请使用足够的样品。
请尝试transformers 4.31.0版本。谢谢.
Thank you. I'm in the process of collecting samples. Training should be fine if I get enough samples. I have Transformers version 4.35.0
I imagine the Aquila 2 from FlagScale and the one here are targeted at different use cases, right? The Aquila 2 in this context seems fine-tuned towards the provided Q&A, while FlagScale's Aquila 2 is probably a more general version trained on some benchmark dataset.
from aquila2.
please use enough samples.
please try transformers 4.31.0 version. thanks.
请使用足够的样品。
请尝试transformers 4.31.0版本。谢谢.Thank you. I'm in the process of collecting samples. Training should be fine if I get enough samples. I have Transformers version 4.35.0 I imagine the Aquila 2 from FlagScale and the one here are targeted at different use cases, right? The Aquila 2 in this context seems fine-tuned towards the provided Q&A, while FlagScale's Aquila 2 is probably a more general version trained on some benchmark dataset.
Aquila2 base models is pretrained on about 2T tokens using FlagScale, which was forked from Megatron-LM framework.
Aquila2 chat models is fine-tuned on hundreds of thousands Q&A samples using Deepspeed framework for better usability.
Base models are foundations of chat models. You can use base models and your own Q&A samples to fine-tune.
from aquila2.
please use enough samples.
please try transformers 4.31.0 version. thanks.
请使用足够的样品。
请尝试transformers 4.31.0版本。谢谢.Thank you. I'm in the process of collecting samples. Training should be fine if I get enough samples. I have Transformers version 4.35.0 I imagine the Aquila 2 from FlagScale and the one here are targeted at different use cases, right? The Aquila 2 in this context seems fine-tuned towards the provided Q&A, while FlagScale's Aquila 2 is probably a more general version trained on some benchmark dataset.
Aquila2 base models is pretrained on about 2T tokens using FlagScale, which was forked from Megatron-LM framework. Aquila2 chat models is fine-tuned on hundreds of thousands Q&A samples using Deepspeed framework for better usability.
Base models are foundations of chat models. You can use base models and your own Q&A samples to fine-tune.
请使用足够的样品。
请尝试transformers 4.31.0版本。谢谢.
请使用足够的样品。
请试用transformers 4.31.0版本。谢谢。谢谢我正在收集样本。如果我有足够的样本,训练应该没问题。我有变形金刚4.35.0版,我想FlagScale的Aquila 2和这里的一个针对不同的用例,对吗?在这种情况下,Aquila 2似乎对提供的Q A进行了微调,而FlagScale的Aquila 2可能是在一些基准数据集上训练的更通用的版本。
Aquila 2基础模型使用从Megatron-LM框架派生的FlagScale在大约2 T的token上进行预训练。 Aquila 2聊天模型使用Deepspeed框架在数十万个问答样本上进行了微调,以获得更好的可用性。
基本模型是聊天模型的基础。您可以使用基本模型和自己的问答样本进行微调。
Thanks
from aquila2.
Related Issues (20)
- 34B模型用qlora做sft,设置一个进程一个主机,5个24G卡,发现脚本加载完一个卡后就提示内存不够异常退出,其他卡空闲 HOT 2
- 7b模型qlora sft出错,transformers Version: 4.35.0 ,错误信息见内 HOT 9
- Aquila2/examples/Aquila_BGE_langchain/BGE# CUDA_VISIBLE_DEVICES=4,3,2,1,0 ./preprocess.sh 行 3: 2866484 段错误 HOT 2
- url 404 HOT 2
- How can I save the config.json when fine-tuning Aquila 2? HOT 3
- bash finetune/34B/finetune_qlora_single_node.sh异常 HOT 27
- How can I take the model trained using the script and apply it in Aquila_BGE_langchain? HOT 1
- 指定2个以上gpu时./preprocess.sh: line 3: 2269 Bus error (core dumped) ,1个就不会,log对比在内文
- CUDA_VISIBLE_DEVICES=4,3 python ./local_search.py时,tool中不能识别到GPU3 HOT 1
- 公开的评测榜单 HOT 2
- 量化GPTQ中Aquilachat34b-4bit 代码的predict文件在哪里,Modelscope上没有上传,谢谢 HOT 1
- from predict import predict
- [Question] V100 FInetuning
- RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
- Aquila_BGE_langchain应用所需配置大概需要在什么区间? HOT 1
- Aquila2 7B and Aquila2-7B LoRA How to merge parameters?
- How to review Aquila2-7B model
- Number of training tokens HOT 2
- 官方可以做一版Aquila2Chat-7B到ModelScope上吗? HOT 1
- BAAI/AquilaChat2-7B 是否可以归纳总结文本内容?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aquila2.