Comments (3)
“为此,智源团队创新提出NLPE(Non-Linearized Position Embedding,非线性位置编码)方法,在 RoPE 方法的基础上,通过调整相对位置编码、约束最大相对长度来提升模型外延能力。”
来自 https://mp.weixin.qq.com/s/ZQF4Y-kJaPKn5q69WoxmzQ 的介绍,对NLPE部分比较感兴趣。我看hf上的代码也好像没发现相关内容。
感谢关注~我们已经在准备开源代码了,预计下周会加到仓库。同时之后也会有详细的技术报告来解释NLPE的工作。
from aquila2.
感谢回复~NLPE的具体原理是基于修正attention分布的frequency-aware & position aware 位置编码修改,具体细节我们后续会发布在技术报告里
from aquila2.
打扰一下,请问这个还有后续介绍吗?我有没有错过啥?
from aquila2.
Related Issues (20)
- 34B模型用qlora做sft,设置一个进程一个主机,5个24G卡,发现脚本加载完一个卡后就提示内存不够异常退出,其他卡空闲 HOT 2
- 7b模型qlora sft出错,transformers Version: 4.35.0 ,错误信息见内 HOT 9
- Aquila2/examples/Aquila_BGE_langchain/BGE# CUDA_VISIBLE_DEVICES=4,3,2,1,0 ./preprocess.sh 行 3: 2866484 段错误 HOT 2
- url 404 HOT 2
- How can I save the config.json when fine-tuning Aquila 2? HOT 3
- bash finetune/34B/finetune_qlora_single_node.sh异常 HOT 27
- How can I take the model trained using the script and apply it in Aquila_BGE_langchain? HOT 1
- 指定2个以上gpu时./preprocess.sh: line 3: 2269 Bus error (core dumped) ,1个就不会,log对比在内文
- CUDA_VISIBLE_DEVICES=4,3 python ./local_search.py时,tool中不能识别到GPU3 HOT 1
- 公开的评测榜单 HOT 2
- 量化GPTQ中Aquilachat34b-4bit 代码的predict文件在哪里,Modelscope上没有上传,谢谢 HOT 1
- from predict import predict
- [Question] V100 FInetuning
- RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
- Aquila_BGE_langchain应用所需配置大概需要在什么区间? HOT 1
- Aquila2 7B and Aquila2-7B LoRA How to merge parameters?
- How to review Aquila2-7B model
- Number of training tokens HOT 2
- 官方可以做一版Aquila2Chat-7B到ModelScope上吗? HOT 1
- BAAI/AquilaChat2-7B 是否可以归纳总结文本内容?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aquila2.