cauyxy/bilivideos

Hi I'm Xinyu 👋

• 🔭 I’m currently working at Bytedance.

• 🌱 I’m currently learning Graph-Embedding/NLP and more.

• ✨ Recently I'm interested in LLM, and here is some Modest Understandings on LLM

• 💬 Ask me about

  • 🐍 Python
  • 🚀 CUDA/C++
  • 🐛 Write bugs

• 📫 How to reach me： [email protected]

• 📱 Tel: [phone]

• 📺 Bilibili: 小杨不努力0v0

• ⚡ Fun fact: My cat's name is Aphelios, which happens to be the name of my favorite hero in League of Legends.

bilivideos's Issues

FastTokenizer下的乱码问题

首先感谢作者提供的可视化工具

使用FastTokenizer时（其实也是我猜的），比如bloom直接AutoTokenizer.from_pretrained()加载进来后，在这一行：

bilivideos/attn-visual/gen.py

Line 24 in 94f9cca

raw_str = tokenizer.convert_ids_to_tokens([tid])[0]

如果使用convert_ids_to_tokens的话会解出一大堆乱码。改用tokenizer.decode能解决这个问题

LLAMA 中文乱码

这个有办法解决吗？+1
例如：LLAMA

 我 <0xE5> <0x88> <0x9A> <0xE5> <0x88> <0x9A> 

希望可以做个 多个 token 映射 汉字 时候，可以加权求和