aisingapore/sealion

South-East Asia Large Language Models

sealion's Issues

Finetuning with LoRA throws errors

Hello,

While trying to train a LoRA for this model, I ran into this error:
TypeError: MptForCausalLM.forward() got an unexpected keyword argument 'token_type_ids'

A minimal reproducible example is available here, and you can view the error logs in the train.log file.

I was able to work around this by cloning transformers locally and patching the MPT class. I can provide my changes if this would be helpful.

Would it be possible to get LoRA finetuning supported officially with SEA-LION?

transformers version mismatch and problems when load the model

Hi, first of all it is really nice to see SeaLion model. However when i try to load the model, i use the README file

please use transformers 4.34.10

I don't know if there is any version mismatch for transformers? i can not find this version. And also, i have already downloaded tokenization_SEA_BPE.py file but it is still show the warning : "Could not locate the tokenization_SEA_BPE.py inside aisingapore/sealion7b."
Hope to hear from you soon. Thanks again!

Prompt template

Good evening team~, till now I know the prompt format like ### USER: {human_prompt} ### RESPONSE:, but what if I want to translate one language into another, may I know if there's any prompt format we could follow, or I need to fine tune model to achieve my demand, thank you!

Contribute Vietnamese instruct version

Hi, we are the Foundation Models Lab from Hanoi University of Science and Technology, Vietnam (bkai.ai). Is it possible to contribute the Vietnamese instruct version to this project? Thank you.