huiguyy/openllama-chinese

OpenLLaMA-Chinese, a permissively licensed open source instruction-following models based on OpenLLaMA

OpenLLaMA-Chinese

OpenLLaMA-Chinese is a 100% free Chinese large language model, and can be utilized for both non-commercial and commercial purposes.

OpenLLaMA-Chinese is built on OpenLLaMA, which is a permissively licensed open-source reproduction of Meta AI's LLaMA 7B and 13B models, trained on the RedPajama dataset. OpenLLaMA also includes a smaller 3B variant of the LLaMA model. We have conducted fine-tuning on Chinese and English instructions using the OpenLLaMA base models and have made our weights publicly available.

Chinese Instructions Fine-tuning Weights:

• OpenLLaMA 3B
• OpenLLaMA 7B
• OpenLLaMA 13B

English Instructions Fine-tuning Weights:

• [OpenLLaMA 3B](coming soon!)
• [OpenLLaMA 7B](coming soon!)
• [OpenLLaMA 13B](coming soon!)

Chinese+English Instructions Fine-tuning Weights:

• OpenLLaMA 3B
• OpenLLaMA 7B
• [OpenLLaMA 13B](coming soon!)

Data

For Chinese fine-tuning, we utilized the alpaca_data_zh_51k.json from the Chinese-LLaMA-Alpaca project.

For English fine-tuning, we employed the alpaca_data.json from the StanfordAlpaca project.

For fine-tuning with both English and Chinese instructions, we used data from both sources.

Usage

We modified the generate code from LLaMA-X.

To use the PyTorch inference code, follow these steps:

1. Download the weights and update the base_model path in inference/gen_torch.py.
2. Run the following command:

python inference/gen_torch.py

Pretraining and Finetuning

FittenTech offers LLMs pretraining and fine-tuning services. For more details, please visit https://llm.fittentech.com/.

Acknowledgments

We would like to express our gratitude to the developers of the following open-source projects, as our project builds upon their work:

• LLaMA
• OpenLLaMA
• Jittor
• JittorLLMs
• LLaMA-X
• Chinese-LLaMA-Alpaca
• StanfordAlpaca
• transformers

License

We adopt the Apache License, following OpenLLaMA's license.