samrawal/llama2_chat_templater

Wrapper to easily generate the chat template for Llama2

• 👋 Hi, I’m @samrawal!
• 🩺 I'm currently an M.D. student at the Carle Illinois College of Medicine at the University of Illinois, Urbana-Champaign.
• 🤖 I'm also working on Natural Language Processing research applied to neurology records:
  • 🌵 Clinical AI software is at @cactiML and cactiml.com
• 💻 More about me at samrawal.com.
• ✏️ I write at samrawal.substack.com.
• 🐦 I'm on Twitter @samarthrawal.

llama2_chat_templater's Issues

Bug: The first [INST] has no corresponding closing [\INST]

As per the Llama 2 chat format. (https://gpus.llm-utils.org/llama-2-prompt-template/)

We should have the following:

<s>[INST] <<SYS>>
{your_system_message}
<</SYS>>

{user_message_1} [/INST]

However, the current implementation will generate something like:

<s>[INST] <<SYS>>
{your_system_message}
<</SYS>>

[INST] {user_message_1} [/INST]

which is wrong.

When having a long conversation, it does not generate new starting <s> tokens from second user - model message pairs

Using the following code:

pt = PromptTemplate(system_prompt="This is a system message")
pt.add_user_message("Hello, how are you?")
pt.add_model_reply("I am doing good! Hope you are doing well too.")
pt.add_user_message("Who is the PM of India?")
pt.add_model_reply("The Prime Minister of India is Narendra Modi.")
pt.add_user_message("Thank you!")

Currently the output looks like this,

'<s>[INST] <<SYS>>\nThis is a system message\n<</SYS>>Hello, how are you? [/INST] I am doing good! Hope you are doing well too. </s>[INST] Who is the PM of India? [/INST] The Prime Minister of India is Narendra Modi. </s>[INST] Thank you! [/INST]'

The actual prompt should be like this,

'<s> [INST] <<SYS>>\nThis is a system message\n<</SYS>> Hello, how are you? [/INST] I am doing good! Hope you are doing well too. </s> <s> [INST] Who is the PM of India? [/INST] The Prime Minister of India is Narendra Modi. </s> <s> [INST] Thank you! [/INST]'