Is it possible to fine-tuning model to extend token limit length?

[microsoft]

Is it possible to fine-tuning model to extend token limit length?

[edwardjhu]

It might work if you also retrain the positional embedding. I'd love to know if it works!

[byamasu-patrick]

@nai-kon any update on this. Did you manage to achieve this token expansion?

[nai-kon]

Thank you for advice.
Actually, I do not try this token expansion.
Because llama2 supports longer contexts, and a method to further extend the context length by changing the rope frequency has appeared.
So close this issue.