moritztng/fltr

Like grep but for natural language questions. Based on Mistral 7B or Mixtral 8x7B.

fltr's Issues

quantification-type

Hi, first I wanted to tell you that you did a very nice job.
And for the rest I had a question, I can't determine the quantification method used for the weights, I wanted to try with a Mixtral model that I had fine-tuned but I quickly saw that it was not not a type quantification (gguf/ggml) nor a torch classic backup

I tried the python script that is provided but it takes neither safetensors nor torch nor even (ggml/gguf)
the error is: invalid load key, '\x00'

Illegal hardware instruction (core dumped)

I have an AMD Ryzen system with 3090 GPU and CUDA 12.4. I installed it using the curl command and the installation was successful. However, when running fltr (using any option), it just throws the error: Illegal hardware instruction (core dumped)