[QA Book PDF LangChain Llama 2/Final_Llama_CPP_Ask_Question_from_book_PDF_Llama] Could not load Llama model from path

[brobles82]

[QA Book PDF LangChain Llama 2/Final_Llama_CPP_Ask_Question_from_book_PDF_Llama] Could not load Llama model from path

[kojomensahonums]

I'm experiencing the same issue currently.

AssertionError                            Traceback (most recent call last)
[<ipython-input-11-b81fe2153712>](https://localhost:8080/#) in <cell line: 1>()
----> 1 lcpp_llm = Llama(
      2     model_path=model_path,
      3     n_threads=2, # CPU cores
      4     n_batch=512, # Consider amount of VRAM on system
      5     n_gpu_layers=32 # Dependent on model and GPU RAM

[/usr/local/lib/python3.10/dist-packages/llama_cpp/llama.py](https://localhost:8080/#) in __init__(self, model_path, n_ctx, n_parts, n_gpu_layers, seed, f16_kv, logits_all, vocab_only, use_mmap, use_mlock, embedding, n_threads, n_batch, last_n_tokens_size, lora_base, lora_path, low_vram, tensor_split, rope_freq_base, rope_freq_scale, n_gqa, rms_norm_eps, mul_mat_q, verbose)
    321                     self.model_path.encode("utf-8"), self.params
    322                 )
--> 323         assert self.model is not None
    324 
    325         if verbose:

AssertionError:
```

[harshkasat]

Are you using the same model that was previously loaded?

[brobles82]

Just running the code in your collab as provided

[Usamawahabkhan]

!pip install -qq langchain wget
!pip install gguf #https://github.com/ggerganov/llama.cpp/tree/master/gguf-py
!git clone https://github.com/ggerganov/llama.cpp
!pip -qq install git+https://github.com/huggingface/transformers
#Assuming you are using a GPU
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip -qq install --upgrade --force-reinstall llama-cpp-python --no-cache-dir

from langchain.llms import LlamaCpp
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

Callbacks support token-wise streaming

callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

Verbose is required to pass to the callback manager

from huggingface_hub import hf_hub_download
repo_id="TheBloke/Llama-2-13B-GGML"; filename="llama-2-13b.ggmlv3.q5_1.bin"
hf_hub_download(
repo_id=repo_id, filename=filename,
local_dir="/content"
)

!python /content/llama.cpp/convert-llama-ggmlv3-to-gguf.py --input ls -t /content/*ggmlv3*.bin | head -1 --output ls -t /content/*ggmlv3*.bin | head -1.gguf

filename=filename+".gguf"

n_gpu_layers = 32
n_batch = 512
n_threads=4
llm = LlamaCpp(
model_path="/content/"+filename,
n_threads=n_threads,
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
callback_manager=callback_manager,
n_ctx=2048,
temperature=0.8,
repeat_penalty=1.18,
top_p=1,
top_k=3,
max_tokens=256,
streaming=True,
#verbose=True,
)

[kojomensahonums]

Thanks @Usamawahabkhan the solution works