Does autoLLM support local LLMs?

[safevideo]

Does autoLLM support local LLMs?

[fcakyon]

Yes @igoralvarezz it is currently supported. You can create an instance of HuggingFaceLLM and use it in the autollm pipeline as:

import torch

from llama_index.llms import HuggingFaceLLM
from autollm import AutoServiceContext, AutoVectorStoreIndex, AutoQueryEngine

llm = HuggingFaceLLM(
    context_window=4096,
    max_new_tokens=256,
    query_wrapper_prompt,
    tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
    model_name="StabilityAI/stablelm-tuned-alpha-3b",
    device_map="auto",
    stopping_ids=[50278, 50279, 50277, 1, 0],
)

service_context = AutoServiceContext.from_defaults(llm=llm)
vector_store_index = AutoVectorStoreIndex.from_defaults()
query_engine = AutoQueryEngine.from_instances(vector_store_index. service_context)

[fcakyon]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They don't work locally, but they do have a larger context window.

You should use HuggignFace and Ollama models as local alternatives with autollm.

[fcakyon]

Hello @jonny7737, OpenAI provides 16K context windiw models, clause-v2 has 100K context window.

Moreover, you can use any opensource HuggingFace and Ollama models with autollm (since we are using LiteLMM underhood). Feel free to ask more questions!

[jonny7737]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

[Meathelix1]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They cant.. The only thing you are hosting locally would be the database that these models speak too, or the model from the Local LLM.

[igoralvarezz]

I was not aware OpenAI and Claude2 could be hosted locally. I stand corrected.

They don't work locally, but they do have a larger context window.

You should use HuggignFace and Ollama models as local alternatives with autollm.

Is there a way to use local (offline) models from huggingface without the use of Ollama (currently not available for Windows OS yet)?