xbotter/semantic-kernel-llamasharp

Custom TextCompletion using LLamaSharp

⚠️The implementation of this project has been migrated to the LlamaSharp repository, and maintenance will be continued there in the future.

---

semantic-kernel-LLamaSharp

Using LLamaSharp to implement custom TextCompletion and EmbeddingGeneration of Semantic Kernel

Basic Usage

1. Clone this repository.
2. Download a llama model. If you don't know which model is available, please refer to LLamaSharp Verified Models.
3. Modify the path of model and prompt file in samples/SK_LLamaSharpWebApi/appsettings.json.
4. Change the backend type if you want to, by modifying SK_LLamaSharpWebApi.csproj and Directory.Packages.props.
5. Run the sample. It's recommended to run with release mode or x64 configuration.
6. Send the request to the web api to test.

semantic-kernel-llamasharp's Issues

Would you like to develop together?

Hi, this repo is a great work so far!

In fact recently we planed to make such an integration, as you can see in SciSharp/LLamaSharp#9 and microsoft/semantic-kernel#1215. Your work provided a great template for us.

Your experience on ASP.NET core and semantic-kernel is exactly what we need. As an open-source community, we are always open for developers and eager for community efforts joining us. Therefore I sincerely invite you to develop LLamaSharp and the integration with sk with us, to make it a better library for using LLM in local environment. I'll appreciate it if you'd like to join us.

Sincerely,
Rinne

too slow with semantic integration than directly interact with LLamaSharp

Model: ggml-model-f32-q4_0.bin

with CPU backend

even just say hello, current package response with in plenty of minutes.
but if use LLamaSharp it will response immediately

just record this situation for your reference, i will dig into it if have time, maybe it because of memory limit of webapi, and console application doesn‘t have this constraint.

sample code in LLamaSharp demo

foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
    {
        Console.Write(text);
    }

the screen shot of slow reponse when using postman interact with webapi sample code。

update LLamaSharp to v0.4.0

ref : https://scisharp.github.io/LLamaSharp/0.4/

• IChatCompletion
• IChatResult
• ITextCompletion
• ITextResult
• ITextEmbeddingGeneration