awaescher/ollamasharp

Ollama API bindings for .NET

OllamaSharp 🦙

OllamaSharp is a .NET binding for the Ollama API, making it easy to interact with Ollama using your favorite .NET languages.

Features

• Intuitive API client: Set up and interact with Ollama in just a few lines of code.
• API endpoint coverage: Support for all Ollama API endpoints including chats, embeddings, listing models, pulling and creating new models, and more.
• Real-time streaming: Stream responses directly to your application.
• Progress reporting: Get real-time progress feedback on tasks like model pulling.
• API Console: A ready-to-use API console to chat and manage your Ollama host remotely

Usage

OllamaSharp wraps every Ollama API endpoint in awaitable methods that fully support response streaming.

The follow list shows a few examples to get a glimpse on how easy it is to use. The list is not complete.

Initializing

// set up the client
var uri = new Uri("http://localhost:11434");
var ollama = new OllamaApiClient(uri);

// select a model which should be used for further operations
ollama.SelectedModel = "llama2";

Listing all models that are available locally

var models = await ollama.ListLocalModels();

Pulling a model and reporting progress

await ollama.PullModel("mistral", status => Console.WriteLine($"({status.Percent}%) {status.Status}"));

Streaming a completion directly into the console

// keep reusing the context to keep the chat topic going
ConversationContext context = null;
context = await ollama.StreamCompletion("How are you today?", context, stream => Console.Write(stream.Response));

Building interactive chats

// uses the /chat api from Ollama 0.1.14
// messages including their roles will automatically be tracked within the chat object
var chat = ollama.Chat(stream => Console.WriteLine(stream.Message?.Content ?? ""));
while (true)
{
    var message = Console.ReadLine();
    await chat.Send(message);
}

Api Console

This project ships a full-featured demo console for all endpoints the Ollama API is exposing.

This is not only a great resource to learn about OllamaSharp, it can also be used to manage and chat with the Ollama host remotely. Image chat is supported for multi modal models.

Credits

Icon and name were reused from the amazing Ollama project.

ollamasharp's Issues

Convert roles in Messages to seperate class

See the ChatRole class in the Azure OpenAi package

There its basically an Enum which you can put in, and then internally when its evaluated, its just being converted back to a string.

This makes the usage on the dev side much easier, since you dont have to google which ones are available first...

See here: https://github.com/jmorganca/ollama/blob/main/docs/api.md#generate-a-chat-completion
For the available roles...

Rename Method Name End With 'Async'

Hi,the project is very good ,but the method name that return Task type not end with 'Async', i think end with 'Async' is more consistent with common naming conventions, this is just my suggestion 😄

keep_alive support in Options

0.1.23 adds keep_alive support: https://github.com/ollama/ollama/releases/tag/v0.1.23

Give Message Class a ctor

Self explanatory :p

For easier usage
The Azure OpenAi package has this too

NO_PROXY

You might want to add a note to the readme for others that sit behind a firewall that are experiencing the challenge of Proxy Service. HTTPClient under the covers supports the environment variable NO_PROXY

Adding this to my code gets it working otherwise it's very difficult to diagnose.
System.Environment.SetEnvironmentVariable("NO_PROXY", "localhost");

You can close this issue.. I just wanted to share it with you as I struggled for an hour this morning trying to determine why the code did not work from my office.

[BUG]: Wrong GenerateEmbeddingResponse Format

Version: OllamaSharp 1.0.2
The GenerateEmbeddingResponse doesn't match the actual returned json format from endpoint api/embeddings

I am using OllamaSharp 1.0.2 to call endpoint api/embeddings with model(llama2:70b) and prompt. However, the returned json is { "embedding": [ 1232321, 1232321, ...] } instead of GenerateEmbeddingResponse with { "embeddings": [ 1232321, 1232321, ...] }

According to the official doc, the returned key word should be embedding without s.

Missing parameter "format" in GenerateCompletionRequest

Hi, I'm missing the format parameter for JSON mode in the GenerateCompletionRequest class.

DOC: https://github.com/ollama/ollama/blob/main/docs/api.md

I'm using JSON mode for long running request, 1 request will take up to 5 minutes.
Is there a way to set the timeout?

TimeOut on long queries

if queries are long enough the client will timeout before the actual result is retrieved:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at OllamaApiClient.GenerateCompletion(GenerateCompletionRequest generateRequest, IResponseStreamer`1 streamer)
   at OllamaApiClient.GetCompletion(String prompt, String model, ConversationContext context)
   at llamaAssistance.AiResponseGenerator.GenerateResponse(String query) in /home/neo/Projects/Personal/masstransit/llamaAssistance/llamaAssistance/AiResponseGenerator.cs:line 17
   at Program.<Main>$(String[] args) in /home/neo/Projects/Personal/masstransit/llamaAssistance/llamaAssistance/Program.cs:line 9
   at Program.<Main>(String[] args)

Options are still a string instead of a JSON object.

It seems that options are still a string, making any request with those set invalid.

Example does not work

the example

// set up the client
var uri = new Uri("http://localhost:11434");
var ollama = new OllamaApiClient(uri);

// list all local models
var models = await ollama.ListLocalModels();

// stream a completion and write to the console
// keep reusing the context to keep the chat topic going
var context = await ollama.StreamCompletion("How are you today?", "llama2", context, stream => Console.WriteLine(stream.Response));

// pull a model and report progress
await ollama.PullModel("mistral", status => Console.WriteLine($"({status.Percent}%) {status.Status}"));

produces :
Cannot use local variable 'context' before it is declared

Convert message options to seperate class

See here for available options: https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values

The options are currently just a string, which is... not optimal
It should rather be a class which has all available options.
So you can set them easier, and then only the ones which arent null get parsed into the request

How to cancel a response stream 'mid-stream'?

I've been looking at the code for Ollama and working with its console for a bit and trying to get my head wrapped around how to perform the .NET equivalent of Ctrl+C in the API. Is there such a command?

The goal that I'm seeking is to have the ability to "stop" the current stream of text for a given response so that it can then be ready to take in another request from the API.

This is that use case where the user is like "oh wait this is the wrong way to go" and wants to shut off the response so a new message can be sent. The optimal scenario would be a cancellation token or a specific command that can be sent to shut down the current response. I have thought about simply shutting down the current connection and re-connecting but that seems pretty inelegant.

I like the stream response. I most definitely want to use it. Any thoughts on "shunting" or "stopping" a stream? Thanks!

How to know when completion has ended with Chat

Hello! I hope you are well. Really cool useful wrapper for Ollama, we use it now that Ollama is available on Windows. I wanted to know if there is a way to know when the completion has ended (no more tokens generated) with the Chat class.

public class OllamaClient
{
    private const string OllamaApiUri = "http://localhost:11434";
    private const string OllamaModel = "DaVinci-v1:latest";

    private readonly OllamaApiClient _ollama = new(new Uri(OllamaApiUri))
    {
        SelectedModel = OllamaModel
    };

    private Chat? _chat;

    public async Task Setup(Action<ChatResponseStream> streamer)
    {
        var models = await _ollama.ListLocalModels();
        Console.WriteLine("Found the following available models : ");
        foreach (var model in models) Console.WriteLine(model.Name);

        _chat = _ollama.Chat(streamer);
    }

    /// <summary>
    /// Asks the model to generate a completion based on the input
    /// </summary>
    public async Task PerformInference(ChatRole chatRole, string input)
    {
        await _chat.SendAs(chatRole, input);
    }
    
    // How do we know when the inference is over?
}

I'd like to be able to perform some code when the completion is over.

Is there a way to know when the completion has completed?

Code docs

Please add summaries to the IOllamaApiClient interface's methods and properties. :)

Should use relative paths, not absolute

It should use api/generate, not /api/generate.
It doesn't work if ollama has a base path (like http://ollamaserver/ollama/api/generate), which is common with reverse proxies.

Full-Framework-Support (.NET 4.x)

Hi Team,

We are working on .Net Framework 4.7.2. Any plan for supporting this?

Thanks,
Rajesh K.

Convert ChatRequest Messages to IList

Currently the Messages property in the ChatRequest class is just a IEnumerable
This makes it "annoying" to add another message
It would make it easier if it was something like a IList
Coming from the Azure OpenAi packages, there the Messages are a IList too, with which you can just use .Add()