marcominerva/openaiembeddingsample

An example that shows how to use Semantic Kernel and Kernel Memory to work with embeddings in a .NET application using SQL Server as Vector Database.

OpenAI Embeddings Sample

An example that shows how to use Semantic Kernel and Kernel Memory to work with embeddings in a .NET application using SQL Server as Vector Database.

The embeddings are stored in a SQL Server database and the Vector Search is efficiently performed thanks to COLUMNSTORE indexes.

To execute the application:

• Create a database in SQL Server
• Open the AppCostants.cs file and set the connection string to the database and the other required parameters. This example assumes you're using Azure OpenAI, but you can easily update it to use OpenAI or whatever LLM you want. Take a look to Kernel and KernelMemoryBuilder configurations in the Program.cs file
• Import some documents into the memory (search for await kernelMemory.ImportDocumentAsync in the Program.cs file

Refer to Program.cs to see how document chunking is performed and how embeddings are calculated, stored and retrieved from the database using Kernel Memory.

If you want to see a manual (explicit) approach to embedding and Vector Search using SQL Server, refer to the manual-approach branch.

openaiembeddingsample's Issues

Update to Semantic Kernel v1.0.1

For me, no error, but both tables [SKMemories] and [SKMemoryCollections] are empty

Hello,

Thanks for your share. I tried to run the program as it is. I am not seeing any error while trying to uplaod a pdf file, but both tables [SKMemories] and [SKMemoryCollections] are empty and no memories generated.

Any suggestion please?

info: Microsoft.KernelMemory.Handlers.TextExtractionHandler[0]
Handler 'extract' ready
info: Microsoft.KernelMemory.Handlers.TextPartitioningHandler[0]
Handler 'partition' ready
info: Microsoft.KernelMemory.Handlers.SummarizationHandler[0]
Handler 'summarize' ready
info: Microsoft.KernelMemory.Handlers.GenerateEmbeddingsHandler[0]
Handler 'gen_embeddings' ready, 1 embedding generators
info: Microsoft.KernelMemory.Handlers.SaveRecordsHandler[0]
Handler save_records ready, 1 vector storages
info: Microsoft.KernelMemory.Handlers.DeleteDocumentHandler[0]
Handler 'private_delete_document' ready
info: Microsoft.KernelMemory.Handlers.DeleteIndexHandler[0]
Handler 'private_delete_index' ready
info: Microsoft.KernelMemory.Handlers.DeleteGeneratedFilesHandler[0]
Handler 'delete_generated_files' ready
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Queueing upload of 1 files for further processing [request 2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298]
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
File uploaded: LIC.pdf, 189007 bytes
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Handler 'extract' processed pipeline 'default/2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298' successfully
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Handler 'partition' processed pipeline 'default/2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298' successfully
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Handler 'gen_embeddings' processed pipeline 'default/2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298' successfully
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Handler 'save_records' processed pipeline 'default/2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298' successfully
info: Microsoft.KernelMemory.Pipeline.BaseOrchestrator[0]
Pipeline 'default/2a4e1d6d2dd842f29957ccda4e9eeeff202401230325092344298' complete

Question: how much payment I made
info: Microsoft.SemanticKernel.Connectors.OpenAI.AzureOpenAIChatCompletionService[0]
Prompt tokens: 93. Completion tokens: 11. Total tokens: 104.
warn: Microsoft.KernelMemory.Search.SearchClient[0]
No memories available
I'm sorry, I haven't found any relevant information about that can be used to answer your question

Question:

Experimenting with SQL Server Connector for Semantic Kernel Memory

Semantic Kernel Memory does not support SQL Server. Try using the connector available At https://github.com/kbeaugrand/SemanticKernel.Connectors.Memory.SqlServer.