Giter Site home page Giter Site logo

neosmart / sqlitecache Goto Github PK

View Code? Open in Web Editor NEW
81.0 6.0 12.0 81 KB

An ASP.NET Core IDistributedCache provider backed by SQLite

Home Page: https://neosmart.net/blog/2019/sqlite-cache-for-asp-net-core/

License: Other

C# 96.87% Shell 3.13%
nuget sqlite cache idistributedcache netcore aspnetcore

sqlitecache's Introduction

SqliteCache for ASP.NET Core

SqliteCache is a persistent cache implementing IDistributedCache for .NET and ASP.NET Core projects.

SqliteCache uses a locally stored SQLite database file (taking advantage of SQLite's battle-tested safe multi-threaded access features) to replicate persistent caching, allowing developers to mimic the behavior of staging or production targets without all the overhead or hassle of a traditional IDistributedCache implementation. You can read more about its design and inspiration in the official release post on the NeoSmart blog.

Why NeoSmart.Caching.Sqlite?

The currently available options for caching in ASP.NET Core projects are either all ephemeral in-memory cache offerings (IMemoryCache and co.) -- aka non-persistent -- or else have a whole slew of dependencies and requirements that require at the very least administrator privileges and background services hogging up system resources and needing updates and maintenance to requiring multiple machines and a persistent network configuration.

  • NeoSmart.Caching.Sqlite has no dependencies on background services that hog system resources and need to be updated or maintained (cough cough NCache cough cough)
  • NeoSmart.Caching.Sqlite is fully cross-platform and runs the same on your Windows PC or your colleagues' Linux, FreeBSD, and macOS workstations (unlike, say, Redis)
  • NeoSmart.Caching.Sqlite doesn't need administrator privileges to install - or even any installation for that matter (SQL Express LocalDB, this one is aimed at you)
  • NeoSmart.Caching.Sqlite is a fully contained IDistributedCache offering that is installed and updated alongside the rest of your packages via NuGet, Paket, or whatever other option you're already using to manage your dependencies.

Installation

SqliteCache is available via the NuGet, and can be installed in the Package Manager Console as follows:

Install-Package NeoSmart.Caching.Sqlite

If using this in an ASP.NET Core project, you can install NeoSmart.Caching.Sqlite.AspNetCore (also or instead) to get a convenient helper method for dependency injection (used below):

Install-Package NeoSmart.Caching.Sqlite.AspNetCore

If you install NeoSmart.Caching.Sqlite.AspNetCore you do not need to manually install NeoSmart.Caching.Sqlite as it it will be installed automatically/transitively.

Usage

Using SqliteCache is straight-forward, and should be extremely familiar for anyone that's configured an ASP.NET Core application before. Starting by adding a namespace import using NeoSmart.Caching.Sqlite makes things easier as the editor will pull in the correct extension methods.

If using SqliteCache in an ASP.NET Core project, the SQLite-backed cache should be added as an IDistributedCache type by adding the following to your ConfigureServices method, by default located in Startup.cs, after using the correct namespace NeoSmart.Caching.Sqlite.AspNetCore:

// using NeoSmart.Caching.Sqlite.AspNetCore;

public void ConfigureServices(IServiceCollection services)
{
    ...

    // Note: this *must* come before services.AddMvc()!
    services.AddSqliteCache(options => {
        options.CachePath = @"C:\data\bazaar\cache.db";
    });

    services.AddMvc();

    ...
}

Afterwards, the SqliteCache instance will be made available to both the framework and the application via dependency injection, and can be imported and used via either the IDistributedCache abstract type or the concrete SqliteCache type:

// using Microsoft.Extensions.Caching.Distributed;
public class FooModel(DbContext db, IDistributedCache cache)
{
    _db = db;
    _cache = cache;

    cache.SetString("foo", "bar");
    Assert.AreEqual(cache.GetString("foo"), "bar");

    Assert.AreEqual(typeof(NeoSmart.Caching.Sqlite.SqliteCache),
                    cache.GetType());
}

To take advantage of SqliteCache-specific features or functionality that aren't exposed via the IDistributedCache façade, you'll need to inject SqliteCache into your classes/methods rather than IDistributedCache. For example, to globally clear the cache after performing some operation:

// using NeoSmart.Caching.Sqlite;
public class BarModel(DbContext db, SqliteCache cache)
{
    _db = db;
    _cache = cache;
}

public ActionResult OnPostAsync()
{
    ...
    await _db.SomethingDestructiveAsync();

    // We need to invalidate all the cache, since it's too hard to
    // account for the changes this operation caused for legacy reasons.
    await _cache.ClearAsync();

    ...
}

License

SqliteCache is developed and maintained by Mahmoud Al-Qudsi of NeoSmart Technologies. The project is provided free to the community under the terms of the MIT open source license.

Contributing

We are open to pull requests and contributions aimed at the code, documentation, unit tests, or anything else. If you're mulling an extensive contribution, file an issue first to make sure we're all on the same page, otherwise, PR away!

sqlitecache's People

Contributors

antonycorbett avatar mqudsi avatar rap22tor avatar ravinduwu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sqlitecache's Issues

Register with Unity?

I need to register the Sqlite Cache within a IUnityContainer any clue how this would be done?

Convert `AbsoluteExpiration` to UTC if it isn't already

Hi @mqudsi 👋!

Given how DateTimeOffset.Ticks depends on DateTimeOffset.Offset, and RemoveExpires compares persisted expiry dates with DateTimeOffset.UtcNow.Ticks,

public void RemoveExpired()
{
var removed = (long) Commands.Use(Operation.RemoveExpired, cmd =>
{
cmd.Parameters.AddWithValue("@now", DateTimeOffset.UtcNow.Ticks);
return cmd.ExecuteScalar();

...shouldn't AbsoluteExpiration be converted to UTC via DateTimeOffset.ToUniversalTime, when entries are persisted? 🤔

private void AddExpirationParameters(DbCommand cmd, DistributedCacheEntryOptions options)
{
DateTimeOffset? expiry = null;
TimeSpan? renewal = null;
if (options.AbsoluteExpiration.HasValue)
{
expiry = options.AbsoluteExpiration.Value;
}
else if (options.AbsoluteExpirationRelativeToNow.HasValue)
{
expiry = DateTimeOffset.UtcNow
.Add(options.AbsoluteExpirationRelativeToNow.Value);
}
if (options.SlidingExpiration.HasValue)
{
renewal = options.SlidingExpiration.Value;
expiry = (expiry ?? DateTimeOffset.UtcNow) + renewal;
}
cmd.Parameters.AddWithValue("@expiry", expiry?.Ticks ?? (object) DBNull.Value);

Internal use of async

What are your thoughts on async in Microsoft.Data.Sqlite.Core? I think the following advice is still relevant:

https://docs.microsoft.com/en-us/dotnet/standard/data/sqlite/async

It would be difficult to remove the public Async method signatures on SqliteCache, but perhaps the underlying database operations could be changed to use sync methods and so minimize the async overhead. It's only a very minor consideration.

Adding bulkinsert support

Hi

I've recently run into a problem where I have to insert a large list of objects/keys into the cache database. Using the existing Set and SetAsync methods, this obviously ran pretty slowly. I now have added a SetBulkAsync method, which reduces the time consumed significantly. In my case, it improved a process that took around ~25 mins to about 25 seconds.
This method takes in a IEnumerable of KeyValuePairs and produces a single Sqlite query. This query is then executed just like in SetAsync. In my program I can now seperate my list into smaller chunks and use SetBulkAsync to insert them pretty quickly.

I hope this is useful for others too. Let me know if you prefer the solution any different, I'll gladly adapt.

Thanks for this library!
Elias

Bug when using with sqlite library that is compiled with SQLITE_DSQ=0

Sqlite has the oddity that it treats double quoted strings that don't evaluate to an identifier as strings. This is not sql standard behavior. Since in the initialization routine values are inserted into the meta table that contain strings that are double quoted. Those will fail with a rather cryptic message 'created is not a column name' when the underlying sqlite does not support this odd configuration option.

Since its recommended to compile sqlite with DSQ=0 (i.e. always treat double quotes as identifiers) it would be nice if this could be reflected in the sql statements by replacing double quoted strings by single quoted strings.

Runtime exception after updating to .net7/EF7

See PR #13
I started getting a runtime exception in my project after updating it (and references library projects) to net7/EF7.

The exception I start getting is:

System.Exception: Could not resolve a service of type 'Microsoft.Extensions.Caching.Distributed.IDistributedCache' for the parameter 'cache' of method 'Configure' on type 'MyApi.Startup'.
---> System.TypeInitializationException: The type initializer for 'NeoSmart.Caching.Sqlite.SqliteCache' threw an exception.
---> System.TypeLoadException: Method 'sqlite3_load_extension' in type 'SQLitePCL.SQLite3Provider_e_sqlite3' from assembly 'SQLitePCLRaw.provider.e_sqlite3, Version=2.0.7.1395, Culture=neutral, PublicKeyToken=9c301db686d0bd12' does not have an implementation.

From what I understand this happens if you have a project containing an interface (Microsoft.Extensions.Caching.Distributed.IDistributedCache), and another project containing an implementation of this interface (NeoSmart.Caching.Sqlite.SqliteCache) and a project using the implementation. If the using-project is build with an other version of the interface-project dll then the implementing-project is, then the exception happens. Solutions are said to be clean everything (e.g. delete bin) and rebuild, but the problem is I can't rebuild the NeoSmart.Caching.Sqlite package, and my project is referencing another project that depends on Microsoft.Data.Sqlite.Core 7.0.2 which I'm guessing is where the conflict is, since NeoSmart.Caching.Sqlite package is depending on Microsoft.Data.Sqlite.Core 6.0.1

I created a PR with the package updates which hopefully solves this issue.

Edit: Well, maybe the above isn't exactly right. I did a bit more digging and found ericsink/SQLitePCL.raw#229 - and its more complicated than I have time to get into now unfortunately. However, I did confirm that my issue when away if I remove the reference to my library project which depends on Microsoft.Data.Sqlite.Core 7.0.2. So, there seems to be some conflict when 7 is introduced - and hopefully a dep version update solves it.

Dependency on ISQLite3Provider?

I am working on .net 8.0 web api project. Trying to setup the cache but ran into a dependency that is not documented in the readme...
image
I installed NeoSmart.Caching.Sqlite. Should I continue to use the .net standard version (NeoSmart.Caching.Sqlite.AspNetCore)?

Sqlite in WAL mode vs. use as actual distributed cache

From SQLite documentation on WAL Mode::

"All processes using a database must be on the same host computer; WAL does not work over a network filesystem."

This seems to effectively block the scenario of using AspSqliteCache as an actual distrubuted cache; otherwise it would seem to be usable as a low volume distributed cache with just two or three nodes.
While the focus in documentation is the persistence part of the cache, not the distributed part it wld be interesting to see if limited distribution is actually possible and if/what changes are required to make that feasible.

Allow using custom SQLite3Provider

I have a need to use a custom SQLite3Provider which could be set using SQLitePCL.raw.SetProvider() method. It's a supported scenario and documented here.

Unfortunately SqliteCache references SQLitePCLRaw.bundle_green and makes a call to SQLitePCL.Batteries.Init() which makes it impossible to use a custom provider.

https://github.com/neosmart/AspSqliteCache/blob/9436df9361b0dc95eef01426970fd82a9c20cb19/SqliteCache/SqliteCache.cs#L30-L33

Do you think it's possible to have a reference to SQLitePCLRaw.core instead of SQLitePCLRaw.bundle_green and make it user's responsibility to call SQLitePCL.Batteries.Init() or whatever method they decide to use?

I can send a PR if you are ok with such a change.

Suggestion: Use of vacuum to truncate the database

Hi

I'm using SqliteCache for a project where the software downloads PDF files and convert them to images. The images are cached using SqliteCache.

Thus the database grows a lot with the usage. Although I can clear the cache, how sqlite works make that, not using VACUUM, the size of the database is not reduced once the cache is cleared.

https://www.sqlite.org/lang_vacuum.html

One option is to run the sqlite database with the auto_vacuum=true or, the add the possibility to execute the VACUUM sql sentence after the clear of the cache, at least as an option.

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.