foundatiofx/foundatio

Pluggable foundation blocks for building distributed apps.

foundatio's Issues

How can i use Foundatio.Lock?

It's like this ?

somecode to get lockprovider

   var rediscco = new RedisCacheClientOptions() { ConnectionMultiplexer = muxer };

 _lockProvider = new CacheLockProvider(new RedisCacheClient(rediscco), new RedisMessageBus(new RedisMessageBusOptions() { Subscriber = rediscco.ConnectionMultiplexer.GetSubscriber() }));

some code to get lock

 public static  async Task<ILock> AcquireAsync(string name,
        TimeSpan? lockTimeout = null,
        CancellationToken cancellationToken = default(CancellationToken),
        bool continueOnCapturedContext = false)
    {
        var @lock = await _lockProvider.AcquireAsync(name, lockTimeout, cancellationToken)
            .ConfigureAwait(continueOnCapturedContext);
        return @lock;
    }

user lock

  var l = LockHelper.AcquireAsync(key, TimeSpan.FromSeconds(5)).Result ;   
  ............my code to process stock............
  l.ReleaseAsync();

MessageBus Subscriptions

Hey,

Great foundations here. I'd be tempted to use this for a project i'm working on (which currently just uses the standard ServiceBus nuget) however we're using the Topics and Subscription functionality.

The AzureServiceBusMessageBus implementation uses the subscription to determine the channel to push to the multiple subscribers however there doesn't seem to be a way to listen to individual subscriptions being exposed. Would there be any interest in this functionality being added?

Redis and InMemory implementations for RenewLock()

Already implemented in Azure Service Bus and Storage Queues.

Redis: https://github.com/exceptionless/Foundatio/src/Redis/Queues/RedisQueue.cs#L297
InMemory: https://github.com/exceptionless/Foundatio/blob/master/src/Core/Queues/InMemoryQueue.cs#L174

Race condition in AzureServiceBusQueue when multiple processes are creating the same queue

Right now, EnsureQueueCreatedAsync will check if the queue exists, and then if not, it will try to create it. If another process creates the queue in-between these two calls, then the create fails with a MessagingEntityAlreadyExistsException. This is caught and handled and works fine.

However, there's a catch - if another process is currently creating the queue, then it will fail with MessagingException, which is not caught.

The error seen in logs is:

Microsoft.ServiceBus.Messaging.MessagingException: The remote server returned an error: (409) Conflict. SubCode=40901. Another conflicting operation is in progress.

Since MessagingEntityAlreadyExistsException derives from MessagingException, the solution is to catch MessagingException instead.

Create strong named nuget packages

Would it be possible for you to publish strong named nuget packages which foundation depdending on strong named/signed package when they are available, for instance StackExchange.Redis.StrongName. This would be helpful for using Foundation along other packages such as SignalR.Redis.

I hope you will consider doing this.

Support Aliyun OSS storage

Recently when deploying exceptionless, in order to improve the scale out ability, we want to make use of cloud storage. Currently the Foundatio only support windows azure, but the Aliyun is with higher efficiency in China.
When we want to request pr to implement the adapter between Foundation and Aliyun OSS, there is no project for this purpose, and we don't know how to request pr for it.

Add logging documentation

Automatic lock renewal in QueueProcessorJobBase

We've recently added the ability to renew the lock on the queue entry.

It would be great if we could (optionally) do this automatically in QueueProcessorJobBase for long running processes. The alternative would be to pass the queue entry around everywhere to renew - including in scenarios where you don't care if the code has been triggered from a queue or otherwise.

[Caching] GetAllByPrefix like method

I'm trying to cache an ecommerce cart items using InMemoryCacheClient (dev) and in the future RedisCacheClient.
My key is in the form S{sessionId}-V{productId} and the value contains a CartItem object (quantity and other related data)
Adding and removing items are working great but i need to retrieve all the items for a sessionId.
I know the existance of using the Set but i needed the lines separated.
Is there a way to perform this? Maybe by using the ScopedCacheClient (i've done it with InMemoryCacheClient through the Keys prop but it will not work on RedisCacheClient)

Queue Management Improvements

Our current implementation of IQueue has a DeleteQueueAsync method which actually recreates the queue.

This clearly isn't expected behaviour (meaning we can't delete the queue easily in the test tear town) and in addition, we are also performing remote calls in the constructor of the queue which isn't ideal.

We could introduce an additional interface to manage the queues, although we'd have to specifically create every queue and inject it into the container - and the bootstrapper class therefore must ideally be async compatible.

public interface IQueueManager<TQueue> where TQueue : IQueue<T> {
    Task<TQueue> CreateQueueAsync(string queueName /* other params */);
    Task<bool> QueueExistsAsync(string queueName);
    Task DeleteQueueAsync(string queueName);
}

A big down side to this would be it would also make it more difficult to specify additional constructor params for the queue (as used in AzureStorageQueue to control poll interval).

Look into using a AsyncSemaphore for queues as it would be much more efficient

https://github.com/StephenCleary/AsyncEx/blob/master/src/Nito.AsyncEx.Coordination/AsyncSemaphore.cs

Add Job backend implementation for Quartz, Hangfire, Azure, AWS

You can provide the IJob interface and some different job implementations that derive from

1. Quartz
2. Hangfire
3. Azure/AWS
4. Memory

ElasticRepository.OnDocumentsSavedAsync will only send notifications if there were saved docs

There could be added documents, so should we also send out a Added ChangeType notification? I almost think that we should have an event handler for Documents Saved and DocumentsAdded that just handles notifications. This would ensure that it's always executed properly.

https://github.com/exceptionless/Foundatio/blob/master/src/Elasticsearch/Repositories/ElasticRepositoryBase.cs#L280-L281

Update Caching, Queues, Messaging and Locks to be async

Ability to receive a notification/check when a queue item has been completed/abandoned(or removed).

Currently we are abandoning queue items when there is an error which is great. But in some scenarios this doesn't work the greatest for example. When the queue item has been abandoned more times then the retry limit, there is no way to see when the queue item will be removed. As such we can't clean up data data payloads on disk that the queue item points to.

Add integration with Microsoft's logging system

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/logging
Microsoft is using their simple standardized and expandable logging library - https://github.com/aspnet/Logging.
It would be great if you implement some sort of integration between Foundatio's logging library and Microsoft's one to see foundatio log messages in unified way.

Add storage unit test that tries and gets GetFileInfoAsync on file that doesn't exist

Implement paging support and unit tests for GetFileListAsync

We currently retrieve all records for FileStorage and don't implement paging at all for AzureFileStorage. We would need to figure out how to do custom paging with ListBlobsSegmentedAsync

Support for on premise WindowsServer Service Bus

Hi

We have a need to support https://www.nuget.org/packages/ServiceBus.v1_1/ I have done a fork (master...dpvreony-forks:master), it's pretty much a carbon copy of the existing Azure project. I can raise a PR if you're happy.

Thanks

Replace Metrics.Net with AppMetrics

The library Metrics.Net is only available for netfx and only can be deployed on windows servers. But the Foundatio currently supports netstandard 2.0, the Metrics.Net should be replaced with AppMetrics which supports netstandard, so that we can deployed on linux or unix.

Inconsistent stream handling on IFileStorage.SaveFile() implementations

The FolderFileStorage class calls stream.Seek(0, SeekOrigin.Begin) on the passed stream before copying the contents to the file stream. While this might be convenient (in some cases), it is inconsistent with the other implementations of IFileStream (InMemoryStream and AzureFileStream). And it seems presumptive to assume the caller did not positioned the stream correctly for the save operation and to "fix it for them". I can't think of any stream handling functions that do this - including the underlying ones called by these classes: Stream.CopyTo() and CloudBlockBlob.UploadFromStream().

My recommendation is to remove the Seek() operation.

Add reentrant read-write lock

Add reentrant read-write lock to get better performance while doing read only operations.
https://github.com/Netflix/curator/wiki/Shared-Reentrant-Read-Write-Lock

Enqueue with MessageId

Hi

We have a scenario where we need to support duplicate detection for messages being enqueued which is done by messageid https://msdn.microsoft.com/library/azure/microsoft.servicebus.messaging.brokeredmessage.messageid.aspx

I've done the basics in a fork (dpvreony-forks@50f18ce) but it is currently off the back of #71

Thanks

Log work item data in error messages so we can capture that metadata.

We need to find a good way to log the work item data when a work item fails

Implement cache tagging

Adding a cache tagging feature would allow us to invalidate keys when we otherwise wouldn't be able to.

http://stackify.com/implementing-cache-tagging-redis/

Race condition in Azure Service Bus queue

As mentioned by @moswald in #38 ...

1. In parallel, Thread A calls EnqueueAsync() and Thread B calls DeleteQueueAsync()
2. Thread A passes through EnsureQueueCreatedAsync(), ensuring _queueClient is not null
3. Thread B deletes the queue, setting _queueClient to null
4. Thread A tries to use _queueCleint which is now null

There's not really a nice way to handle this without locking. Couple of solutions:

Option 1

Wrap _queueClient in a property which throws a slightly different exception, leaving the user to retry.

Option 2

Implement retry logic (preferably in QueueBase).

The MessagePackSerializer should add a optional parameter to determine whether compress the serialized data.

https://github.com/FoundatioFx/Foundatio/blob/14db27dd26f54ff63841629f115eac47b7d57b12/src/Foundatio/Serializer/MessagePackSerializer.cs

When the value of parameter 'compress' is set true, it should serialize/deserialize with MessagePack.LZ4MessagePackSerializer.NonGeneric, otherwise should use MessagePack.MessagePackSerializer.NonGeneric

Do you have any plans to introduce support for AWS SQS/SNS?

Just like there's storage support for Azure Blob and S3, it would be quite nice to have the opportunity to be cloud-agnostic in what regards to Queues.

ElasticRepository Retry on BulkResult error

Some items may fail due to version change or multiple bulk results running at the same time. We should retry this and possibly return the number of modified documents (bulkResult.Items). Also, we should invalidate the cache before returning..

https://github.com/exceptionless/Foundatio/blob/master/src/Elasticsearch/Repositories/ElasticRepositoryBase.cs#L178

System.Web.Caching.Cache implementation of ICacheClient

Love the building blocks!

It is not a bug, it is a feature request

It would be really cool, if there could be an implementation of ICacheClient via System.Web.Caching.Cache

Add support for DefaultMessageTimeToLive and AutoDeleteOnIdle in AzureServiceBusQueue

Obviously there are many more "Azure ServiceBus Queue" specific properties that are unsupported, but these 2 are the ones I need right now. Do you guys have a strategy for how you might support these types of properties moving forward?

I started to do a pull request to set the properties via the constructor, but bolting on properties piecemeal seems like an unsustainable strategy.

Add Core CLR support

We should be using the new project.json file format and target cross plat.

Update QueueItem to allow consumers to know when it's reached DeadLetter

This would allow consumers to know when it's reached it's max retry account and will be discarded.

We could possibly do this by adding a new event for when an item is placed into DeadLetter.

will it support log4net?

I use log4net in my project, so I wonder if Foundatio.Loggging will have a log4net appder?

Create async initialization method for AzureServiceBusQueue.

A lot of the Microsoft.ServiceBus.NamespaceManager methods called in the AzureServiceBusQueue constructor have async counterparts. Since constructors can't be async, I propose deprecating the current constructor (shakes fist at semver) and adding a new private one that does very little. Then introduce a new factory method: static async Task<AzureServiceBusQueue> factory method that does all the same work the current constructor does synchronously.

i want override some methods of messagebusbase

in my case i both use java and .net ,in the messagebusbase class ,
field: _knownMessageTypesCache
method: GetMessageBodyType,SendMessageToSubscribers
in java the type is different from .net
so i want create an attribute in both java and .net ,use the _knownMessageTypesCache to mapping the type to class in java and to type in .net .

please change field protected and methods virtual,thanks

the code is in https://github.com/FoundatioFx/Foundatio/blob/master/src/Foundatio/Messaging/MessageBusBase.cs

Feature - Bulk Messaging

The current implementations of messaging use message pumps with callbacks which is not as efficient as bulk requesting in some circumstances. High volume scenarios should be bulk fetching to reduce costs as well since you are charged per request.

Publish and subscribe should ideally have corresponding bulk overload/versions as well.

NuGet Package Error

Hi all!,
I'm trying to install package on project with .NET 4.5.2. and when

Install-Package Foundatio -Version 4.2.1183

I've this error

Executing nuget actions took 1,94 sec
Install-Package : Could not install package 'Foundatio 4.2.1183'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5.2', but the package does not contain any assembly 
references or content files that are compatible with that framework. For more information, contact the package author.
At line:1 char:1
+ Install-Package Foundatio -Version 4.2.1183
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

Why?

Idea: It would be handy to have an Events component

Similar to the logging and metrics (probably), it would be useful to have an eventqueue that all of your classes/methods publish to (in the situations where someone initializes it). Then subscribers of the event queue could listen for certain events and perform actions based on it.

This concept could basically leverage existing components you have already. And I haven't looked at your all of your code yet, but maybe this is how your metrics already works.

The use case for this would be something like a Job that subscribes for all SaveFile events... and as soon as it happens, it runs a command. But that same Job may run for DeleteFile events. And so, it can just subscribe to that event too. I know this feature is something that easily could be built into a custom project utilizing other foundatio components. However, I think it would be great to be a foundational pipeline used by your framework.

ASB MessageBus completes a poisonous message rather than DLQ it.

When AzureServiceBusMessageBus receives a poisonous message, it completes it with a log entry. ASB supports deadletter queues. Poisonous messages should be forwarded to the DLQ rather being completed.

RedisMessageBus there is no effect

IMessageBus messageBus = new RedisMessageBus(new RedisMessageBusOptions
{
    Subscriber = connectionMultiplexer.GetSubscriber()
});

await messageBus.SubscribeAsync<SimpleMessage>(msg =>
{
    Console.WriteLine(msg.Data);
});

await messageBus.PublishAsync(new SimpleMessage { Data = "Hello" });

Send work item failed progress messages

We don't send a progress message when a work item fails. What should the status be?

Using Fondatio with .net 4.5.2

I'm looking to be able to use some of the great functionality that I just found in the Foundatio library.

However, we have an application that we currently support on .net 4.5.2. It seems that a few months ago there was a change that Foundatio only supports 4.6 and later.

Is my only choice to use v4.1? Any plans to backport fixes to this branch for people who can't move ahead to 4.6 at this time?

Thanks.

Roadmap suggestion

For the roadmap you could mirror the go-kit roadmap.

Error in documentation, JobRunContext should be JobContext

It looks like in the Job sample in the documentation
https://github.com/FoundatioFx/Foundatio#sample-4

The parameter to RunInternalAsync should be of type JobContext to match
https://github.com/FoundatioFx/Foundatio/blob/master/src/Foundatio/Jobs/JobBase.cs

Azure Storage Queue support

How to declare rabbitmq hostname

Update IQueue.Dequeue implementations to not throw a timeout exception.

Throwing any exception in a tight loop/continuously running code path is not a good idea and can really cause performance issues. I think we should return null instead.

Create strong name nuget packages

Would it be possible to create strong name versions of the nuget packages? I am using Signal.Redis which has a dependency on StackExchange.Redis.StrongName while Foundation.Redis has a dependency on StackExchange.Redis. I can build the Foundation solution with the proper dependency but I would imagine I am not the only one having this issue and the community could benefit from such strong named nuget packages. Thank you.

Add scoped metrics client

This would allow you to scope all the metric names easily.