jessebenson/service-fabric-indexing

Adds support for automatic indexing of reliable collections. Supported indexing is filters (exact value matches) and full-text search.

service-fabric-indexing's Issues

[NEW FEATURE REQUEST] Generate index for old data

After add index into a Dictionary with old data, this index can generate indexed data for old data.
Or can rebuild indexed data for whole data include old data and new data.

[NEW FEATURE REQUEST] FilterableIndex support unique index

I can set FilterableIndex just have one filter to one key

Correctness problem and suggestions

Hi - thanks for taking a stab at the indexing functionality which should have ideally been a part of SF packages directly. Here is my understanding based on going over your code.

Here is a scenario where the current implementation on the package might not work

using (ITransaction tx = ..)
{
  1. Update indexed dictionary
  2. Something causes an exception
  3. tx.CommitAsync is not called
}

Based on my reading of the code,

1. Since you are updating the indexes in a callback part of your exposed methods, you might have to support rollbacks on transactions. Its not clear if this will work.
2. The approach which could work might be to build these indexes as part of the commits via the notification channel on the reliable dictionary.
3. Even in (2) - there is a chance that your code could cause a commit to fail. But that helps atleast reduce the correctness problems.
4. With notifications, your can build the indexes on the secondary as and when primary is writing data. This helps improve startup times when secondary is promoted as a primary as we do not have to build index in run async.

Please, add recommendation regarding how to work with partitions

Hi,

It would be interesting to add in the documentation information about how to work with indexes and partitions. Do you have any initial idea of what would be a good pattern?

Thanks in advance.
Kind Regards,
José.

Allow Nullable values for Index

It would be nice to be able to use null values inside the index. For example, if I'm indexing on a string, null values is often a normal case. I don't need to filter by them necessarily, but right now i have to turn all null strings to empty strings or I get exceptions.

Performance issue after adding about 8000+ data

When add a lot of data into a collection with 5 indexes
about 8000+, I get a error:

System.InvalidOperationException: Transaction 131929400802878326 is committing or rolling back or has already committed or rolled back
   at Microsoft.ServiceFabric.Replicator.TransactionBase.ThrowIfTransactionIsNotActive()
   at System.Fabric.Store.TStore`5.AddOrUpdateAsync(IStoreWriteTransaction transaction, TKey key, Func`2 addValueFactory, Func`3 updateValueFactory, TimeSpan timeout, CancellationToken cancellationToken)
   at Microsoft.ServiceFabric.Data.Collections.DistributedDictionary`2.AddOrUpdateAsync(Transaction tx, TKey key, Func`2 addValueFactory, Func`3 updateValueFactory, TimeSpan timeout, CancellationToken cancellationToken)
   at ServiceFabric.Extensions.Data.Indexing.Persistent.ReliableIndexedDictionary`2.OnAddAsync(ITransaction tx, TKey key, TValue value, TimeSpan timeout, CancellationToken token) in 

this issue maybe relate to https://github.com/Azure/service-fabric-issues/issues/435

And I have a workaround ideas :
First provide a BulkInsert method , this method pause adding data into index temporarily
Then rebuild index for new data
Of course, this workaround need this feature: #4

Stemmer, tokenizer, thesaurus, etc

Do you plan to provide support for typical natural language processing utilities like stemmer, tokenizer, thesaurus for different languages in your full text index implementation or is it more like a PoC?