Comments (5)
Yes, this is definitely a Service Fabric issue. I believe that Service Fabric has a max size on transactions. The error message and user experience they provide for this is very poor. I would suggest adding items in batches. Depending on your item size, add say 1000 items in a transaction then commit, and repeat that in a loop. As long as that works for your consistency model, of course.
from service-fabric-indexing.
from service-fabric-indexing.
That is interesting. What is the size of your main items, and what are the indexes you are using? Service Fabric has a max size on transactions and/or if transactions take too long they can be aborted.
FilterableIndex is relatively small - one row per item using your original key plus the property being indexed. These will generally not cause problems unless the property being indexed on is large. In Service Fabric, keys should always be small and I use the property being indexed (e.g. item.Name) as the key for the FilterableIndex (which is itself a reliable dictionary).
SearchableIndex can be very large - N rows per item where N is the number of "words" in your string. If you have even moderate length items (e.g. >100 words) then it can easily explode the size of the transaction.
If neither of the above are true, then your best bet is to post the issue on the Service Fabric repo (https://github.com/Microsoft/service-fabric).
from service-fabric-indexing.
I have tried 1600 items with indexed in my PR #8 , it's working.
I think that it maybe is performance issue inside SF.
from service-fabric-indexing.
Any progress on this? I need to add 100k+ records to my collection and having no luck getting past this issue. I have a single index on a groupId property and adding my items with an integer id as the key.
If I insert the items into a standard reliable dictionary it succeeds, but takes around 15 seconds to complete. I need the ability to get all items for the groupId though, so using reliable collections without this library isn't really an option.
When I do try to use this library, the error I get is a bit more specific
"Transaction 131735920369201969 was internally aborted by the replicator as it was active for too long and blocked a checkpoint"
It does work if I chunk my collection into chunks of 100, but it ends up taking >10min to add the records. At that point I'm back to using sql server as that is a severe loss of performance attempting to "cache" my data.
from service-fabric-indexing.
Related Issues (7)
- Stemmer, tokenizer, thesaurus, etc HOT 2
- [NEW FEATURE REQUEST] Generate index for old data HOT 5
- [NEW FEATURE REQUEST] FilterableIndex support unique index HOT 1
- Allow Nullable values for Index HOT 1
- Correctness problem and suggestions HOT 3
- Please, add recommendation regarding how to work with partitions
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from service-fabric-indexing.