Upon RLMObject mutation, UI gets blocked completely

[roobiq]

Upon RLMObject mutation, UI gets blocked completely

[bigfish24]

I would recommend doing any Realm mutation on a background thread that will notify an RBQFRC instance. Note too (it's on the bottom of the README in more detail) that you need to restrict the mutations to either main thread or background thread, if you do some on background and some on main, a deadlock can occur. So the recommended best practice is always background thread when editing a Realm object with a notification.

[haitham-reda]

thanks @bigfish24 for the fast reply.
Actually i'm moving all the -fetch and mutation to a custom RACScheduler which is a background scheduler, i have 2 issues here:

• the UI still gets blocked
• the optimum way to handle jumping to mainThread again, so for example i have a mark as favorite RACCommand which is triggered from mainThread anyway, so should i pass the objectId here to the foreground thread, refetch that specific object from Realm (on main thread ) and -commitTransaction again ? would it be reflected in fetchedObjects immediately ?

thanks.

[LeffelMania]

Can you go into further detail regarding how/why a deadlock can occur if you mix background and main thread transactions? I think fixing this library (if possible) is preferable to worrying about what thread Realm transactions are done through a whole project. One of the big attractions of Realm is its simpler threading model, it would be shame to complicate it.

[bigfish24]

@LeffelMania happy to explain!

The first "problem" is that the index change calculation based off which objects were edited needs to happen synchronously after the Realm write transaction. If it were async then it is possible the index change calculation would overlap with another Realm write, resulting in index values that are based off of a mix of two database states.

The second "problem" is that after the FRC calculates the index changes, it then passes those to the delegate. If the original write and processing (synchronously from the write caller) was on a background thread, then the delegate call is async'd to the main thread... however, the FRC has to wait for the final delegate call (controllerDidChangeContent:) to return on the main thread before it can return.

The reason for this is that this is where you would call endUpdates on the tableview, which will then trigger calls to numberOfSectionsInTableView:, numberOfRowsInSection:, etc. All of those will then ask the FRC for the appropriate values, which means the FRC must still be in the same state as the most recent change processing. If controllerDidChangeContent: was called async, a new write could occur triggering the FRC to process the change and update its internal state (which manages the sections), causing an assertion failure from the table view, which expects the FRC to return the correct values based off of which changes were passed to it between beginUpdates and endUpdates.

Hopefully this makes sense, but the gist is that the Realm write--> FRC processing --> delegate calls --> tableview update all need to occur synchronously so that the state doesn't change out from under it.

The deadlock then can happen if you perform a background write, which means the FRC delegate calls happen as dispatch_async to the main thread, but if a main thread write had queued up in the middle of the various delegate calls, it will be waiting on the Realm background write to finish which is waiting on the FRC to finish which is waiting on the async delegate calls to finish, which are behind the main thread Realm write that is waiting for the original Realm transaction to finish.... hence a dead lock.

To prevent this all of your changes can be on the main thread or all on a background thread. It is the mixing of the two that can result in a deadlock.

The proper way to fix this is through version pinning within Realm. If it were possible to have two Realm instances that were pinned to the before and after state of a specific commit you could do all the FRC processing for that commit async, allowing Realm to advance and trigger its own FRC processing off its before/after state.

Luckily, a colleague of mine at Realm is getting very close to finishing fine grained notifications similar to RBQFRC but that work off pinned versions, getting around the limitation.

[LeffelMania]

At a high level I understand I think, but I'd have to dig in for a while to fully grok it. Is it your position, then, that this problem is unsolvable due to the necessity of synchronous execution here? Or would it be possible to add some state to the calculation logic to detect and avoid deadlock?

[bigfish24]

@LeffelMania I was editing my post to add info specific to that question:

The proper way to fix this is through version pinning within Realm. If it were possible to have two Realm instances that were pinned to the before and after state of a specific commit you could do all the FRC processing for that commit async, allowing Realm to advance and trigger its own FRC processing off the before/after state for the new commit.

The current Realm Cocoa API doesn't really offer a good way to do this, even though internally Realm has versioning. I could drop down and use Realm Core's C++ API to accomplish this, but it would be redundant since a colleague at Realm is working on that for fine-grained notifications.

Once that is released (most likely late Feb), you could switch and not use RBQFRC unless you need sectioning (since Realm results don't yet support grouping).

In the mean time, the simplest is just to do all Realm writes that trigger FRC processing on a background thread.

[bigfish24]

@haitham-reda I am not that familiar with ReactiveCocoa, but this is the ideal pattern:

1. Perform Realm mutation on background thread (via transaction)
2. On commit of the transaction, FRC synchronously processes changes on the same background thread
3. Changes are passed async onto the main thread from FRC
4. FRC delegates call necessary UITableView methods to apply changes on the main thread
5. Final FRC delegate controllerDidChangeContent: calls UITableView endUpdates and returns, which ultimately allows the original Realm transaction to return since the FRC is waiting on this

As long as all Realm mutations that will trigger the FRC occur on a background thread, this process will occur and no deadlocks happen.

[haitham-reda]

@bigfish24 , i have tried a lot of solutions for background threading, finally i've got back to - (void)changeWithNotificationInTransaction:(RBQChangeNotificationBlock)block, steps:

RLMRealm *rlm = [RLMRealm defaultRealm];
Song *songToFollow = [Song objectInRealm:rlm forPrimaryKey:[self.viewModel.model.songId copy]];
[songToFollow changeWithNotificationInTransaction:^(Song*  _Nonnull _song) {
 _song.favorite = FAVORITE_REVERSED(_song.favorite);
}];

then updating results in 2 views.
so, as i have more than 8,000 objects without fetchLimit, i get the UI blocked completely for 6-7 seconds ( iPhone 6+ ) and huge disk IO , as in the screenshot:

am i getting it wrong ?

[bigfish24]

What is FAVORITE_REVERSED() function doing?

[haitham-reda]

ah sorry, it's a macro to check nullability and reverse status ( 0 <--> 1 ).
i've been digging more, and the block is in -[RBQRealmNotificationManager sendNotificationsWithRealm:entityChanges:] method, it takes 28 seconds.

further more, i believe it's the semaphore in -[RBQFetchedResultsController calculateChangesWithAddedSafeObjects:deletedSafeObjects:changedSafeObjects:realm:] or rebuilding the cache

[bigfish24]

@haitham-reda well that method is where RBQRealmNotificationManager is passing the changes from the current Realm write transaction to listeners, of which RBQFetchedResultsController registers itself as a listener, so it takes the changes and then processes them to identify the index changes.

If this processing is taking awhile, my best guess is that you are passing a lot of changes and thus RBQFRC is going through them as expected. The way RBQFRC is designed is that the change processing is based primarily on the number of changes, versus how many objects are in the fetch.

Another thought I have though is that you might not have RBQFRC setup correctly, so it is recreating its internal cache for every change, versus simply using the existing cache and processing the change.

Is there any way you can share more of the code/project with me? It is hard to debug without more detail...

[haitham-reda]

@bigfish24 objects changes are not many, usually 1 object, but it's definitely the number of objects in the fetch, as i've been using it app wide without any issues, but only in my full list tableView which are 8,000+ objects this problem occurs ( it's a business requirement to fetch them all in list ) , so please advice building up RBQFRC correctly so it doesn't rebuild cache each time, as possibly that's what's happening now form the amount of data written to disk ( ~ 14MB )

i can share snippets privately if you want to .

[bigfish24]

@haitham-reda yeah happy to handle it privately, send what you are comfortable with to [email protected].

I did a quick test with the example app, which by default creates 1000 objects for the fetch and splits them into two sections 10 in the first section and the remaining in the second. For the first test, I adjusted the total number of objects but simply deleted the objects in the first section:

Processing Diff: 0.019833 (1k objects, with 10 changes)
Processing Diff: 0.076833 (10k objects, with 10 changes)

This test, I adjusted the deletion to be the second section, which highlights that lots of changes are the main contributor to the processing time:
Processing Diff: 0.089898 (1k objects, with 990 changes)
Processing Diff: 1.731977 (10k objects, with 9990 changes)

[haitham-reda]

i wish to see those numbers here :) mine is 28 seconds.
thank you for your time, i will email you some snippets, meanwhile is there any way to prevent cache rebuilding ?

[bigfish24]

When you create an instance of RBQFRC, if you pass a cache name the cache will be persisted to disk, but even if you don't the cache is held in-memory, so that is why I am a bit stumped on why the cache would be gone...

But if it is gone, then it would lead to very slow processing since the cache has to go through every object. If it is rebuilding the cache for every change, then this would quickly balloon into the 28 seconds you are seeing.

Are you using a cacheName in initWithFetchRequest:sectionNameKeyPath:cacheName:?

[haitham-reda]

i'm pretty sure it re-writes it every time, that's the only explanation to the 14MB IO to disk i attached in yesterday's screen shot.
yes, HYG:

RLMRealm *realm = [RLMRealm defaultRealm];
    self.predicate = [NSPredicate predicateWithFormat:@"song_id != nil"];
    self.fetchRequest = [RBQFetchRequest fetchRequestWithEntityName:[Song entityName]
                                                            inRealm:realm
                                                          predicate:self.predicate];

    self.desc = [RLMSortDescriptor sortDescriptorWithProperty:@"artist_name" ascending:YES];
    [self.fetchRequest setSortDescriptors:@[self.desc]];

    _fetchedResultsController = [[RBQFetchedResultsController alloc]
                                     initWithFetchRequest:self.fetchRequest
                                     sectionNameKeyPath:@"artist_name"
                                     cacheName:kRBQCacheNameForFullList];

    [_fetchedResultsController setDelegate:self];

[bigfish24]

Can you run the time profiler to better pin point the method that is taking awhile?

Also, are you calling performFetch multiple times? This method triggers the cache to be rebuilt if the number of items in the cache doesn't match the number returned from Realm.

[haitham-reda]

no it's called only once, i will proceed with time profiling as soon as i get to work computer tomorrow morning for developer key :S
but as i'm digging, it's really deleting the cache each time due to difference in fetchRequest.hash

[bigfish24]

That's very odd, as I am not sure how it is possible for the cache to be deleted and rebuilt without calling performFetch multiple times.

Are you deleting Song objects in Realm without notifying RBQFRC?

[haitham-reda]

Not actually, all deletions are using

[obj changeWithNotificationInTransaction:^(Song*  _Nonnull object) {
                    object.favorite = @(NO);
                }];

but i add objects from the API without notifying RBQFRC, parse JSON to model then using
-createOrUpdateInRealm

[bigfish24]

Can you use the notification method for the createOrUpdate? This does mean the cache is getting out of date.

[haitham-reda]

createOrUpdate happens only one time till user pull to refresh which invalidates cache anyway because i receive all objects again from the API.
But after more digging i could spot the problem, there are 2 issues:

• the cache was rebuilt each time, because i have 2 tabs, 1st tab for favorite songs only ( favorite = @(YES)) and the other for full list of Songs, those are the 2 views where i -performFetch, the cache was rebuilt each time because the entityName is the same but the count is different ( i.e 7 favorite songs in the 1 view meanwhile i have 8,000 songs in the second view tableView ), so it rebuilds it due to count difference while matching entityName.
• even when manually commented out cache rebuild, the real problem that takes time is - (RBQSectionChangesObject *)createSectionChangesWithChangeSets:(RBQChangeSetsObject *)changeSets state:(RBQStateObject *)state

the for loop in there for sectionNames if sectionNameKeyPath is present takes ~27 seconds ( 3321 sections :S ) and that's the real issue, unable to think how to resolve it yet. I guess the timings you posted yesterday for section-less objects or very few of them. any suggestions ?

[jaylyerly]

Any update on the deadlock situation? I'm doing a new project with Realm, using RealmTableViewController (https://github.com/bigfish24/ABFRealmTableViewController.git) which uses RBQFetchedResultsController under the sheets. Multiple instances of the RealmTableViewController end up deadlocking down in RBQFetchedResultsController.

[haitham-reda]

@jaylyerly i think it's different situation than what i had here, this issue was mainly due to the huge object/tableView section numbers : ( 3,000+ ) sections, which i ended up using core data.
However i still believe that Realm & RBQFRC are great, i won't hesitate using them again with smaller object graph/count.