Synchronise local and server id

[jcavar]

Synchronise local and server id

[dimazen]

Hello, @jcavar!
How about managing two mappings (based on a single, probably, base mapping):

• mapping A for sending and receiving a client-generated message. It will use clientId as a primary key which will prevent duplication
• mapping B: A that will use the same properties as A expect that for the primary key you'll be using an id. This will handle duplication resolution for the rest.
  Of course it is possible only in case you have a separate responses (i.e. not a single stream) for posting and receiving messages.

Do you have a single stream for the backend communication?

[jcavar]

Hmm, that an is interesting idea! I think we could have partial success with that.

In some places, it is simple to understand which mapping we need, in others - where we get all of the messages - it might be more complicated, but still possible.

[dimazen]

Keep me updated. I think we can come up with some solution :)

[jcavar]

Just an update on this. I haven't been able to get it working properly by using different mapping.
In a simple case when we are processing only Message entity, it works well. However, when we are processing entity that has Message as relationship, there is no way to know which mapping to use. An option might be to inspect response and see if it contains clientId but this is quite ugly as we would need to cover each edge case by itself.

The other option that I tried is updating database object in FEMDeserializer delegate invocation. This didn't work either, because cache is already constructed at that point and contains only objects with required identifiers.

[jcavar]

I think I managed to get this working by using willMapObjectFromRepresentation delegate method. In that method:

1. Check if representation contains clientId
2. Get normal id from representation
3. Check if there is an object with same clientId in database
4. If yes, update its id, and add to deserialiser store.

This has performance penalty, but in our use case, it should be fine.
Now, for this to work, I had to make 2 changes in library itself:

1. willMapObjectFromRepresentation needs to be called before inserting object in database. This gives us ability to act and avoid duplicate
2. FEMManagedObjectStore should allow adding object that was not inserted (isInserted = false)

I will follow up with pull request, but as far as I could see, those changes don't break anything. But we can discuss that as part of PR.

[dimazen]

Went through your issue one more time. Can you a bit clarify what do you mean by a dynamic primary key? I can't fully understand your case when Message is a relationship.

Lets say we have some parent entity (lets name it a ChatRoom for now). Looks like that you're receiving a sort of a response regarding messages in this chat room, right? And issue is that backend doesn't provide your local ids but uses his clientId instead. Therefore it is tricky to figure out which Message in the responses related to the corresponding Message in the database, since database can already have the same Message but without a clientId.

Is that correct? Also does backend sends it's own ids all the time? i.e. same message that you've send with ID = 352 and no cliendId (not assigned yet) can come back with ID of 2304 and cliendId = A1 (for example).

[dimazen]

@jcavar also, why you can not use clientId in this case since you're using it as a primary key anyway. Or am i wrong?

[jcavar]

Backend does provide locally generated id but the problems is following:
Let's say that we have 2 API calls:

1. api/messages
   Response of this one is in this format: [{id: 1}, {id: 2, clientId: xy}]
2. api/chats
   Response of this is in this format: [{id: 13, lastMessage: {id: 2, clientId: xy}}, {id: 14, lastMessage: {id: 1}}]

Now, in first case, it is easy to split array into messages that have clientId, and another that doesn't have it - and then use different mapping on them.
In second case, it is little more challenging - if not impossible when you have 2 relationships that could have clientId (e.g lastMessage, firstMessage)

Backend always sends their ids, and only returns clientId to sender (device that sent the message).

[dimazen]

@jcavar ok, dummy question: why we can not use "id" as a primary key? Because this id can be the same but clientId will be different?

[jcavar]

We can't use id as primary key in this scenario:

1. Create message locally (id = nil, clientId = something generated on client)
2. Send message but not receive response (e.g lose internet connection)
3. Call api/messages

In step 3 if we use id as primary key we would end up with duplicated message.

[dimazen]

@jcavar and cliendId is not a unique value per message but rather represents a client's device id (or some similar concept), correct?

[jcavar]

It is unique id.

[dimazen]

@jcavar ok, here is what i end up with:

1. we need to set id as a primary key, since it is always presented in the backend response, therefore we don't need to handle it somehow additionally.
2. we should also subclass FEMManagedObjectStore and do a following trick:

• in the OurObjectStore do a following trick:

// declare private object cache
@property (nonatomic, strong) FEMObjectCache *messagesCache;
@property (nonatomic, strong) FEMMapping *messageMapping;

- (void)prepareTransactionForMapping:(nonnull FEMMapping *)mapping ofRepresentation:(nonnull NSArray *)representation {
  [super prepare...];
  
  // here we can inspect mapping to see whether it includes messages to handle our special case: 
  NSSet <NSString *> *names = FEMMappingCollectUsedEntityNames(mapping);
  if (![names containsObject:@"Message"]) {
    // default case, no extra handling is needed.
    return;
  }
  
  // now let's alter Message's mapping to use clientID as the primary key to collect presented clientIDs 
  FEMMapping *mappingCopy = [mapping copy];
  // here we're flattening the hierarchy because it is not clear where exactly our mapping lies.
  for (FEMMapping *mapping in [mappingCopy flatten]) { 
    if ([mapping.entityName isEqual:@"Message"]) { 
      // this was previously set to `id` so let's change it to a clientId to grab all primary keys below.
      mapping.primaryKey = @"clientId";
      // we also need to use this mapping later on so let's save it.
      self.messageMapping = mapping; 
      break;
    }
  }

  // Now when all changes are done we can collect primary keys for clientId for a proper prefetch. It won't hurt us, since prefetch is lazy. 
  self.messagesCache = [[FEMObjectCache alloc] initWithMapping:mappingCopy representation:representation, context: self.context];
}

- (NSError *)commitTransaction {
  // don't forget to cleanup
  self.messagesCache = nil;

  return [super commitTransaction];
}

// Now lets do the main trick:
- (id)registeredObjectForRepresentation:(id)representation mapping:(FEMMapping *)mapping {
  id object = [super registedObjectForRepre...];
  // if either object is not nil or it is not a mapping - let's skip it's handling.
  if (object != nil || ![mapping.entityName isEqual:@"Message"]) {
    return object;
  } else {
    // it is a message, and it is nil, i.e. no object with such id is known to our database
    // so we can try to fetch it by the cliendId 
    Message *localObject = [self.messagesCache existingObjectForRepresentation:representation mapping:self.messageMapping];
    // from that point we can return our localObject. 
    return localObject;
  }
}

[jcavar]

Quite interesting! I will try it out and come back.
Thank you!

[dimazen]

It also might be useful to add one more method that I forgot to put initially:

- (void)registerObject:(id)object forMapping:(FEMMapping *)mapping {
  [super ...];
  
  if ([mapping.entityName isEqual:@"Message"]) { 
    id primaryKeyValue = [object valueForKey:self.messagesMapping];
    if (primaryKeyValue != nil) {
      [self.messagesCache addExistingObject:object forMapping:self.messagesMapping];
    }
  }
}

This gonna store your new object properly in case there is a cliendId. It prevents duplicates creations due to two separate objects caches.

Also, put a self.messagesMapping = nil in commitTransaction to finish cleanup.

[jcavar]

I don't see methods that you implemented available on FEMManagedObjectStore. (e.g. prepareTransactionForMapping) Are you suggesting to update library to support those methods?

[dimazen]

@jcavar can you make sure that you have:

#import <FastEasyMapping/FEMObjectStore.h>
#import <FastEasyMapping/FEMRepresentationUtility.h>
#import <FastEasyMapping/FEMMappingUtility.h>

All of the aforementioned methods are presented in the framework already :)

[jcavar]

I think I have that, could you link to them here? I can't find them even when searching repo.

[dimazen]

@jcavar weird. They should be in the public header:

• https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Cache/FEMObjectCache.h
• https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Store/FEMObjectStore.h
• https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Store/FEMManagedObjectStore.h
• https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Utility/FEMRepresentationUtility.h

[jcavar]

I still don't see - (void)prepareTransactionForMapping:(nonnull FEMMapping *)mapping ofRepresentation:(nonnull NSArray *)representation

Did you maybe mean:
https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Store/FEMObjectStore.h#L28?

[jcavar]

I don't see - (id)registeredObjectForRepresentation:(id)representation mapping:(FEMMapping *)mapping either.

Did you maybe mean:
https://github.com/Yalantis/FastEasyMapping/blob/master/FastEasyMapping/Source/Store/FEMObjectStore.h#L107?

[dimazen]

@jcavar i’m so sorry. Turns out I was looking at the outdated branch locally, so that’s the source of the confusion. You’re totally correct with your assumptions. I’ll post correct code tomorrow in the morning - have no access to my laptop at the moment

[dimazen]

@jcavar I'm looking into a solution, since latest API because of the changes doesn't provide representation to the ObjectStore anymore and parses primary keys on it's own instead which is a bit sad, since we can not inject our code anymore. Will get back with a solution in an 1-2 hours

[dimazen]

@jcavar sorry for a delay. Can you check out my fork here? You can grab latest master and checkout FallbackObjectStore.

Reason why I didn't make those changes in the main repo is because of the compatibility reasons: initially, I moved all of the JSON-related stuff to the deserialiser out of the stores to enable better extensibility and less code receptions. Thus I was able to write another one store for the realm without duplicating json parsing code. Your case is a bit special since requires passing this information back to the stores (as it was before). Not sure at this point that main repo should do this, but you should be fine with the fork, since there are no plans on FEM's ObjC version updates.
However, I'll try to address that in the Swift version that is in the draft state atm.

[jcavar]

Yes, I understand. We would like to be on official version of the library to be able to update to new versions easily. I understand our use case is little special, so I think we will keep our implementation as it required less library changes and it seems that it works well.

Thank you for your time and support!