plivesey/rocketdata

A caching and consistency solution for immutable models.

🚀 Data

Rocket Data is a model management system with persistence for immutable models.

Motivation

Immutability has many benefits, but keeping models consistent and making changes is difficult. This library manages the consistency and caching of immutable models. It is intended to be an ideal replacement for Core Data. However, unlike Core Data, it does not block the main thread and does not crash whenever you do something slightly incorrect (see Core Data Comparison). In most setups, the backing cache does not need a schema, and you never need to add migration logic.

Scale

Rocket Data scales extremely well to large numbers of models and data providers. Since it does nearly all of its work on a background thread, you never need to worry about one change slowing down the whole application. You can also choose to stop listening to changes when a view controller is off screen to further increase performance.

The library is optimized for applications that fetch data from an external source, display it on the device, and allow the user to perform actions on this data. It implements an easy model for synchronizing this data in-memory between view controllers and with the cache.

Bring Your Own Cache

With Rocket Data, you can choose your own caching solution. We recommend a fast key-value store, but you can use any store you can imagine. This also makes it easy to add LRU eviction.

Installation

Installation via both CocoaPods and Carthage is supported.

CocoaPods

Add this to your Podspec:

pod 'RocketData'

Then run pod install.

Carthage

Add this to your Cartfile:

github "plivesey/RocketData"

Then run carthage update RocketData --platform ios

NOTE: Currently, --platform ios is necessary for some reason. We are investigating the issue.

Swift Version

We are currently not maintaining separate branches for different Swift versions. You can use an older version of Rocket Data for older versions of Swift though. HEAD currently supports Swift 4.2.

Swift Version	Rocket Data Version
1	Not supported
2.0 - 2.1	1.x.x (untested)
2.2	1.x.x
2.3 (Cocoapods)	1.x.x
2.3 (Carthage)	1.2.0
3 (Easy migration API)	2.0.0
3 (Better API)	4.x.x
4	5.x.x
4.2	7.x.x

NOTE: If you are migrating to Swift 3, consider using version 2.0.0 first, then migrating to 3.x.x. 3.0.0 migrates the code to the new syntax without making any API changes. 3.x.x introduces a better API which is more consistent with the new Swift 3 API guidelines.

Documentation

To get started, you should take a look at the docs.

Consistency Manager

Rocket Data uses ConsistencyManager to manage the in-memory consistency of models. While you never need to access the Consistency Manager directly, understanding how it works will help you understand Rocket Data.

rocketdata's Issues

Missing argument for parameter `dataModelManager` in call...

Attempting to incorporate RocketData in my first app, following along through the sample app and ran into an issue that doesn't crop up in the sample app, but does in my app.

The sample app contains the following code in ChatsViewController.swift:

fileprivate let dataProvider = CollectionDataProvider<UserModel>()

When I put the following in my app, I get the error "Missing argument for parameter dataModelManager in call"

fileprivate let dataProvider = CollectionDataProvider<Episode>()

Both view controllers import RocketData and implement the CollectionDataProviderDelegate protocol. What am I missing here?

Find or Create when saving data in cache

Hey there!

How can I ensure that every model is referencing a copy of the cached data?

For example, imagine Twitter, where each Tweet has a User.

struct Tweet {
  let id: String
  let user: User
}

struct User {
  let id: String
  let someRandomString = String.random() // For example purposes. :)
}

Now when importing from the API, each Tweet in the API response has a copy of its User, but on the iOS client we add a generated property (e.g. someRandomString) and cache this generated property. Every copy of User, even if they have the same modelIdentifier, will have a very different someRandomString value upon import since the value does not come from the server. Is there any way to "find or create" this data smartly?

So far I see two options:

1. Every time I create a user, fetch the copy in the cache manually and set someRandomString from cache without setting it locally. This is problematic since modelFromCache is async.
2. Call updateModel on the DataModelManager which will then let all instances of CollectionDataProvider know that a change has been made and to update from the cache (which will then force all copies of User to sync.)

Any other options/ideas?

One-to-many relationship: retrieve child object from cache

First of all, thanks for the library.

I have two models, both conform to Model protocol:

public struct Person {
    public let id: String
    public let name: String
}

public struct Group {
    public let id: String
    public let persons: [Person]
}

I'm putting a group into the cache:

let person1 = Person(id: "1", name: "jj")
let person2 = Person(id: "2", name: "name")
let group = Group(id: "1", persons: [person1, person2])
let groupDataProvider = ModelDataProvider<Group>()
groupDataProvider.setData(group)

and then trying to take a person out of there:

let dataProvider = ModelDataProvider<Person>()
dataProvider.fetchDataFromCache(withCacheKey: person1.modelIdentifier) { (model, error) in
    if let error = error {
        print("error \(error)")
    } else {
        print("model \(model) \(type(of: model))")
    }
}

However, nil is returned. Looks like we're unable to get child objects out of cache?

Cache Key isn't set if it misses the cache

It's unclear if this is intentional, but seems to be incorrect.

If we miss data in the cache, we still should set the current cache key. That way, if we append/insert data, we'll save these inserts to the cache as well. It seems like whatever data we have is the best data (even if it's empty).

DataProviders don't call delegate for update after first miss?

Hey, thanks for a great library. I was trying to integrate it into my project. I use a wrapper ReactiveProvider<T> around DataProvider<T> for RAC Support. I have my ReactiveProviders in each viewModel, but networking is done elsewhere so I initiate another DataProvider and call setData(_:updateCache:context:) on that data provider, and the ReactiveProvider doesn't get the delegate call for update.

Is this expected behaviour? I saw another issue, is this related to that? #8

Here is the code for ReactiveProvider

class ReactiveProvider<M: SimpleModel> {
    private let provider = DataProvider<M>()

    let signal: Signal<M?, NoError>
    fileprivate let pipe: Observer<M?, NoError>

    init() {
        let (signal, pipe) = Signal<M?, NoError>.pipe()
        self.signal = signal
        self.pipe = pipe
        provider.delegate = self
    }

    func start(with cacheKey: CacheKey) {
        if provider.isPaused {
            provider.isPaused = false
        }
        provider.fetchDataFromCache(withCacheKey: cacheKey.key, context: cacheKey) {
            [weak self] model, error in
            self?.pipe.send(value: model)
        }
    }

    func pause() {
        provider.isPaused = true
    }
}

extension ReactiveProvider: DataProviderDelegate {
    func dataProviderHasUpdatedData<T>(_ dataProvider: DataProvider<T>,
                                    context: Any?) {
        guard let data = dataProvider.data as? M else {
            pipe.send(value: nil)
            return
        }

        pipe.send(value: data)
    }
}

and here is the code that sets the data (in networking part of the app, not in viewModel)

class CacheProvider {
    private var providers: [String: Any] = [:]

    func cache<T: SimpleModel>(object: T?, for key: CacheKey) {
        let provider: DataProvider<T>
        if let p = providers[key.key] as? DataProvider<T> {
            provider = p
        } else {
            provider = DataProvider<T>()
            providers[key.key] = provider
        }
        provider.setData(object, updateCache: true, context: key)
    }
}

CollectionDataProvider.fetchDataFromCache can fail to set the cacheKey

When calling CollectionDataProvider.fetchDataFromCache, If there are no objects in the cache matching the cache key, nil is returned for the object array (I might prefer an empty array), but more importantly, the cache key is not set on the provider, so even though you have a delegate listening to the data provider, it will not get notified of subsequent additions to the collection. Here's the sequence I'm doing:

collectionDataProvider = CollectionDataProvider<StoreItemPendingPurchase>()
collectionDataProvider.delegate = self
collectionDataProvider.fetchDataFromCache(withCacheKey: cacheKey)

This returns nil for the error and nil for the object array. And the data provider's cacheKey is still nil. If somewhere else in the app someone sets data on the same cacheKey, I don't find out about it because my data provider's cacheKey is still nil.

Is this expected? If there some way to set the cacheKey independently of using setData() or fetchDataFromCache()?

ConsistencyManager 5.1 and Cocoapods

Can you please do a new release for Cocoapods to update ConsistencyManager to the latest version? Version 5.1 has an important update, without it there is a crash in Swift 4.

Handling cutdown API objects

I have things working with an ugly hack. I'm looking for a more robust solution.

I have a social media type model with Posts that have a User (i.e. author). The JSON delivered by the server's /feed API contains a cut down version of the user.

/feed:

[
  {
    "content": "Eating dinner :-)",
    "author": {"id":"1", "username": "bob"}}
    …
  }
]

The full list of users is delivered by another API

/friends:

[
  {
    "id":"1", 
    "username": "bob",
    "first_name": "Robert",
    "avatar": "http://example.com/avatar.gif"
    …

My User model contains:

  public var modelIdentifier: String? {
    return "User:\(id)"
  }

In order to prevent the cut-down version from overwriting the full version of the User, I am using the following hack in my User.init?(data):

authorData["id"] = "Author~\(authorData["id"] ?? "")"

Every time I add a child model that could come as a cut-down version I need to remember to add this hack. This hack causes problems when comparing user objects too, as an author no-longer has the same ID as a friend.

Has anyone encountered this issue and can point me towards a more robust solution?

Add a sample app

Can this lib be used in conjunction with Realm?

Is it overkill to use this lib with Realm, is it a supplement or a replacement?
Even with Realm's updated notification capabilities...

Thanks!

Add an overview of the architecture

Include a diagram

Get collectionChanges on setData

Let's say I have a network request that returns, and I call:

dataProvider.setData(data, cacheKey: cacheKey)

I expect collectionDataProviderHasUpdatedData to be called, but it isn't. How else should I get the collectionChanges?

Right now, I'm updating my UI in three separate places instead of one: fetchDataFromCache (on view load), setData (after network requests), and collectionDataProviderHasUpdatedData. 😵

Error "Couldn't find platform in Info.plist CFBundleSupportedPlatforms or binary for ConsistencyManager"

Hello,

I'm encountering the following error when trying to install RocketData through Carthage in a brand new project. It's basically an empty folder with github "linkedin/RocketData" defined in a Cartfile.

2016-10-24 11:42:10.277 xcodebuild[74733:2842940] [MT] DVTAssertions: Warning in /Library/Caches/com.apple.xbs/Sources/IDEFrameworks/IDEFrameworks-11246/IDEFoundation/Execution/RunDestinations/IDERunDestination.m:201
Details: Unable to find platform for /Users/plivesey/Library/Developer/Xcode/DerivedData/ConsistencyManager-ezmcupkggqbxrkawtipbzcwqqolg/Build/Products/ConsistencyManager: Error Domain=DVTFoundationNSBundleAdditionsErrorDomain Code=1 "Couldn't find platform in Info.plist CFBundleSupportedPlatforms or binary for ConsistencyManager" UserInfo={NSLocalizedDescription=Couldn't find platform in Info.plist CFBundleSupportedPlatforms or binary for ConsistencyManager}
Function: DVTPlatform *_IDEPlatformForPathRunnable(IDEPathRunnable *__strong)
Thread: <NSThread: 0x7fa5afd13660>{number = 1, name = main}
Please file a bug at http://bugreport.apple.com with this warning message and any useful information you can provide.

Setting CollectionDataProvider cacheKey without setting data

Hey ! I have two CollectionDataProvider in two different classes (let's say A and B). A is trying to listen at changes made on the collection by B but the delegate is never called. I guess it's because A is not pushing data in the collection so there is no cacheKey for the collection of the class A. If i'm adding a userCollection.setData([], cacheKey: "TEST") in class A, the delegate method is called. Is there a way to register a cacheKey without overriding all data in the collection ?

Why does siblingProvidersForProvider not return current provider?

The code documentation on siblingProvidersForProvider says it doesn't return current provider in the return array but doesn't state why. The view controller has to manually trigger an update to refresh the view after setting the data on current provider, we expected the view controller to receive updates but it took us a while until we found this comment.

We would like the VC to automatically receive any updates even to the current provider, we din't find any issues removing this filter but does anyone see this causes issues if not we can submit a PR?

/**
     Returns all providers with the same cache key as the current provider.
     It does not return the current provider in the returned array.
     */
    func siblingProvidersForProvider(_ provider: SharedCollection, cacheKey: String) -> [SharedCollection] {
        if var weakArray = providers[cacheKey] {
            // We'll use this opportunity to prune the weak array and clean up some memory
            let siblingProviders = weakArray.prune().filter { $0 !== provider }
            providers[cacheKey] = weakArray
            return siblingProviders
        } else {
            return []
        }
    }

Add ability to listen to ids with Rocket Data

For collection data providers, it'd be great to filter on model changes and insert them into the collection if they meet a certain filter. This would be like predicates in core data.

Add some docs clarifying where changes come from

#31

e.g. For network calls and setData, the delegate is not called. The delegate is only called for consistency changes.

Sorting CollectionDataProvider

What is the recommendation for sorting data when using a CollectionDataProvider? Do you have to use a different CollectionDataProvider or cache key for each possible sort order? Or maybe you maintain your own sorted list of objects that is updated whenever the CollectionDataProvider changes?

dataProviderHasUpdatedData is not called when passing a new different model to the DataProvider

In the demo project, I insert the following code in method heyButtonPress()
let user = UserModel(id: 2, name: "new name", online: true)
self.userDataProvider.setData(user)
so when I press the hey button I am expecting dataProviderHasUpdatedData to be called, but it's not.
Am I doing something wrong here?

Update docs to include more information on how models can be updated

We should include some examples here of how people might mutate models. It depends on the modeling solution, but may help put things in context.

https://linkedin.github.io/RocketData/pages/080_updatingModels.html

Manual Installation

CocoaPods and Carthage are awesome tools and make our life really easier, but there are some devs who still don't know how to use them.

It would be cool to add the Manual installation guide in your README.md. You can take a look at my iOS Readme Template to see how you can do it.

DataProviderDelegate

Hi Peter
Is there any reason the DataProviderDelegate is defined with a generic method rather than an associated type?

https://github.com/linkedin/RocketData/blob/master/RocketData/DataProvider.swift#L196-L206

Let's say I want to create a Reactive extension that emit RocketDataDelegate updates
Right now I can write something like this

// given some kind of Generic Emitter class that emit T elements
class Emitter<T> {
  func on(data: T) { /* implementation */ }
}

class RocketDataEmitter<T: SimpleModel>: DataProviderDelegate {
  let emitter = Emitter<T?>()

  func dataProviderHasUpdatedData<V>(dataProvider: DataProvider<V>, context: Any?) {
    guard let data = dataProvider.data as? T? else { fatalError() }
    emitter.on(data)
  }
}

With associated type I could avoid the weird casting

public protocol DataProviderDelegate: class {
  associatedtype T: SimpleModel
  func dataProviderHasUpdatedData(dataProvider: DataProvider<T>, context: Any?)
}

class RocketDataEmitter<T: SimpleModel>: DataProviderDelegate {
  let emitter = Emitter<T?>()

  func dataProviderHasUpdatedData(dataProvider: DataProvider<T>, context: Any?) {    
    emitter.on(data)
  }

Add documentation for projections

Included in updated consistency manager to 2.0.0

If you insert a new item to a collection, other rows aren't updated

A subtle but if you have a collection with ids:

[ 1, 2 ]

and you insert a new 1 at the end (with new data), you end up with:

[ 1, 2, 1' ] instead of [ 1', 2, 1 ]

Should be relatively easy to fix.

Example App with complex models

Has anyone implemented RocketData with complex models and collection data providers other than LinkedIn? Can LinkedIn share or has any open source projects that demonstrate an app with complex data model?

Support for Swift 3.0

Hey, I noticed that you guys are actively adding Swift 3 support for ConsistencyManager-iOS.

Are you planning on adding Swift 3 support for RocketData? Do you need help with that?
Or do you recommend using ConsistencyManager directly?

Is there a way to wipe all models from both cache and memory?

Hey,

When a user logs out, I want to wipe everything so that cached items don't leak into the next session. I went through the docs but couldn't make out if this was supported?

Compiler error when building SampleApp: <unknown>:0: error: no such file or directory: ... WeakListenerArray.swift

Just cloned the repo, opened the included SampleApp workspace, and tried building but it fails with <unknown>:0: error: no such file or directory: '/Users/me/Desktop/RocketData/SampleApp/Pods/ConsistencyManager/ConsistencyManager/DataStructures/WeakListenerArray.swift'

I was able to fix the problem by updating the ConsistencyManager pod from 4.0.0 to 5.0.0: pod update ConsistencyManager

Would be more than happy to submit a quick PR for this @plivesey

Using model objects in Objective-C

Are there any examples of using Models with existing Objective-C models? I am running into this compilation issue try to define @objc protocols.

../RDModel.swift:8:16: @objc protocol 'RDModel' cannot refine non-@objc protocol 'Model'

//RDModel.swift
import RocketData

@objc protocol RDModel: Model {

init?(data: [AnyHashable: Any])
func data() -> [AnyHashable: Any]

}

Release new version for Carthage support

#30

Example CacheDelegate implementations

When trying to start using this, you(me) get in the weeds pretty quick as you need to implement your own caching layer to begin using this.

It would be great if there was an example or few cache delegate implementations to reference or use to help people get started quicker. 😎

Fix CI so it passes

Can u maintain an objective-c version?

love the simplicity and powerfulness of rocketData, I mainly need the notification feature rocketData provide to, it seems the realm and core data can provide the model notification function, what's your suggestion if I have to do the notification model in objective-c? Thanks.

App Store submission fails because of embedded frameworks

First off, I'm using Carthage. I'm not sure if that matters. App submission fails because there are embedded frameworks inside RocketData.framework. Specifically:

• a Frameworks folder full of swift libs
• libswiftRemoteMirror.dylib at the root level

Based on this discussion, I believe the fault is because "Always Embed Swift Standard Libraries" is set to Yes in RocketData.

I didn't have time to fully investigate, but simply deleted those frameworks using a build phase script in my project, like this:

rm -rf $SRCROOT/Carthage/Build/iOS/RocketData.framework/libswiftRemoteMirror.dylib
rm -rf $SRCROOT/Carthage/Build/iOS/RocketData.framework/Frameworks

Can you provide a way to completely clean data

Hi, I'm writing an application that provide different content for logged in and non-logged in users. So each time user login/logout, I need to completely clean all data from RocketData.

Currently, I'm using removeAllObjects of PINCache. But it may cause problem with ConsistencyManager.

Can you provide an easy way to completely clean data? Thank you!

Possible bug, default built would fail for carthage build

I use Carthage to build the project. While in the General Tab of project, Pod_RocketData is labelled required framework, and thus build would fail (missing directory pod_rocketdata in link stage). After removing it, my build succeeded with Carthage. You may want to check into that.

xcode 8.2.1. MacOS Sierra 10.12.2, swift 3

Link Realm video

https://realm.io/news/slug-peter-livesey-managing-consistency-immutable-models/

paging

hi @plivesey
How do you use RocketData in combination with paging? Like for the main Linkedin feed, do you only cache the first page, and then load the following through the network exclusively?

fetchDataFromCache(withCacheKey:listenToModelIdentifier... methods unavailable

In Xcode 10, latest Carthage; example code screenshot with errors.

Add build sample app to travis

I always forget to update the sample app and it keeps breaking. Should add this to travis.

Integrate Codecov

https://codecov.io/

Crash: DataProvider.swift line 92 DataProvider.init(dataModelManager : DataModelManager) -> DataProvider<A>

Hey,
I have started using Rocket Data recently and faced this crash. Please help me to identify what is this issue:-

#0. Crashed: com.apple.main-thread
0  libswiftCore.dylib             0x1089ae5f0 swift_storeEnumTagSinglePayload + 36
1  RocketData                     0x107d3addc DataProvider.init(dataModelManager : DataModelManager) -> DataProvider<A> (DataProvider.swift:92)
2  HDCommons                      0x104b08bdc DataProvider.init() (CacheContextUtils.swift:38)
3  HDCommons                      0x104b07818 CaheContext.__allocating_init() (CacheContext.swift:87)
4  HDCommons                      0x104b0a75c specialized CachedAttributes.save<A>(value:) (CachedAttributes.swift:16)
5  ABCApp                        0x102845d68 specialized PatientConfigUsecase.(getConfig(parameters : [String : Any]?) -> Promise<PatientConfig>).(closure #1).(closure #1) (PatientConfigUsecase.swift:82)
6  ABCApp                        0x10284133c partial apply for PatientConfigUsecase.(getConfig(parameters : [String : Any]?) -> Promise<PatientConfig>).(closure #1).(closure #1) (PatientConfigUsecase.swift)
7  ABCApp                        0x1028413a0 DataResponse<PatientConfig> (PatientConfigUsecase.swift)
8  Alamofire                      0x102dae8fc closure #1 in closure #1 in DataRequest.response<A>(queue:responseSerializer:completionHandler:) + 156
9  Alamofire                      0x102db2a8c partial apply for closure #1 in closure #1 in DownloadRequest.response<A>(queue:responseSerializer:completionHandler:) + 44
10 Alamofire                      0x102d8e4c0 _T0Ieg_IeyB_TR (NetworkReachabilityManager.swift)
11 libdispatch.dylib              0x200dff6c8 _dispatch_call_block_and_release + 24
12 libdispatch.dylib              0x200e00484 _dispatch_client_callout + 16
13 libdispatch.dylib              0x200ddfb44 _dispatch_main_queue_callback_4CF$VARIANT$armv81 + 1012
14 CoreFoundation                 0x2013561bc __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 12
15 CoreFoundation                 0x201351084 __CFRunLoopRun + 1964
16 CoreFoundation                 0x2013505b8 CFRunLoopRunSpecific + 436
17 GraphicsServices               0x2035c4584 GSEventRunModal + 100
18 UIKitCore                      0x22e348bc8 UIApplicationMain + 212
19 ABCApp                        0x1027a5c54 main (AppDelegate.swift:69)
20 libdyld.dylib                  0x200e10b94 start + 4

Do a 1.0.1 release with the new fixes

`testAddCollectionWhileCacheLoads` fails on travis-ci.org

Can't reproduce locally.

e.g. https://travis-ci.org/linkedin/RocketData/builds/147242144

testAddCollectionWhileCacheLoads, Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "waitForCache".
  /Users/travis/build/linkedin/RocketData/RocketDataTests/CollectionDataProviderTests/SharedCollectionDataProviderTests.swift:817

How to purge all items in the DataProvide and empty the cache in one shot?

When a user logs out, we need to purge but it doesn't seem that you guys support a complete purge...

Is this the only way?

// Purge let count = self?.usersDataProvider.count ?? 0 for index in stride(from: count, to: 0, by: -1) { self?.usersDataProvider.remove(at: index) }

Question about local persistence

I understand the primary function of RocketData is to cache and internally represent data that is available from another source, like a server.

I'm curious about storing metadata about this cached data that isn't destined for the original backend. For example, lets say I have a list of something, and I want to create (separately, stored only on the local device) a set of bookmarks pointing to those items.

Could I store my bookmarks in RocketData somehow and ensure they stick around? Or perhaps it would be better to use another storage solution to save a list of IDs with the metadata for each ID, and then ask RocketData for the item attached to that ID. Or maybe some other solution I'm not thinking of?

Support for 'macOS'

Just saw the video presentation, and I'm eager to use the framework. Curious what the delta is to supporting macOS, also?

Add CI (travis)

Upgrade Consistency Manager to 2.0.0

Shouldn't have any API changes for Rocket Data, but need to add a bit of functionality and testing.

Wrong version of ConsistencyManager installed when installing via cocoapods

Juts noticed something strange when installing the latest release (v4.0.1) via cocoapods. If you look in RocketData.podspec, you'll see that it specifies version 5.0.0 of ConsistencyManager as a dependency:

spec.dependency 'ConsistencyManager', '~> 5.0.0'

Whenever I run pod install to install RocketData, the terminal output always says Installing ConsistencyManager (4.0.0) instead of 5.0.0