dreymonde/shallows

🛶 Your lightweight persistence toolbox

shallows's Issues

Memory cache returns nothing

I have a strange case where a Storage.retrieve completely skips its completion handler. The cache is composed of a MemoryStorage and a DiskStorage. I wish I could say more about it, but there's not much to say. I can see the Key go in, but the completion gets skipped entirely... ??

[EDIT: This wouldn't necessarily be a huge problem, except that I have a DispatchGroup.leave() waiting on the completion...]

Issue with combined cache creation and DiskFolderStorage

let companySettingsStorage = DiskStorage.main.folder("companysettings.cache", in: .documentDirectory).mapJSONObject(AllCompanySettings.self) let companyCache = MemoryStorage<String, AllCompanySettings>()
let combinedCache = companyCache.combined(with: companySettingsStorage)

I am getting the following error on the combined cache creation. My assumption is the compiler isn't picking up the companySettingsStorage as conforming to StorageProtocol

Generic parameter 'StorageType' could not be inferred

Please advise on the proper creation of a combined cache.

Cocoapods?

Very nice library you have here. I see the README says one installs through Carthage, but I also see a Shallows.podspec implying (possible eventual) Cocoapods support. Is that planned, or should the .podspec be removed?

Congrats on the iOS Dev Weekly inclusion too.

Archiving fails in Swift5

I'm getting a very frustrating build error of Shallows / my adoption of Shallows that only occurs when archiving my application. I can run it just fine in debug mode with Swift 5 and it used to work fine to archive it when I was using Swift 4. But now I get a very weird build error that I don't understand. @dreymonde have you or anyone else using Shallows seen this problem or know what the issue is caused by?
`CompileSwift normal armv7
.........
:0: error: fatal error encountered while reading from module 'Shallows'; please file a bug report with your project and the crash log

*** DESERIALIZATION FAILURE (please include this section in any bug report) ***
(see "While..." info below)
Stack dump:
0. Program arguments: ..........

1. While running pass #1 SILModuleTransform "PerformanceSILLinker".
2. While deserializing SIL function "$s8Shallows15StorageProtocolPAAE02asB0AA0B0Vy3KeyQz5ValueQzGyF"
   0 swift 0x000000010705eee3 PrintStackTraceSignalHandler(void*) + 51
   1 swift 0x000000010705e6bc SignalHandler(int) + 348
   2 libsystem_platform.dylib 0x00007fff68619b5d _sigtramp + 29
   3 libsystem_platform.dylib 000000000000000000 _sigtramp + 2543740096
   4 libsystem_c.dylib 0x00007fff684d36a6 abort + 127
   5 swift 0x00000001041de1e0 swift::ModuleFile::fatal(llvm::Error) + 1856
   6 swift 0x0000000104204560 swift::SILDeserializer::readSILFunctionChecked(llvm::PointerEmbeddedInt<unsigned int, 31>, swift::SILFunction*, llvm::StringRef, bool, bool) + 9328
   7 swift 0x00000001042158e1 swift::SILDeserializer::lookupSILFunction(swift::SILFunction*) + 673
   8 swift 0x0000000103ecc69b swift::SILLinkerVisitor::maybeAddFunctionToWorklist(swift::SILFunction*) + 107
   9 swift 0x0000000103ecc984 swift::SILLinkerVisitor::processFunction(swift::SILFunction*) + 292
   10 swift 0x0000000103dabfe7 (anonymous namespace)::SILLinker::run() + 263
   11 swift 0x0000000103cb63e8 swift::SILPassManager::execute() + 7416
   12 swift 0x0000000102ec3668 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 50280
   13 swift 0x0000000102eb392e swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 6862
   14 swift 0x0000000102e519ce main + 1246
   15 libdyld.dylib 0x00007fff6842e3d5 start + 1
   16 libdyld.dylib 0x0000000000000069 start + 2545753237`

update(forKey: ) is not thread safe

Seems that the update method is not thread safe, if using the storage on a concurrent DispatchQueue. I'm not sure if it is good to have mutable models stored anyway, but I happened to run into this ¯\_(ツ)_/¯

Workaround is to do the updates in own serial queue. Could not fix this on my own class conforming to StorageProtocol since the update-method is a protocol extension and using asStorage() always directs the linker to that implementation.

Small test to illustrate the issue:

import Dispatch
import Shallows

let storage = MemoryStorage<String, [Int]>().asStorage().defaulting(to: [])
let syncStorage = storage.makeSyncStorage()
let queue = DispatchQueue(label: "1", attributes: [.concurrent])

try? syncStorage.set([0], forKey: "Test")

queue.async {
    storage.update(forKey: "Test", { (value) in
        usleep(500)
        value.append(1)
    })
}
queue.async {
    storage.update(forKey: "Test", { (value) in
        value.append(2)
    })
}

print("Wait...")
usleep(50000)

storage.retrieve(forKey: "Test", completion: { (value) in
    print(value) // Prints success([0, 1]), but should print success([0, 1, 2])
})

Base64 encoding isn't great for filenames

"/" is a valid base64 character and invalid for filenames. I recommend you use Base64URL encoding. This is probably just adding the following to lines:

      .replacingOccurrences(of: "+", with: "-")
      .replacingOccurrences(of: "/", with: "_")

Purge from memoryCache (while leaving in composed diskCache)?

Just curious whether there's a simple way (that I haven't seen) to remove an item from a memoryCache, while leaving it in the diskCache (I have a disk+memory composition). I want to be able to free up memory (on iOS), but retain the option to quickly reload from diskCache.

Thanks in advance.

Clearer Error handling

Would it be possible to define a common Error enum for the StorageProtocol? Seems that now the errors are simply Error protocol, so making switch cases to determine the real cause is difficult.

For example

enum StorageError {
    case general(error: Error)
    case write(error: Error)
    case noValue(for: Key)
}

Cache Expiration

Hi, very nice library. It's API has given me a lot of ideas about how to approach composition in my own app.

I've recently integrated with an ObjC caching library (SDWebImage). It supports cache expiration, and I was wondering if you planned on providing such functionality in this library?

It seems like some of this would be supported by making use of combined storages and possibly the "zipped" functionality. So you could have one storage contain keys and "inserted on" dates while another storage contains keys and the values you care about (for example, images). Cleaning up of expired keys could be handled on an as-needed basis by iterating through all the keys and comparing their "inserted on" dates with some expiry date.

Where I'm unsure, is how you'd implement something like a "Least Recently Used" strategy - wherein accessing a key updates it's "inserted on" date (you'd probably rename it to "updated on").

Imagine the following:

LRU Image Storage = Date Storage zipped with (In Memory Image Storage+Disk Image Storage)

If I read from this LRU Image Storage, I would want it to get from the In Memory storage first followed by the Disk storage if needed (that would be using the default pull strategy for combined storages). But how would I go about updating the entry in the Date Storage as the read occurred? I suppose a brute-force way to do so would be to just re-insert the image that was returned during the read, but that seems non-ideal from a usability/performance stand-point.

Xcode 11.4.1, Swift 5.2.2 Abort trap: 6 points to Shallows

I'm hitting an Abort trap: 6 while trying to build my project and Shallows appears to be involved:

1.	Apple Swift version 5.2.2 (swiftlang-1103.0.32.6 clang-1103.0.32.51)
2.	While running pass #340 SILModuleTransform "MandatorySILLinker".
3.	While deserializing SIL function "$s8Shallows15StorageProtocolPAAE02asB0AA0B0Vy3KeyQz5ValueQzGyF"
0  swift                    0x00000001109e24ea PrintStackTraceSignalHandler(void*) + 42
1  swift                    0x00000001109e1cc0 SignalHandler(int) + 352
2  libsystem_platform.dylib 0x00007fff6f90842d _sigtramp + 29
3  libsystem_platform.dylib 0x00007ffbf001d1b0 _sigtramp + 2154909088
4  libsystem_c.dylib        0x00007fff6f7dda1c abort + 120
5  swift                    0x000000010d412470 std::__1::__vector_base<swift::Diagnostic, std::__1::allocator<swift::Diagnostic> >::clear() (.16933) + 0
6  swift                    0x000000010d420e40 swift::DeclDeserializer::deserializeAnyFunc(llvm::ArrayRef<unsigned long long>, llvm::StringRef, bool) + 0
7  swift                    0x000000010d43ed82 swift::SILDeserializer::readSILFunctionChecked(llvm::PointerEmbeddedInt<unsigned int, 31>, swift::SILFunction*, llvm::StringRef, bool, bool) + 11202
8  swift                    0x000000010d45f277 swift::SILDeserializer::lookupSILFunction(swift::SILFunction*, bool) + 679
9  swift                    0x000000010d0799cd swift::SILLinkerVisitor::maybeAddFunctionToWorklist(swift::SILFunction*) + 253
10 swift                    0x000000010d079c54 swift::SILLinkerVisitor::processFunction(swift::SILFunction*) + 244
11 swift                    0x000000010cf4cb10 (anonymous namespace)::SILLinker::run() + 304
12 swift                    0x000000010ce3e0d9 swift::SILPassManager::execute() + 2329
13 swift                    0x000000010ce43408 swift::runSILDiagnosticPasses(swift::SILModule&) + 2264
14 swift                    0x000000010ca40a02 swift::CompilerInstance::performSILProcessing(swift::SILModule*, swift::UnifiedStatsReporter*) + 66
15 swift                    0x000000010c73e450 performCompileStepsPostSILGen(swift::CompilerInstance&, swift::CompilerInvocation&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, bool, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, bool, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 1712
16 swift                    0x000000010c73429a swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 48426
17 swift                    0x000000010c6abf53 main + 1283
18 libdyld.dylib            0x00007fff6f70f7fd start + 1
error: Abort trap: 6 (in target 'SpliqsTool' from project 'SpliqsTool')

Not sure whether this is Apple's issue or yours, but I thought I'd point it out.
The Shallows project, on its own, builds fine, so I'm not sure why linking to it in my project is an issue... Xcode... grrr...

Clear object from MemoryStorage?

I've been debugging some memory usage issues and just noticed/remembered that Shallows is keeping my objects in a MemoryStorage, pretty much permanently. Is there an easy way to clear objects from MemoryStorage, or clear the MemoryStorage completely?

No mechanism for deleting from disk storage

I really like this very thin, simple storage layer, also props for dropping support for legacy package management systems!

I may be missing something, but is there no way to remove a single item from disk storage? Something like storage.delete(Key)?

Update: It looks like there is DiskStorage.fileURL(for:) but that requires me to manually create the filename again (if I've mapped it to a different type)

3-layer composition?

I'm trying to build a 3-layer, memory+disk+server (Firebase), composition, but I'm getting a "Type of expression is ambiguous without more context" error on the first line.

    let myObjectMemoryCache = MemoryStorage<Filename, MyObject>()   // <-- ambiguous type error
        .combined(with: DiskStorage.main.folder("myObjects", in: .documentDirectory).mapJSONObject(MyObject.self))
        .combined(with: RemoteStorage.main.collection("myObjects").mapJSONObject(MyObject.self))

I'll admit, I'm really not sure how I'm supposed to do this. It would be great to have a little more info on how these kinds of custom compositions should be implemented. (I should mention that I do have a running memory+disk version that works fine, based on the examples you provide—I'm just not sure how to add another layer.)

How to handle success in StorageProtocol set(value:forKey:completion:)

Stupid question, but I'm finally getting around to updating all my Shallows-related code to support the latest SwiftPM version, and I'm not sure how to handle a successful result in set(value:forKey:completion:). All I really want is to be able to fire some function in the caller on success, but I'm not clear on using ShallowsResult<Void> in this context...
To clarify: my confusion is about implementing it in my custom Storage.

Memory limitations and responding to warnings with combined cache?

For the combined example of Memory cache with Disk Storage cache, I assume the memory cache wouldn't be cleared automatically if didReceiveMemoryWarning? If not, I can't see a way to manually clear the memory cache. Any suggestions? or have i missed something obvious, thanks.