linkedin/cassette

An efficient, file-based FIFO Queue for iOS and macOS.

Cassette

Cassette is a collection of queue-related classes for iOS and macOS. It is maintained by LinkedIn. Cassette was originally implemented by Segment. Cassette was inspired by Tape.

QueueFile is an efficient, file-based FIFO queue. Addition and removal from an instance is an O(1) operation. Writes are synchronous; data will be written to disk before an operation returns. The queue is intended to be reliable and survive system or process crashes.

Installing the Library

CocoaPods

target 'MyApp' do
  pod 'Cassette', '1.0.0-beta3'
end

Manual

Download the latest binary of the library.

Usage

CASObjectQueue works with arbitrary objects that abide by the NSCoding protocol. An CASObjectQueue may be backed by a persistent CASQueueFile, or in memory.

CASObjectQueue<NSNumber *> *queue;

// Persistent ObjectQueue
NSError *error;
queue = [[CASFileObjectQueue alloc] initWithRelativePath:@"Test-File" error:&error];

// In-Memory ObjectQueue
queue = [[CASInMemoryObjectQueue alloc] init];

Add some data to the end of the queue.

NSError *error;
if ([queue add:@1 error:&error]) {
  // Success
} else {
  NSLog(@"Error: %@", error);
}

Read data at the head of the queue.

// Peek the eldest element. Note that -peek:error: on an empty queue
// returns @[], but on error (e.g., I/O error) it returns nil.
NSNumber *data = [queue peek:1 error:&error].firstObject;

// Peek the eldest `n` elements.
NSArray<NSNumber *> *data = [queue peek:n error:&error];

Remove processed elements.

// Remove the eldest element.
if ([queue pop:1 error:&error]) { ... }

// Remove 'n' elements.
if ([queue pop:n error:&error]) { ... }

// Remove all elements.
if ([queue clearAndReturnError:&error]) { ... }

cassette's Issues

NSFileHandle iOS 13 Deprecations

e.g. writeData is deprecated. https://developer.apple.com/documentation/foundation/nsfilehandle/1410936-writedata?language=objc

+unarchiveObjectWithData: deprecated in iOS12 & macosx10.14

NSUnarhiver unarchiveObjectWithData: is deprecated for
macosx-10.14 and ios 12.0

+ (nullable id)unarchiveObjectWithData:(NSData *)data API_DEPRECATED("Use +unarchivedObjectOfClass:fromData:error: instead", macosx(10.2,10.14), ios(2.0,12.0), watchos(2.0,5.0), tvos(9.0,12.0));

This is way only way I have found that works to do the same thing without the deprecated methods:

NSKeyedUnarchiver *unarchiver = [[NSKeyedUnarchiver alloc] initForReadingFromData:data error:nil];
[unarchiver setRequiresSecureCoding:NO];
CustomObject *object = [unarchiver decodeObjectForKey:NSKeyedArchiveRootObjectKey];

Use OSWriteXXX for possible performance improvement

Atomic Writes

Instead of writing to the main file, we should create a backup, write to it and then copy it over.

Consider removing the "commitFile" logic from CASQueueFile

CASQueueFile writes its header to a a temporary commitFile with comments suggesting it should try and recover from failure when updating the metadata header.

This logic does not exist in the Java implementation of tape.

Worse yet, the Objective-C implementation does not actually recover from failure; it silently discards the queue file entirely and pretends like the queue was empty.

Probably this logic should just be deleted. Either we assume the OS writes the header atomically (in which case we don't need the temporary commitFile), or we need to write the entire queue (header and data) to a temporary file, then rename the temporary file over the real file.

Durability Issue

In the following circumstances:

• when expanding the database file during an add, and
• when an existing entry is wrapped around the end of the database file (since this is a ring buffer)

We erase the old wrapped tail, prior to updating the database header.
https://github.com/linkedin/cassette/blob/master/Cassette/CASQueueFile.m#L453

        [self ringEraseAtPosition:QueueFileHeaderLength length:count];

Although the window is small, if a crash occurs during this period it will result in corrupted data.

---

This solution should be simple, removing the line in question is most likely all we need

4.26gb database file limit b/c of `unsigned int` limitation

the size of the database file maxes out at 4.26gb or roughtly 4,294,967,295 bytes because of storing _fileLength as unsigned int

Use Library directory instead of Documents directory in CASFileObjectQueue

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html

Signal error if user provides an existing directory instead of a file

[fileManager fileExistsAtPath:path] is returning true when a user provides a directory (in CASQueueFile.m). The assumption made by the code that path is a file. We should signal an error instead.

Add appropriate file header comments to core files

remove `CASFileObjectQueue` `relativePath` option.

If we want the library to not be involved with folder management, file management, and want to require users to create intermediary directories themselves, having the relativePath option no longer makes sense.