openmined/swiftsyft

The official Syft worker for iOS, built in Swift

swiftsyft's Issues

Migrate SwiftSyft from Travis to Github Actions

We're switching from using Travis to Github actions (OpenMined/PySyft#3013). Please migrate this project to do so as well. It's important to also ensure that pull requests cannot be merged without an all-green CI report.

Implement Protobuf classes in iOS

We need to add the following Protobuf classes to SwiftSyft as they are completed:

• Plan
• State
• Operation
• Placeholder
• TorchParameter
• Protocol
• PromiseTensor
• ... more to be defined

Execute protocols in iOS

After plan execution has been finished in KotlinSyft (#33), we'll need to implement protocols, as per the Protocol refactor (OpenMined/PySyft#2903).

Implement class to check upload/download speed

Investigate how to implement a class that can get the upload/download speed using the app connection.

Make connection speed test optional

Reference this PyGrid issue: OpenMined/PyGrid-deprecated---see-PySyft-#557

Only run the speed test if the requires_speed_test field from the authentication response is true.

Do this: OpenMined/PyGrid-deprecated---see-PySyft-#557 (comment)

Implement sleep/wake detection in iOS

While we want to allow the end-user developer integrating SwiftSyft into their application the ability to choose to execute models while the user is asleep, we don't want to force them into this paradigm.

This issue includes developing a basic "asleep or awake" algorithm that a developer can choose to use. It's worth noting that the default option for a syft client would be to "enable" this as a requirement for training. It's up to the developer to state that they want this option "disabled".

Add documentation for model report http and web socket requests

Implement ping checker

Create a ping checker that returns the ping in milliseconds. This will be used in the FL cycle request as specified here: OpenMined/PyGrid-deprecated---see-PySyft-#445

Create new project to demonstrate using background task scheduler with SwiftSyft

iOS 13's BGTaskScheduler API's limitations (mainly having to register the background task in applicationDidFinishLaunchingOptions) have made it hard to integrate into the SwiftSyft library's API. Instead, a new project will be created to show an example of how to use it along with SwiftSyft library.

• Create new Example-background project and use it in the same workspace.
• Add BGTaskScheduler code to create a one-time background task to execute MNIST plan example
• Optionally add a repeating background task logic
• Update github flow yaml to reflect new project/workspace structure.

Add documentation for `SyftProtoExtensions`

File is located here: https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Classes/SyftProtoExtensions.swift

Set up deployment to Cocoapods

We need to at some point deploy SwiftSyft onto Cocoapods so that it may be used in libraries.

Here's the corresponding Android issue: OpenMined/KotlinSyft#3

Fix Travis build configuration

From the build errors for #6 found here

https://travis-ci.org/OpenMined/SwiftSyft/builds/640457469?utm_source=github_status&utm_medium=notification

Web Socket Connection Messages Model

As specified in #3 , we're going to need models to serialized/deserialized messages sent to and received from the grid server.

Here is the documentation specification for messages in grid.js

https://github.com/OpenMined/grid.js/blob/master/SOCKETS.md

• Get Protocol
• WebRTC Internal Message (Offer, Answer and Ice Candidate)
• Socket ping (URLSessionWebSocketTask has a ping method. Not sure if we need this)

Notes:

• Use Codable for serialization and deserialization. Eventually we, will use Protobuf, so we don't need unit tests for these for now.
• Protocol/Plan field from Get Protocol message is in serde (PySyft custom serialization). This is being ported to Protobuf, so we can skip it for now.
• In order to determine the type of the message, you're going to have to implement init(fromDecoder:) and encode(to encoder:) and select the correct model to serialize with. A reference on how to implement it is here: https://github.com/stasel/WebRTC-iOS/blob/master/WebRTC-Demo-App/Sources/Services/Message.swift

If this serialization method doesn't fit our use case or you run into some problems, we can discuss it here.

Add line chart graphing loss from training

Assert wss:// protocol for signaling client

Ensure that secure websocket connections to PyGrid are mandatory: wss:// being favored over ws://

SwiftSyft Connection Clients Roadmap

SwiftSyft establishes connection to a grid server (grid.js or pygrid) via a signalling server (websocket) and receives instructions (plans, protocols or tensor operations) sent by PySyft. The operations can be shared to other workers (mobile or web) via a WebRTC peer-to-peer connection.

In order to facilitate these functionality we're going to need three main components

Components

• iOS 12 Socket connection
• iOS 13 Socket connection
• Signalling client
• WebRTC client
• Syft client

Components Description

1. iOS 12 and 13 Socket Connection implementation

• Should conform to https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Protocols/SocketClientProtocol.swift
• Socket client connection should have a periodic keep alive/ping : Reference: https://github.com/OpenMined/grid.js/blob/master/SOCKETS.md
• Assert that socket protocol should use wss://
• iOS 13 implementation should use
• Should only be concerned with sending and receiving Data objects via wss connection.

2. Signalling client:

• Should connect via wss:// to a grid server (grid.js for now). Asser that the URL is ws:/
• Is initialized with a socket client implementation
• Serializes/Deserializes messages received or to be sent to/from the socket client.
• Be able to send and receive messages (Socket messages documentation can be found here: https://github.com/OpenMined/grid.js/blob/master/SOCKETS.md)
• Reference in syft.js workere is here (https://github.com/OpenMined/syft.js/blob/master/src/sockets.js)

3. WebRTC client

• Should establish and keep multi-peer connection and data channel from the connections it receives from the signalling client.
• Handles storing connections, data channels and ice candidates for each peer connection.
• Sends SDP offers/answers using signalling client.
• Should use GoogleWebRTC iOS library.
• Reference: https://github.com/OpenMined/syft.js/blob/master/src/webrtc.js

4. Syft client

• Handles coordination between signalling client and webrtc client.
• Stores business logic states (workerId and scopeId)
• Stores plans/protocols received from PySyft
• Reference: https://github.com/OpenMined/syft.js/blob/master/src/syft.js

5. Messages

• These are the messages sent among the grid server and mobile/web workers.
• For now, only socket messages (between workers and grid server) are defined
• Socket messages are serialized as JSON for now. PySyft protocols, on the other hand, are serialized in serde (custom serialization by Pysyft) by grid.js. We're not going to implement serde deserialization since eventually all the messages will be serialized in Protobuf.
• Reference: https://github.com/OpenMined/grid.js/blob/master/SOCKETS.md

Supporting features

• Data channel chunking
• UI to specify server and peer connection

1. Data Channel Chunking

• Messages greater than 16 KB sent via WebRTC will have to be sent in 16 KB chunks.
• Reference is here:
  https://webrtc.github.io/samples/src/content/datachannel/datatransfer/

2. UI

• Specifies socket server to connect to
• Initiates connection to grid server via button tap

Edit:

• Added requirement to enforce ws:// connection in signalling client.

RTCCVPixelBuffer is implemented in both Apple System Library and GoogleWebRTC Pod

Crash happens when connecting to a signalling server using GoogleWebRTC.

Issue was also found here:

stasel/WebRTC-iOS#19

https://bugs.chromium.org/p/webrtc/issues/detail?id=10560

Safe to say, Apple probably won't fix this on their end and we're going to have to recompile GoogleWebRTC from source and rename RTCCVPixelBuffer.

For those who can help building the library from source here's where you can get the source:

https://webrtc.org/native-code/ios/

Take note, it says the source code is 6GB, but when I downloaded it it's actually 12 GB :(

Edit:

• Crash doesn't happen all the time. Sometimes it's just a warning. I think iOS randomly chooses which class to use and may cause a crash depending on the device.

iOS 13 Socket Connection Implementation

As referenced in #3 , this is the socket connection implementation that will be used by the signalling client. It should conform to the SocketClientProtocol and be responsible for maintaining socket connection via ping, sending and receiving data.

For iOS 13, use URLSessionWebSocketTask.

Add documentation for using `SyftPlan`

WebRTC Client Implementation

• GoogleWebRTC integration
• PeerConnection Wrapper
• Initialize peer connection
• Peer connection events
• PeerConnection Events Hooks/Observers
• Signalling message processing logic
• Data channel observers
• Unit tests

Add support for charge detection and wifi detection in iOS

While we want to allow the end-user developer integrating SwiftSyft into their application the ability to choose to execute models while the user is charging their phone, we don't want to force them into this paradigm.

This issue includes developing charge detection that a developer can choose to use. It's worth noting that the default option for a syft client would be to "enable" this as a requirement for training. It's up to the developer to state that they want this option "disabled".

Update README.MD with proper tutorial on hosting and running MNIST plan

• Tutorial on how to run PyGrid using docker
• Tutorial on how to host MNIST plans using PySyft
• How to run iOS sample app
• Gif demonstrating loss

Add type mapping from Swift -> Objective-C -> C++ tensors

Add app icon

We need app icon (already created a set of all necessary images)

Add bandwidth and Internet connectivity test in iOS

We need to have some sort of way to run a basic bandwidth and Internet connectivity test in iOS so that we may submit these values to PyGrid. This allows PyGrid to properly select candidates for pooling based on internet connection speed. This does not check for wifi connectivity. This will be included in a separate issue.

We must determine the average ping, upload speed, and download speed of the device and report these values to PyGrid.

Add `PingChecker` documentation

Additional documentation for PingChecker.swift file:

• PingChecker class
• PingChecker properties

Add documentation for authenticate, cycle-request, get-plan and get-model HTTP requests

Found here: https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Classes/SyftClient.swift

You can ask here if you have any questions.

Establish WebRTC Connection using PyGrid

Test connecting to a PyGrid server implement any additional functions to make this possible.

Scaffold basic proposed worker API in iOS

The following is the proposed worker API in syft.js: https://github.com/OpenMined/Roadmap/blob/master/web_and_mobile_team/projects/federated_learning.md#4-execute

I propose that we have a nearly identical API for iOS.

Allow for training state to be persisted to temporary storage in the event of a failure in iOS

In the event that training is interrupted on a device, we need to have a "saved state" of the training process that allows us to resume. Such interruptions could be:

• An error in the training
• User picks up their phone and begins using it
• Wifi connection drops
• Phone is unplugged

Execute plans in iOS

This epic issue is somewhat self-explanatory, but in theory, we need to be able to execute a PySyft plan. This should ideally only be done after the API has been finalized in the iOS worker (#28).

Add documentation for MNIST Loader

While the MNIST Loader is not part of the SwiftSyft library, it does demonstrate how to load data and transform it to be able to be consumed by our torchscript models.

Feel free to ask me if you have any questions about it. The original code for the loader came from this repository: https://github.com/simonlee2/MNISTKit

Release initial version of SwiftSyft in CocoaPods

Create serializable/deserializable web socket messages for FL endpoints

In line with the intial request formats specified in PyGrid here: OpenMined/PyGrid-deprecated---see-PySyft-#445 , we need a new set of messages in that can be sent over websockets.

• Authentication with PyGrid
• FL Worker Cycle Request
• Training Plan Download
• Protocol Download
• Model Download
• Report

Add a stopping method that stops the training process in iOS

We need to have a stopping method that will terminate the current job in question. Reasons for stopping training could be any of the following:

• The user wanted to... like they clicked a "stop" button
• The plan has an error and can't execute
• The user started using their device again
• The device loses wifi
• The device loses active charging
• Or perhaps most importantly... if the model isn't really going anywhere (the error rate isn't going down)

At this point, we should stop and notify the user with some sort of message.

Use corrected Google WebRTC binary to prevent class name conflicts in iOS 13

As previously discussed, GoogleWebRTC binary has naming conflicts with some of iOS's classes that may cause a crash in some devices. Reference: stasel/WebRTC-iOS#19

Also in the same issue someone has
a.) Created step by step guide on how to rename the classes and rebuild the library from scratch
stasel/WebRTC-iOS#19 (comment)
and
b.) Provided a pre-built binary.
stasel/WebRTC-iOS#19 (comment)

We should test the pre-built binary if it doesn't cause anymore warnings or crashes. But for production, we need to follow the steps above in building the library from scratch for security purposes.

Background Execution Example Documentation

There's currently a separate example app to demonstrate how to use SwiftSyft job using iOS 13's new Background Task Scheduler. We need some form of documentation (Background-Execution.md) explaining the rationale behind using it outside of the library and how BGTaskScheduler is used with SwiftSyft

The file with the example is here

I think some main points that should be included.

• Why we chose to not integrate BGTaskScheduler into SwiftSyft itself?
  Answer: Because registering background task handlers can only be done in applicationDidFinishLaunchingOptions (source)

• Set-up of the task scheduler itself from the app side: https://developer.apple.com/documentation/backgroundtasks/bgtaskscheduler

• Why it is necessary to require external power and network connectivity for background task?
  Answer: To remove CPU limit restrictions on background tasks (external power) and connect to PyGrid and other workers (network connectivity).
  Relevant line: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L37

• How to handle background task expiration
  Answer:
  Expiration handler: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L98
  Expiration checker during training: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L63

• Situations to set the background task as completed:
  Finished Training: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L84
  Training Error: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L88
  Syft Job Error: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L94

• How to simulate a background task in the debugger:
  https://developer.apple.com/documentation/backgroundtasks/starting_and_terminating_tasks_during_development

• Link to the background example and the background example documentation from README.md

Add `onRejected` and `onError` handlers

We need to add 2 additional event handlers for the federated learning cycle as specified here: https://github.com/OpenMined/Roadmap/pull/13

1. onRejected - handling the rejection of an FL cycle request

2. onAccepted - handling the error in the plan or protocol execution.

Can't build app for a real device, have error "does not contain bitcode"

SwiftSyft/Example/Pods/LibTorch/install/lib/libtorch.a(CPUGenerator.cpp.o)’ does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. file ‘/Pods/LibTorch/install/lib/libtorch.a’ for architecture arm64

iOS 12 Socket Connection Implementation

As referenced in #3 , this is the socket connection implementation that will be used by the signalling client. It should conform to the SocketClientProtocol and be responsible for maintaining socket connection via ping, sending and receiving data.

For iOS 12, you can research for a updated socket connection library that supports all of our requirements specified in #3 . Please post it here so we can discuss the pros and cons of your chosen library.

For reference, in the WebRTC iOS demo here: https://github.com/stasel/WebRTC-iOS , uses https://github.com/daltoniam/Starscream

Implement reachability publisher

This will listen to reachability changes (internet connectivity). Users of this class can check changes by subscribing to a publisher (check Combine framework) exposed by the Reachability class.

iOS 12 should have a new class that makes this easier.

Add documentation for generating and serializing diffs

Code for generating diffs is in TorchTrainingModule. Serializing diffs is in SyftProtoExtensions

UI for initiating connection to grid server

We need a UI in the example app similar to that found in syft.js

You can run the example syft.js site by following the instructions in https://github.com/OpenMined/syft.js and https://github.com/OpenMined/grid.js/ .

Easiest way to do it would be to use docker.
Grid

• cd into docker director in grid.js and enter docker-compose -f example-seed.yml up in the commandline

Syft.js

• Follow instructions in "Running Demos" section of syft.js README.md (https://github.com/OpenMined/syft.js)

This can be iPhone only. We don't need to support iPad since it's just an example. You're free to use
the storyboard for this.

UI Components:

• OpenMined Logo
• description
• Text box for socket server URL
• Text box for protocol ID (this can be hardcoded value)
• Button to connect to grid
• Text box to write message to other participants (hidden when disconnected, visible when connected)
• Button to send message to other participants. (hidden when disconnected, visible when connected)

Add support for background task scheduling in iOS

In order to properly execute training plans, we must do so in a background task. This allows for training to take place without a visual API (as a library of another app), and do so separate from the main thread.

Implement FL Authentication and Cycle Socket Requests

Similar to #47, but implemented using socket messages. This may require a refactor of signalling client to be able to easily chain both requests.

• Refactor SignallingClient to use Combine framework for observing messages. This makes it easier to chain requests
• Separate SignallingMessages model to separate SignallingMessagesRequest and SignallingMessageResponse. Previous version assumed socket request and response formats were the same.
• Add auth request and response models
• Add cycle request and response models
• Add socket auth request and response handler in syft client
• Add socket cycle request and response handler in syft client

Create the basic structure of SwiftSyft

Scaffold the library!

High memory use while training

Memory use of the example app using MNIST data increases incrementally until it reaches OS limits for the app and crashes on a real device.

Doesn't happen on the simulator due to higher memory limit.

Implement FL Authentication and Cycle API Request

This will be implemented in SyftClient class start method here : https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Classes/SyftClient.swift#L28

The API requests are described here:
OpenMined/PyGrid-deprecated---see-PySyft-#445
Refer to Authentication with PyGrid and FL Worker Cycle Request

The requests can be made by socket messages or HTTP requests but this issue is mainly for the HTTP requests made using URLSession.dataTaskPublisher and chaining the requests.

A seperate issue will be made for using signalling requests.

Integrate Signalling Client with iOS 12 and iOS 13 socket implementation

Wait for #16 and #15 implementation. This should serialize/deserialize the signalling messages as well.

Implement split and stitch for large data channel messages in iOS

This was already done in syft.js: OpenMined/syft.js#78

This will also need to be done for iOS. You should read these two classes so that you fully understand the problem and theoretical solution:

• https://github.com/OpenMined/syft.js/blob/master/src/data_channel_message.js
• https://github.com/OpenMined/syft.js/blob/master/src/data_channel_message_queue.js