openmined / swiftsyft Goto Github PK
View Code? Open in Web Editor NEWThe official Syft worker for iOS, built in Swift
License: Apache License 2.0
The official Syft worker for iOS, built in Swift
License: Apache License 2.0
In line with the intial request formats specified in PyGrid here: OpenMined/PyGrid-deprecated---see-PySyft-#445 , we need a new set of messages in that can be sent over websockets.
As previously discussed, GoogleWebRTC binary has naming conflicts with some of iOS's classes that may cause a crash in some devices. Reference: stasel/WebRTC-iOS#19
Also in the same issue someone has
a.) Created step by step guide on how to rename the classes and rebuild the library from scratch
stasel/WebRTC-iOS#19 (comment)
and
b.) Provided a pre-built binary.
stasel/WebRTC-iOS#19 (comment)
We should test the pre-built binary if it doesn't cause anymore warnings or crashes. But for production, we need to follow the steps above in building the library from scratch for security purposes.
This will listen to reachability changes (internet connectivity). Users of this class can check changes by subscribing to a publisher (check Combine
framework) exposed by the Reachability class.
iOS 12 should have a new class that makes this easier.
Code for generating diffs is in TorchTrainingModule
. Serializing diffs is in SyftProtoExtensions
As referenced in #3 , this is the socket connection implementation that will be used by the signalling client. It should conform to the SocketClientProtocol
and be responsible for maintaining socket connection via ping, sending and receiving data.
For iOS 13, use URLSessionWebSocketTask
.
From the build errors for #6 found here
Memory use of the example app using MNIST data increases incrementally until it reaches OS limits for the app and crashes on a real device.
Doesn't happen on the simulator due to higher memory limit.
Investigate how to implement a class that can get the upload/download speed using the app connection.
Scaffold the library!
As referenced in #3 , this is the socket connection implementation that will be used by the signalling client. It should conform to the SocketClientProtocol
and be responsible for maintaining socket connection via ping, sending and receiving data.
For iOS 12, you can research for a updated socket connection library that supports all of our requirements specified in #3 . Please post it here so we can discuss the pros and cons of your chosen library.
For reference, in the WebRTC iOS demo here: https://github.com/stasel/WebRTC-iOS , uses https://github.com/daltoniam/Starscream
This epic issue is somewhat self-explanatory, but in theory, we need to be able to execute a PySyft plan. This should ideally only be done after the API has been finalized in the iOS worker (#28).
Found here: https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Classes/SyftClient.swift
You can ask here if you have any questions.
Additional documentation for PingChecker.swift
file:
PingChecker
classPingChecker
propertiesCrash happens when connecting to a signalling server using GoogleWebRTC
.
Issue was also found here:
https://bugs.chromium.org/p/webrtc/issues/detail?id=10560
Safe to say, Apple probably won't fix this on their end and we're going to have to recompile GoogleWebRTC from source and rename RTCCVPixelBuffer
.
For those who can help building the library from source here's where you can get the source:
https://webrtc.org/native-code/ios/
Take note, it says the source code is 6GB, but when I downloaded it it's actually 12 GB :(
Edit:
SwiftSyft/Example/Pods/LibTorch/install/lib/libtorch.a(CPUGenerator.cpp.o)’ does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. file ‘/Pods/LibTorch/install/lib/libtorch.a’ for architecture arm64
In the event that training is interrupted on a device, we need to have a "saved state" of the training process that allows us to resume. Such interruptions could be:
We need to add the following Protobuf classes to SwiftSyft as they are completed:
Plan
State
Operation
Placeholder
TorchParameter
Protocol
PromiseTensor
We need to have some sort of way to run a basic bandwidth and Internet connectivity test in iOS so that we may submit these values to PyGrid. This allows PyGrid to properly select candidates for pooling based on internet connection speed. This does not check for wifi connectivity. This will be included in a separate issue.
We must determine the average ping, upload speed, and download speed of the device and report these values to PyGrid.
We need to have a stopping method that will terminate the current job in question. Reasons for stopping training could be any of the following:
At this point, we should stop and notify the user with some sort of message.
We need app icon (already created a set of all necessary images)
We're switching from using Travis to Github actions (OpenMined/PySyft#3013). Please migrate this project to do so as well. It's important to also ensure that pull requests cannot be merged without an all-green CI report.
In order to properly execute training plans, we must do so in a background task. This allows for training to take place without a visual API (as a library of another app), and do so separate from the main thread.
There's currently a separate example app to demonstrate how to use SwiftSyft
job using iOS 13's new Background Task Scheduler. We need some form of documentation (Background-Execution.md
) explaining the rationale behind using it outside of the library and how BGTaskScheduler
is used with SwiftSyft
The file with the example is here
I think some main points that should be included.
Why we chose to not integrate BGTaskScheduler
into SwiftSyft itself?
Answer: Because registering background task handlers can only be done in applicationDidFinishLaunchingOptions
(source)
Set-up of the task scheduler itself from the app side: https://developer.apple.com/documentation/backgroundtasks/bgtaskscheduler
Why it is necessary to require external power and network connectivity for background task?
Answer: To remove CPU limit restrictions on background tasks (external power) and connect to PyGrid and other workers (network connectivity).
Relevant line: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L37
How to handle background task expiration
Answer:
Expiration handler: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L98
Expiration checker during training: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L63
Situations to set the background task as completed:
Finished Training: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L84
Training Error: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L88
Syft Job Error: https://github.com/OpenMined/SwiftSyft/blob/master/Example-Background/SwiftSyft-Background/AppDelegate.swift#L94
How to simulate a background task in the debugger:
https://developer.apple.com/documentation/backgroundtasks/starting_and_terminating_tasks_during_development
Link to the background example and the background example documentation from README.md
After plan execution has been finished in KotlinSyft (#33), we'll need to implement protocols, as per the Protocol refactor (OpenMined/PySyft#2903).
This will be implemented in SyftClient
class start
method here : https://github.com/OpenMined/SwiftSyft/blob/master/SwiftSyft/Classes/SyftClient.swift#L28
The API requests are described here:
OpenMined/PyGrid-deprecated---see-PySyft-#445
Refer to Authentication with PyGrid and FL Worker Cycle Request
The requests can be made by socket messages or HTTP requests but this issue is mainly for the HTTP requests made using URLSession.dataTaskPublisher
and chaining the requests.
A seperate issue will be made for using signalling requests.
We need to at some point deploy SwiftSyft onto Cocoapods so that it may be used in libraries.
Here's the corresponding Android issue: OpenMined/KotlinSyft#3
While we want to allow the end-user developer integrating SwiftSyft into their application the ability to choose to execute models while the user is charging their phone, we don't want to force them into this paradigm.
This issue includes developing charge detection that a developer can choose to use. It's worth noting that the default option for a syft client would be to "enable" this as a requirement for training. It's up to the developer to state that they want this option "disabled".
This was already done in syft.js: OpenMined/syft.js#78
This will also need to be done for iOS. You should read these two classes so that you fully understand the problem and theoretical solution:
SwiftSyft establishes connection to a grid server (grid.js or pygrid) via a signalling server (websocket) and receives instructions (plans, protocols or tensor operations) sent by PySyft. The operations can be shared to other workers (mobile or web) via a WebRTC peer-to-peer connection.
In order to facilitate these functionality we're going to need three main components
Components
Components Description
wss://
Data
objects via wss
connection.wss://
to a grid server (grid.js for now). Asser that the URL is ws:/
GoogleWebRTC
iOS library.Supporting features
Edit:
ws://
connection in signalling client.We need a UI in the example app similar to that found in syft.js
You can run the example syft.js site by following the instructions in https://github.com/OpenMined/syft.js and https://github.com/OpenMined/grid.js/ .
Easiest way to do it would be to use docker.
Grid
cd
into docker
director in grid.js and enter docker-compose -f example-seed.yml up
in the commandlineSyft.js
This can be iPhone only. We don't need to support iPad since it's just an example. You're free to use
the storyboard for this.
UI Components:
While we want to allow the end-user developer integrating SwiftSyft into their application the ability to choose to execute models while the user is asleep, we don't want to force them into this paradigm.
This issue includes developing a basic "asleep or awake" algorithm that a developer can choose to use. It's worth noting that the default option for a syft client would be to "enable" this as a requirement for training. It's up to the developer to state that they want this option "disabled".
Ensure that secure websocket connections to PyGrid are mandatory: wss://
being favored over ws://
Create a ping checker that returns the ping in milliseconds. This will be used in the FL cycle request as specified here: OpenMined/PyGrid-deprecated---see-PySyft-#445
While the MNIST Loader is not part of the SwiftSyft library, it does demonstrate how to load data and transform it to be able to be consumed by our torchscript models.
Feel free to ask me if you have any questions about it. The original code for the loader came from this repository: https://github.com/simonlee2/MNISTKit
We need to add 2 additional event handlers for the federated learning cycle as specified here: https://github.com/OpenMined/Roadmap/pull/13
onRejected
- handling the rejection of an FL cycle request
onAccepted
- handling the error in the plan or protocol execution.
Test connecting to a PyGrid server implement any additional functions to make this possible.
Similar to #47, but implemented using socket messages. This may require a refactor of signalling client to be able to easily chain both requests.
SignallingClient
to use Combine
framework for observing messages. This makes it easier to chain requestsSignallingMessages
model to separate SignallingMessagesRequest
and SignallingMessageResponse
. Previous version assumed socket request and response formats were the same.As specified in #3 , we're going to need models to serialized/deserialized messages sent to and received from the grid server.
Here is the documentation specification for messages in grid.js
https://github.com/OpenMined/grid.js/blob/master/SOCKETS.md
Notes:
Get Protocol
message is in serde (PySyft custom serialization). This is being ported to Protobuf, so we can skip it for now.init(fromDecoder:)
and encode(to encoder:)
and select the correct model to serialize with. A reference on how to implement it is here: https://github.com/stasel/WebRTC-iOS/blob/master/WebRTC-Demo-App/Sources/Services/Message.swiftIf this serialization method doesn't fit our use case or you run into some problems, we can discuss it here.
The following is the proposed worker API in syft.js: https://github.com/OpenMined/Roadmap/blob/master/web_and_mobile_team/projects/federated_learning.md#4-execute
I propose that we have a nearly identical API for iOS.
iOS 13's BGTaskScheduler API's limitations (mainly having to register the background task in applicationDidFinishLaunchingOptions
) have made it hard to integrate into the SwiftSyft library's API. Instead, a new project will be created to show an example of how to use it along with SwiftSyft library.
Example-background
project and use it in the same workspace.Reference this PyGrid issue: OpenMined/PyGrid-deprecated---see-PySyft-#557
Only run the speed test if the requires_speed_test
field from the authentication response is true
.
Do this: OpenMined/PyGrid-deprecated---see-PySyft-#557 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.