A project developing privacy-preserving,
vertical federated learning,
using syft.
- 🔗 Private entity resolution using Private Set Intersection (PSI)
- 🔒 Trains a model on vertically partitioned data using SplitNNs, so only data holders can access data
Vertically-partitioned data is data
in which
fields relating to a single record
are distributed across multiple datasets.
For example,
multiple hospitals may have admissions data on the same patients,
or retailers have transaction data on the same shoppers.
Vertically-partitioned data could be applied to solve vital problems,
but data holders can't combine their datasets
by simply comparing notes with other data holders
unless they want to break user privacy.
PyVertical uses PSI
to link datasets in a privacy-preserving way.
We train SplitNNs on the partitioned data
to ensure the data remains separate throughout the entire process.
See the changelog
for information
on the current status of PyVertical.
NOTE: PyVertical does not currently work with syft 0.3.0
PyVertical process:
- Create partitioned dataset
- Simulate real-world partitioned dataset by splitting MNIST into a dataset of images and a dataset of labels
- Give each data point (image + label) a unique ID
- Randomly shuffle each dataset
- Randomly remove some elements from each dataset
- Link datasets using PSI
- Use PSI to link indices in each dataset using unique IDs
- Reorder datasets using linked indices
- Train a split neural network
- Hold both datasets in a dataloader
- Send images to first part of split network
- Send labels to second part of split network
- Train the network
| Windows | Linux | MacOS |
|---|---|---|
| ❌ | ✔️ | ✔️ |
The Windows PyTorch version 1.4.0 is experiencing issues.
It cannot be updated on a working version, until Syft will be updated, too.
3.6 |
3.7 |
3.8 |
3.9 |
|---|---|---|---|
| ✔️ | ✔️ | ✔️ | ❌ |
syft and PSI
upstream dependencies do not have Python 3.9
packages.
To install the dependencies, we recommend using Conda:
- Clone this repository
- In the command line, navigate to your local copy of the repository
- Run
conda env create -f environment.yml- This creates an environment
pyvertical-dev - Comes with most dependencies you will need
- This creates an environment
- Activate the environment with
conda activate pyvertical-dev - Run
conda install notebook
N.b. Installing the dependencies takes several steps to circumvent versioning incompatibility between
syft and jupyter.
In the future,
all packages will be moved into the environment.yml.
To install the dependencies, we recommend using Conda:
- Clone this repository
- In the command line, navigate to your local copy of the repository
- Run
conda env create -f tf_environment.yml- This creates an environment
pyvertical-dev-tf - Comes with most dependencies you will need
- This creates an environment
- Activate the environment with
conda activate pyvertical-dev-tf - Run
conda install notebook
You can instead opt to use Docker.
To run:
- Build the image with
docker build -t pyvertical:latest . - Launch a container with
docker run -it -p 8888:8888 pyvertical:latest- Defaults to launching jupyter lab
PyVertical is applying fake medical data
generated by synthea
to demonstrate multi-party,
vertical federated learning.
Read the synthea docs
for requirements to generate the data.
With those pre-requisites installed,
run the scripts/download_synthea.sh
bash script
from the root directory
of this project,
which generates a deterministic dataset
and stores it in data/synthea.
Check out
examples/PyVertical Example.ipynb
to see PyVertical in action.
- MVP
- Simple example on MNIST dataset
- One data holder has images, the other has labels
- Extension demonstration
- Apply process to electronic health records (EHR) dataset
- Dual-headed SplitNN: input data is split amongst several data holders
- Integrate with
syft
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Read the OpenMined contributing guidelines and styleguide for more information.
 |
 |
 |
 |
 |
|---|---|---|---|---|
| TTitcombe | Pavlos-p | H4LL | rsandmann | daler3 |
We use pytest to test the source code.
To run the tests manually:
- In the command line, navigate to the root of this repository
- Run
python -m pytest
CI also checks the code is formatting according to contributing guidelines.
Romanini, D., Hall, A. J., Papadopoulos, P., Titcombe, T., Ismail, A., Cebere, T., Sandmann, R., Roehm, R. & Hoeh, M. A. (2021). PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN. arXiv preprint arXiv:2104.00489. (link)
Angelou, N., Benaissa, A., Cebere, B., Clark, W., Hall, A. J., Hoeh, M. A., Liu, D., Papadopoulos, P., Roehm, R., Sandmann, R., Schoppmann, P. & Titcombe, T. (2020). Asymmetric Private Set Intersection with Applications to Contact Tracing and Private Vertical Federated Machine Learning. arXiv preprint arXiv:2011.09350. (link)
You can cite this work using:
@article{romanini2021pyvertical,
title={PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN},
author={Romanini, Daniele and Hall, Adam James and Papadopoulos, Pavlos and Titcombe, Tom and Ismail, Abbas and Cebere, Tudor and Sandmann, Robert and Roehm, Robin and Hoeh, Michael A},
journal={arXiv preprint arXiv:2104.00489},
year={2021}
}
@article{angelou2020asymmetric,
title={Asymmetric Private Set Intersection with Applications to Contact Tracing and Private Vertical Federated Machine Learning},
author={Angelou, Nick and Benaissa, Ayoub and Cebere, Bogdan and Clark, William and Hall, Adam James and Hoeh, Michael A and Liu, Daniel and Papadopoulos, Pavlos and Roehm, Robin and Sandmann, Robert and others},
journal={arXiv preprint arXiv:2011.09350},
year={2020}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent