openmined/psi

Private Set Intersection Cardinality protocol based on ECDH and Bloom Filters

PSI

Private Set Intersection protocol based on ECDH and Golomb Compressed Sets or Bloom Filters.

Protocol

The Private Set Intersection (PSI) protocol involves two parties, a client and a server, each holding a dataset. The goal of the protocol is for the client to determine the intersection between their dataset and the server's dataset, without revealing any information about their respective datasets to each other.

The protocol proceeds as follows:

1. Setup (server)

The server encrypts all its elements x under a commutative encryption scheme, computing H(x)^s where s is its secret key. The encrypted elements are then inserted into a container and sent to the client in the form of a serialized protobuf and resembles* the following:

[ H(x_1)^(s), H(x_2)^(s), ... , H(x_n)^(s) ]

2. Request (client)

The client encrypts all their elements x using the commutative encryption scheme, computing H(x)^c, where c is its secret key. The client sends its encrypted elements to the server along with a boolean flag, reveal_intersection, indicating whether the client wants to learn the elements in the intersection or only its size (cardinality). The payload is sent as a serialized protobuf and resembles* the following:

[ H(x_1)^(c), H(x_2)^(c), ... , H(x_n)^(c) ]

3. Response (server)

For each encrypted element H(x)^c received from the client, the server encrypts it again under the commutative encryption scheme with its secret key s, computing (H(x)^c)^s = H(x)^(cs). The result is sent back to the client in a serialized protobuf and resembles* the following:

[ H(x_1)^(cs), H(x_2)^(cs), ... , H(x_n)^(cs) ]

4. Compute intersection (client)

The client decrypts each element received from the server's response using its secret key c, computing (H(x)^(cs))^(1/c) = H(x)^s. It then checks whether each decrypted element is present in the container received from the server, and reports the number of matches as the intersection size.

It's worth noting that the protocol has several variants, some of which introduce a small false-positive rate, while others do not generate false positives. This behavior is selective, and the false-positive rate can be tuned. The selection has implications on communication costs as well.

NOTE resembles*: The protocol has configurable containers. Golomb Compressed Sets (Gcs) is the default container but it can be overridden to be BloomFilter or Raw encrypted strings. Gcs and BloomFilter will have false positives whereas Raw will not. Using Raw increases the communication cost as it is sending raw strings over the wire while the other two options drastically reduce the cost at the price of having false positives.

Security

See SECURITY.md.

Requirements

There are requirements for the entire project which each language shares. There also could be requirements for each target language:

Global Requirements

These are the common requirements across all target languages of this project.

• A compiler such as clang or gcc
• Bazel

Installation

The repository uses a folder structure to isolate the supported targets from one another:

private_set_intersection/<target language>/<sources>

C++

See the C++ README.md

JavaScript

See the JavaScript README.md

Go

See the Go README.md

Python

See the Python README.md

Rust

See the Rust README.md

Usage

To use this library in another Bazel project, add the following to your WORKSPACE file:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
   name = "org_openmined_psi",
   remote = "https://github.com/OpenMined/PSI",
   branch = "master",
)

load("@org_openmined_psi//private_set_intersection:preload.bzl", "psi_preload")

psi_preload()

load("@org_openmined_psi//private_set_intersection:deps.bzl", "psi_deps")

psi_deps()

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "pip_deps",
    # Generated via pip-compile requirements.in
    requirements_lock = "@org_openmined_psi//private_set_intersection/python:requirements.txt",
)

load("@pip_deps//:requirements.bzl", "install_deps")

install_deps()

load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "npm_install")

node_repositories()

npm_install(
    name = "npm",
    package_json = "//:package.json",
    package_lock_json = "//:package-lock.json",
)

load("@emsdk//:emscripten_deps.bzl", emsdk_emscripten_deps = "emscripten_deps")

emsdk_emscripten_deps()

A full description of the protocol can be found in the documentation of the PsiClient class. The corresponding server class is PsiServer. An example of how to interleave the different phases of the protocol can be found in psi_server_test.cpp.

Changes

See CHANGES.md.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

Contributors

See CONTRIBUTORS.md.

License

Apache License 2.0

psi's Issues

Go benchmarks are slow

Description

Go benchmarks were found to be much slower than the C++ benchmarks. On closer examination, there are two issues with Go benchmarks:

1. The setup code need to be moved out of the hot benchmark loop. Currently, the setup is counted in the benchmark time.
2. Test assertions should be removed from the benchmark code. There are already many tests. Having tests in the benchmark would only slow it down.

If these two issues are addressed, then the Go benchmark should be much more accurate.

How to Reproduce

N/A

Expected Behavior

N/A

Screenshots

N/A

System Information

N/A

Additional Context

N/A

I have a few questions and look forward to the recovery of.

1. why choose ecdh instead of OT?
2. has a performance comparison been done with the oprf based kkrt16?
3. have tests with more than 10 million data been executed?
4. i want to use csv file as input, so how do i do it?

Intersection size is bigger than the server size

Description

The result of the intersection of client and server items should be equal or less than both client's items size and server's items size. But with latest code, the intersection in some cases, larger than the server size.

If the intersection indices is larger than client size, we can simply drop the indices > the client items size, but if the intersection is larger than server size, the result is probably wrong.

for example,
client size = 263
server size = 132
The resulting intersection size = 167 > 132, this shouldn't be possible.

How to Reproduce

1. Go to https://github.com/chesterxgchen/psi_mpc_related/blob/main/psi/psi_intersect.ipynb
2. first import psi for version > 1.0.3 (cell 4)
3. then cell 5
4. Scroll down to cell 8 to run the test with emails
5. See output

Expected Behavior

Intersection size <= min (client items, server items) = (132, 263) = 132
but it returns 167

Screenshots

System Information

• OS: Unbuntu
• OS Version: 20.04
• Language Version: [Python 3.8.16]
• Package Manager Version: Pip3

Additional Context

Find out this during NVFLARE test

PSI Future Directions

Welcome to the OpenMined Private Set Intersection repository! Thank you for your interest in contributing to the PSI project.

Description

The OpenMined PSI team works on the core C++ code for Private Set Intersection, which uses Bloom Filters and elliptic-curve Diffie-Hellman (ECDH). Additionally, we want to make our implementation accessible to downstream projects (like PyVertical), so we provide bindings in a variety of languages, from Go to Javascript.

Potential Avenues of Future Work

We are always looking forward to new contributors! Here are some potential areas where you could start contributing:

• Unit and integration tests - help verify that our code is secure and reliable.
• Documentation - help make our code more user-friendly. The documentation needs to be updated for the newly added Golomb Compressed Sets.
• New language bindings - help make PSI more accessible. Currently, we are interested in Java bindings, and we need to update all bindings to handle Golomb Compressed Sets.
• New features and algorithm improvements - any new features or improvements to our algorithm would be beneficial. There are many applications of Private Set Intersection, and implementing new applications is always welcome.

Feel free to expand on this issue with more ideas! Open a new issue or pull request to get feedback from other contributors.

How to install and run in pycharm?

How How to install and run this package in pycharm?

I tried pip3, conda. pip installed openminded.psi, but conda is not able to find the package, and in pycharm, it is not able to import what pip3 installed.

Thanks for your help.

Python Bazel builds fail on Windows

Description

I am trying to execute a python test using bazel test //private_set_intersection/python:tests --test_output=all or a benchmark using bazel run -c opt //private_set_intersection/python:benchmarks --test_output=all but the bazel builds fail.

On my Ubuntu WSL, they work fine as intended.

How to Reproduce

1. Attempt to build the test on Windows

Expected Behavior

Build to be completed successfully.

Screenshots

System Information

• OS: Microsoft Windows 10 Pro
• OS Version: 10.0.19041 Build 19041
• Language Version: Python 3.7, 3.8
• Bazel Version: Bazel 3.3.1

Cannot build for Go: Unable to load package for //private_set_intersection/python:requirements.txt

Question

During bazel build an error occurs that BUILD file is not found in /private_set_intersection/python

Further Information

1. I edited my WORKSPACE and BUILD files according to the instructions
2. I run "bazel build //..."
3. An error occurs:

INFO: Repository pip_deps instantiated at:
  /Users/a1/git_repos/repo_name/WORKSPACE:46:9: in <toplevel>
  /private/var/tmp/_bazel_a1/dfe3157b64ac0d90774c2ee53fc75141/external/org_openmined_psi/private_set_intersection/deps.bzl:111:14: in psi_deps
  /private/var/tmp/_bazel_a1/dfe3157b64ac0d90774c2ee53fc75141/external/rules_python/python/pip.bzl:160:19: in pip_parse
Repository rule pip_repository defined at:
 /private/var/tmp/_bazel_a1/dfe3157b64ac0d90774c2ee53fc75141/external/rules_python/python/pip_install/pip_repository.bzl:474:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'pip_deps':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_a1/dfe3157b64ac0d90774c2ee53fc75141/external/rules_python/python/pip_install/pip_repository.bzl", line 308, column 18, in _pip_repository_impl
                rctx.path(requirements_txt),
Error in path: Unable to load package for //private_set_intersection/python:requirements.txt: BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /Users/a1/git_repos/repo_name/private_set_intersection/python

I even tried to add this folder locally, but after that I began receiving other errors. I'm not sure if it's a bug or I'm doing something incorrectly.

My WOKRSPACE:
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_go",
    integrity = "sha256-fHbWI2so/2laoozzX5XeMXqUcv0fsUrHl8m/aE8Js3w=",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.44.2/rules_go-v0.44.2.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.44.2/rules_go-v0.44.2.zip",
    ],
)

http_archive(
    name = "bazel_gazelle",
    integrity = "sha256-MpOL2hbmcABjA1R5Bj2dJMYO2o15/Uc5Vj9Q0zHLMgk=",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.35.0/bazel-gazelle-v0.35.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.35.0/bazel-gazelle-v0.35.0.tar.gz",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
load("//:deps.bzl", "go_dependencies")

# gazelle:repository_macro deps.bzl%go_dependencies
go_dependencies()

go_rules_dependencies()

gazelle_dependencies(go_repository_default_config = "//:WORKSPACE.bazel")

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
   name = "org_openmined_psi",
   remote = "https://github.com/OpenMined/PSI",
   branch = "master",
)

load("@org_openmined_psi//private_set_intersection:preload.bzl", "psi_preload")

psi_preload()

load("@org_openmined_psi//private_set_intersection:deps.bzl", "psi_deps")

psi_deps()

load("@pip_deps//:requirements.bzl", "install_deps")

install_deps()

load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "npm_install")

node_repositories()

npm_install(
    name = "npm",
    package_json = "//:package.json",
    package_lock_json = "//:package-lock.json",
)

load("@emsdk//:emscripten_deps.bzl", emsdk_emscripten_deps = "emscripten_deps")

emsdk_emscripten_deps()

System Information

• OS: MacOS
• OS Version: 13.6 (22G120)
• Language Version: bazel 6.4.0-homebrew, go 1.21.6.

Additional Context

Tried on Win10 with WSL, the result is the same.

Error during server.ts compilation

Hello! Im trying to compile-run the server.ts file, but i face the following issue:

What can i do to overcome this problem?

To find out server side intersection, is possible to use bloom filter bit operation ?

Question

In the federated learning case, the intersections needs to be calculated for both sites. using PSI to achieve this, requires reverse process. If we want to improve the performance, can we leverage the bit operation on filter rather than reverse the PSI setup, request, response process.

Further Information

Use Case: Federated Vertical learning, two sites has different user records, we need find the match user Ids before we can training the mode. Both Sites will need to know the intersection.
Suppose we use PSI to find the match user Ids. Take site-1 as PSI Server (S), site-2 as PSI client (C). The current process

Forward Pass:
S (setup msg) --> bloom filter ==> delivery ==> C (receive setup bloom filter)
S (receive request) <== request <== C (prepare request msg)
S (response msg) ==> deliver ===> C ( receive response, compare with the setup bloom filter to find the intersection)

at the end of forward Pass, Site-1 doesn't know intersection, Site-2 knows the intersection.

To calculate the S intersection, we can reverse the process, now takes Site-2 as Server Role, use the intersection as the Items. Repeat above process, called Reverse Pass. then Site-1 knows the intersection as well.

This works, but I am wondering if there is a faster way to do this. Looking at above Forward Pass.

Step 3 Client: Intersection calculation,
One we have the intersection, we can derive an new bloom filter based on the setup bloom filter where only the intersection bits are 1, others are 0. Let's call it intersect bloom filter

On the reverse pass, instead repeat above step, can we simply sent the intersect bloom filter back to Server.

C (intersect bloom filter) ===> S (receive intersect bloom filter)

Server
loop all items to check against the intersect bloom filter to figure out the intersection.

The question, is this reasonable and secure ? If yes, can you add these in the Python APIs ?

Screenshots

None

System Information

N/A

Additional Context

Add any other context about the problem here.

Question regarding python version requirements >=3.8.10

Question

Is it possible to support python version >=3.7.0 ?

Further Information

based on the pypi page, https://pypi.org/project/openmined.psi/

Meta
License: Apache 2.0

Author: OpenMined

Requires: Python >=3.8.10

PSI requires 3.8.10+, is it possible to support 3.7.0+ ?

Screenshots

n/a

System Information

Ubuntu

Additional Context

NVFLARE is currently requires python >=3.7. my current pip install openmined.psi failed in CI env.

Create a better Readme

Based on this readme template, we should improve our readmes across all OpenMined projects.

Please consult the following document for our desired best practices.

Fix workflow for MacOS

The MacOS CI workflow currently fails at installing dependencies because it doesn't find apt. I think we need to have a different install script for Mac. Maybe outsource the install script to a shell script so that the tests.yml can be the same for all platforms?

Unhandled promise rejection: ReferenceError: Can't find variable: WebAssembly

Description

In my react-native + expo app (ios 15.1), I'm carefully following the javascript readme file instructions. I have installed @openmined/psi.js and react-native-get-random-values. Then I initialise the code as following:

import 'react-native-get-random-values';
import PSI from '@openmined/psi.js/combined_wasm_web';

(async () => {
  // Spoof the browser document
  global.document = {} as any;
  // Wait for the library to initialize
  const psi = await PSI();
  const client = psi.client!.createWithNewKey();
  //...
})();

When I start the app, I get the error:
no native wasm support detected
Unhandled promise rejection: ReferenceError: Can't find variable: WebAssembly

It works fine on android.

How to Reproduce

1. Create new react-native + expo app: expo init my_app
2. install @openmined/psi.js and react-native-get-random-values
3. Add following code in file App.tsx:

import 'react-native-get-random-values';
import PSI from '@openmined/psi.js/combined_wasm_web';

(async () => {
  // Spoof the browser document
  global.document = {} as any;
  // Wait for the library to initialize
  const psi = await PSI();
  const client = psi.client!.createWithNewKey();
  //...
})();

4. See error

Expected Behavior

App should start without error

Screenshots

System Information

• OS: IOS 15.1
• react-native: 0.64.3
• expo: 44
• node: v14.18.1
• npm: 8.3.0

PSI builds failing

Description

When I try PyVertical (https://github.com/OpenMined/PyVertical), there are some problems in running build-psi.sh. The ERROR infomation is as follows:

INFO: Repository rule 'org_openmined_psi' returned: {"remote": "https://github.com/OpenMined/PSI", "commit": "ca2866e3c70a018ed9a5a2e7af5bcbd31dc49df9", "shallow_since": "2020-14-07", "init_submodules": True, "verbose": False, "strip_prefix": "", "patches": [], "patch_tool": "patch", "patch_args": ["-p0"], "patch_cmds": [], "name": "org_openmined_psi"}
ERROR: /home/tanghengzhu/.cache/bazel/_bazel_tanghengzhu/9b06aa84c50b71eaea609540d6aa319b/external/org_openmined_psi/private_set_intersection/javascript/toolchain/cc_toolchain_config.bzl:171:17: name 'CcToolchainConfigInfo' is not defined
ERROR: error loading package '': Extension 'private_set_intersection/javascript/toolchain/cc_toolchain_config.bzl' has errors
ERROR: error loading package '': Extension 'private_set_intersection/javascript/toolchain/cc_toolchain_config.bzl' has errors
INFO: Elapsed time: 8.584s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

How to Reproduce

1. Installed bazel
2. run the build script with .github/workflows/scripts/build-psi.sh..'

Expected Behavior

This should generate a _psi_bindings.so file and place it in src/psi/.

System Information

• OS Version: Linux sigmaster 3.10.0 Red Hat 4.8.5-39
• Language Version: python 3.7.8
• Package Manager Version: Conda 4.6.14
• Bazel Version: 1.0.0 (3.7.1 was also tried)

Request for GetPrivateKeyBytes to return a vector<uint8_t>

Problem

PSICardinalityServer::GetPrivateKeyBytes() returns a binary string containing the private key.

Several language bindings do not play nicely with binary strings resulting in various workarounds that are fragile. This is especially true for JavaScript and Go that don't play nicely with bindings handling binary strings.

Proposal

Suggestion to change the output to a vector<uint8_t> for better interoperability our current and future bindings.

When Client and Server items are in some certain sizes, server.CreateSetupMessage raise ValueError: basic_string::resize

Description

When client items size and server items size are in certain values, the PSI code throw basic_string::resize Value Error.
For example, using the tests.py https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/tests.py#L28

if we change the client items and server items range into the following values:
client_items = ["Element " + str(i) for i in range(333334)]
server_items = ["Element " + str(2 * i) for i in range(397)]

You should be able to get the exception

File ~/nvflare-env2/lib/python3.8/site-packages/openmined_psi/__init__.py:140, in server.CreateSetupMessage(self, fpr, num_client_inputs, inputs)
    129 def CreateSetupMessage(
    130     self, fpr: float, num_client_inputs: int, inputs: List[str]
    131 ) -> ServerSetup:
    132     """Create a setup message from the server's dataset to be sent to the client.
    133     Args:
    134         fpr: the probability that any query of size `num_client_inputs` will result in a false positive.
   (...)
    138         A Protobuf with the setup message.
    139     """
--> 140     interm_msg = self.data.CreateSetupMessage(fpr, num_client_inputs, inputs).save()
    141     msg = ServerSetup()
    142     msg.ParseFromString(interm_msg)

ValueError: basic_string::resize

How to Reproduce

1. check the code in https://github.com/chesterxgchen/psi_mpc_related/blob/main/psi/psi_intersect.ipynb
   command Cell No. 5 should re-produce the error.
   The code in this notebook is essentially the same as https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/tests.py#L28

2. pip install the openmined.psi

3. start the jupyter notebook with https://github.com/chesterxgchen/psi_mpc_related/blob/main/psi/psi_intersect.ipynb

4. execute the import the psi packages, and def dup() function, then run cmd 5, to see error

Expected Behavior

No error

Screenshots

System Information

• OS: Ubuntu
• OS Version: 20.04
• Openmined PSI version: openmined.psi-0.3.5 ( directly downloaded from Pipy)
• Language Version: python 3.8.16 (default, Dec 7 2022, 01:12:06)
• Package Manager: Version pip 22.3.1
• Browser (if applicable): NA
• Browser Version (if applicable): NA

Additional Context

We experience this issue while testing our openmined-based PSI solution

Python: use py_proto_library

Description

For consistency, we should use Python protobuffers for the Python bindngs instead of wrapping the C++ one.

Additional Context

One potential starting point -> https://thethoughtfulkoala.com/posts/2020/05/08/py-protobuf-bazel.html

Building a Python Wheel (on Ubuntu) with lower version of Protobuf

Question

What's the approach to build Python wheel with lower version of Protobuf ?

Further Information

While trying to integrate with Openmined PSI with NVFLARE, our CI/CD process reported that

openmined-psi 1.0.0 depends on protobuf==4.21.12
    nvflare-nightly 0+untagged.1.g34f97a7 depends on protobuf==3.20.2

So we hope to build a wheel that has lower protobuf version requirement.

Look at the code, we changed the following files

https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/requirements.in#L7
https://github.com/OpenMined/PSI/blob/master/private_set_intersection/python/BUILD#L105

and use

pip-compile requirements.in

to generate the requirements.txt

then use the following command to build the wheel:

bazel build -c opt //private_set_intersection/python:wheel

The output showed the following warning:

~/projects/PSI/private_set_intersection/python$ bazel build -c opt //private_set_intersection/python:wheel
INFO: Analyzed target //private_set_intersection/python:wheel (135 packages loaded, 2284 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/google/protobuf/message_lite.cc [for tool]:
In file included from /usr/include/string.h:535,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/port.h:39,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/stubs/common.h:48,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.h:46,
                 from external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:36:
In function 'void* memcpy(void*, const void*, size_t)',
    inlined from 'uint8_t* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, uint8_t*)' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/io/coded_stream.h:684:16,
    inlined from 'virtual uint8_t* google::protobuf::internal::ImplicitWeakMessage::_InternalSerialize(uint8_t*, google::protobuf::io::EpsCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/implicit_weak_message.h:103:28,
    inlined from 'bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const' at external/com_github_protocolbuffers_protobuf/src/google/protobuf/message_lite.cc:411:30:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: warning: 'void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)' specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |                                  __glibc_objsize0 (__dest));
      |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
Target //private_set_intersection/python:wheel up-to-date:
  bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
INFO: Elapsed time: 27.419s, Critical Path: 11.53s
INFO: 737 processes: 27 internal, 709 linux-sandbox, 1 local.
INFO: Build completed successfully, 737 total actions

Although it said its successful, but there is no wheel

~/projects/PSI/private_set_intersection/python$ ls -l bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl
ls: cannot access 'bazel-bin/private_set_intersection/python/openmined.psi-1.0.0-INTERPRETER-ABI-manylinux2014_x86_64.whl': No such file or directory

Screenshots

System Information

• OS: Ubuntu x86_64
• OS Version: 20.04
• Language Version: python 3.8.16
• Package Manager Version:, bazel 6.0.0

Additional Context

This is part of the effort trying to integrate openmined PSI with NVFLARE in Federated learning. Although with protobuf version conlicts. My local tests seems to be fine. But the unit tests failed in CI process. So we are hoping there is a build with lower protobuf requirements.

Does the PSI python needs such the latest version of protobuf ?

Error in python psi for v2.0.1 on mac10.15.4

Description

A clear and concise description of the bug.

How to Reproduce

1. In source code, modify init.py
   

2. run PSI/private_set_intersection/python/tests.py directly

3. See error:

Expected Behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

System Information

• OS: mac
• OS Version: 10.15.4
• Language Version: python3.9

Additional Context

Error: AttributeError: module '_openmined_psi' has no attribute 'data_structure'
Thanks very much!!

Could you provide more details about this project?

Question

Describe your question in ONE SENTENCE.

1. Can both parties get the intersection itself instead of its cardinality ?
2. What PSI protocols are implemented in this project?

Failing build when applying PSI

Description

The PyVertical project has applied the python PSI bindings.
Despite compiling locally, we are suddenly getting failing bazel builds on every build.
See this build, for example.

The issue is an incorrect checksum. We're not sure if the cause is:

• Incorrect implementation within PyVertical of PSI
• PSI
• pybind
• github CI's bazel

Our workspace file points to a specific commit of this repo, not master branch,
to avoid upstream breaking changes. I don't know yet whether that is a cause.

Screenshots

If applicable, add screenshots to help explain your problem.

System Information

• OS: Ubuntu
• OS Version: 18.04
• Language Version: Python 3.6, 3.7, 3.8

MacOS: Python package name with incorrect platform name

Description

The wheel build generates a package with the name openmined.psi-0.3.3-cp37-cp37m-macosx_11_0_x86_64.whl , but the build is executed on MacOS 10.14. distutil.util.get_platform() seems to get the wrong platform name.

The versioning worked fine for the 0.3.2 release https://pypi.org/project/openmined.psi/0.3.2/#files

Result

pip install ignores the version name on older MacOS-es, although the binary works fine.

Expected Behavior

We need to support at least MacOS 10.14 for now.
We need to correct the wheel name on MacOS to openmined.psi-0.3.3-cp37-cp37m-macosx_10.14_x86_64.whl

not a supported wheel on this platform

Question

Describe your question in ONE SENTENCE.

Further Information

When I build on mac10.15.4, buile sucess, but failed when install.

cd PSI/private_set_intersection/python
pip install requirements.in
pip install requirements.txt
cd ~/PSI
bazel build -c opt private_set_intersection/python:wheel
ls bazel-bin/private_set_intersection/python/openmined.psi-2.0.1-INTERPRETER-ABI-macosx_12_0_x86_64.whl
python private_set_intersection/python/rename.py 
pip install ./bazel-bin/private_set_intersection/python/openmined.psi-2.0.1-cp39-cp39-macosx_12_0_x86_64.whl

Screenshot

System Information

• OS: mac
• OS Version: 'macOS-10.15.4-x86_64-i386-64bit'
• Language Version: python 3.9.5
• Package Manager Version: conda 4.3

Additional Context

Add any other context about the problem here.

Unable to compile in React-Native

Description

We are unable to compile and use the library in react native
Tried both es and umd versions

import PSIClient from '@openmined/psi.js/client/js/umd';
import PSIClient from '@openmined/psi.js/client/js/es';

node libraries seem to be required path stream fs as in the attached screenshot

Despite several hacks to import and map react-native counterparts for node libraries like react-native-level-fs readable-stream rn-nodeify, the issue persists

How to Reproduce

Import libraries in react native

Expected Behavior

I would expect the project to compile without errors

Screenshots

System Information

• OS (target): Androidd
• OS Version: 9
• Language Version: React Native 0.62
• Package Manager Version: Yarn 1.16, NPM 6.11

In a react-native app, how to read back data from server (node.js) when using fetch api

Question

In a react-native app, how to read back data from server (node.js) when using fetch api.

Further Information

I have a react-native app sending contacts phone numbers to server for processing as described in the javascript readme document. Upon receiving data back, the call client.getIntersectionSize throws error:

BindingError: _emval_take_value has unknown type N4absl11string_viewE

I suspect this is due to the way I'm reading data back from server. Here are relevant excerpts from source code:

1. client sends serialized request:

...
const psi = await PSI();
const client = psi.client!.createWithNewKey(true);
...
const serializedClientRequest = request.serializeBinary();
const res = await fetch('__server_address__', {
   method: 'POST',
   body: serializedClientRequest,
});

2. server receives, processes then sends response back:

...
const psi = await PSI();
const server = psi.server!.createWithNewKey(true);
...
const serverResponse = server.processRequest(deserializedClientRequest);
const serializedServerResponse = serverResponse.serializeBinary();
const serializedServerSetup = serverSetup.serializeBinary();

// sends this json back. Is this correct?
return {
  serializedServerResponse,
  serializedServerSetup
};

3. clients receives data and attempts to get intersection size:

...
const psi = await PSI();
const client = psi.client!.createWithNewKey(true);
...
const res = await fetch('__server_address__', {
   method: 'POST',
   body: serializedClientRequest,
});
const response = await res.json();

// I must convert data received from server into Uint8Array otherwise below lines throw exception
const deserializedServerResponse = psi.response.deserializeBinary(new Uint8Array(response.serializedServerResponse));
const deserializedServerSetup = psi.serverSetup.deserializeBinary(new Uint8Array(response.serializedServerSetup));

// this line fails with error in title
const intersectionSize = client.getIntersectionSize(deserializedServerSetup, deserializedServerResponse);

Is the above code the correct way to process data received from server before attempting to retrieve intersection size"? This part is not very explicit in the javascript readme document. I have tried with the axios library then the fetch api, not sure if it makes any difference.

You help will be very appreciated as we would like to strictly adhere to best practices when it comes to mobile contacts discovery.

System Information

 - OS: IOS 15.1
 - react-native: 0.64.3
 - expo: 44
 - node: v14.18.1
 - npm: 8.3.0

Rust Bindings

Hey, I am willing to help create a Rust binding for this project over the summer. Here's what I think may be possible through my brief look at the code:

1. Create a simple wrapper over the C++ code (like the bindings in the other languages). Rust does not support directly interfacing with C++, so the backend bloom filter stuff has to be exposed through C, before they are called in Rust.
2. Reimplement the bloom filter in Rust. Rust is very efficient and safe, so this may be beneficial in guaranteeing memory safety and eliminating bugs.
3. If possible, I want to try to use SIMD intrinsics in the Rust reimplementation of the bloom filter for speedups. I do remember some people applying SIMD to bloom filters in other fields like genomic sketching. I am not sure if this is possible with the current implementation but I will think about it if I reach this point.

If the maintainers are interested in this, then I can get started with getting familiar to the code. I would prefer to do this project from around mid-June to August/September.

About me: I am a high school student going into undergrad in CS next year. I have a lot of experience with writing research software in fields like deep learning or bioinformatics. Currently, I am getting familiar with Rust by working on a SIMD-accelerated library for string edit distances. In fact, in my project, I've seen SIMD give up to 20-30x improvement over their scalar implementations. I have no formal background in security, but I am relatively familiar with the concept of bloom filters and I've worked on adversarial robustness for deep learning. I am willing to dedicate a significant chunk of time to learning, writing code, benchmarking, and testing for this project. My goal is to make a useful Rust binding and also allow myself to gain some experience working on a large software project.

`response_array` Python build error on Ubuntu

Description

Building the latest commit fails as response_array was used without being declared in external/org_openmined_psi/private_set_intersection/cpp/psi_client.cpp.

See this build in PyVertical

Our workspace file is:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
   name = "org_openmined_psi",
   remote = "https://github.com/OpenMined/PSI",
   commit = "2110b0c78aaf535f0f154541a5239ae1d4876023",
   init_submodules = True,
)

load("@org_openmined_psi//private_set_intersection:preload.bzl", "psi_preload")

psi_preload()

load("@org_openmined_psi//private_set_intersection:deps.bzl", "psi_deps")

psi_deps()

and we build by running

bazel build @org_openmined_psi//private_set_intersection/python:private_set_intersection

How to Reproduce

1. Attempt to build the Python code for the latest commit

Expected Behavior

Build should pass

Screenshots

System Information

• OS: Ubuntu
• OS Version: 18.04
• Language Version: Python 3.6,3.7,3.8

Go Server crash

The go bindings will crash when trying to create a setup message with an empty set.

This is because the first element of the set is used to get a pointer to the whole set.

PSI/private_set_intersection/go/server/server.go

Line 159 in dfcb30f

rcode := C.psi_server_create_setup_message(s.context, C.double(fpr), C.int64_t(inputCount), &input[0], C.size_t(len(input)), &out, &outlen, &err)

Is the security of the Golomb-based scheme solely based on having a large domain and a strong hash?

Question

Is the security of the Golomb-based scheme solely based on having a large domain and a strong hash?

Further Information

First, this repository looks very nice! Congratulations on the project and the support for so many different languages. I noticed that you have switched from Bloom filters to Golomb-coded sets since they are more compact. However, looking through the code I got the impression that the receiver would be able to see the entire Golomb-encodings. Does this not allow the receiver to retrieve the entire encoded set if the domain is small enough (or the hash is not strong)?