google-parfait / federated-compute Goto Github PK
View Code? Open in Web Editor NEWLibraries for executing federated programs and computations.
License: Apache License 2.0
Libraries for executing federated programs and computations.
License: Apache License 2.0
When trying to build the platform from Ubuntu 22.04 LTS, the build fails. It's due to different versions of GCC. In Debian, we have access to GCC 10, but in Ubuntu, we have access to GCC 11.
Here is the stacktrace I get when I try to build the platform on ubuntu:
==> googlecompute.federated-compute-platform: ERROR: /home/packer/.cache/bazel/_bazel_packer/ca0f3d8f2b007992a7f5dcb18911d5f2/external/boringssl/BUILD:130:11: Compiling src/third_party/fiat/curve25519.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 36 arguments skipped)
==> googlecompute.federated-compute-platform:
==> googlecompute.federated-compute-platform: Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
==> googlecompute.federated-compute-platform: external/boringssl/src/third_party/fiat/curve25519.c:511:57: error: argument 2 of type 'const uint8_t[32]' {aka 'const unsigned char[32]'} with mismatched bound [-Werror=array-parameter=]
==> googlecompute.federated-compute-platform: 511 | int x25519_ge_frombytes_vartime(ge_p3 *h, const uint8_t s[32]) {
==> googlecompute.federated-compute-platform: | ~~~~~~~~~~~~~~^~~~~
==> googlecompute.federated-compute-platform: In file included from external/boringssl/src/third_party/fiat/curve25519.c:41:
==> googlecompute.federated-compute-platform: external/boringssl/src/third_party/fiat/internal.h:117:58: note: previously declared as 'const uint8_t *' {aka 'const unsigned char *'}
==> googlecompute.federated-compute-platform: 117 | int x25519_ge_frombytes_vartime(ge_p3 *h, const uint8_t *s);
==> googlecompute.federated-compute-platform: | ~~~~~~~~~~~~~~~^
==> googlecompute.federated-compute-platform: external/boringssl/src/third_party/fiat/curve25519.c:831:57: error: argument 2 of type 'const uint8_t *' {aka 'const unsigned char *'} declared as a pointer [-Werror=array-parameter=]
==> googlecompute.federated-compute-platform: 831 | void x25519_ge_scalarmult_base(ge_p3 *h, const uint8_t *a) {
==> googlecompute.federated-compute-platform: | ~~~~~~~~~~~~~~~^
==> googlecompute.federated-compute-platform: In file included from external/boringssl/src/third_party/fiat/curve25519.c:41:
==> googlecompute.federated-compute-platform: external/boringssl/src/third_party/fiat/internal.h:125:56: note: previously declared as an array 'const uint8_t[32]' {aka 'const unsigned char[32]'}
==> googlecompute.federated-compute-platform: 125 | void x25519_ge_scalarmult_base(ge_p3 *h, const uint8_t a[32]);
==> googlecompute.federated-compute-platform: | ~~~~~~~~~~~~~~^~~~~
==> googlecompute.federated-compute-platform: cc1: all warnings being treated as errors
There is no README file in the following folder
https://github.com/tensorflow/federated/blob/main/tensorflow_federated/python/program
so the link to the Federated Program API in TFF
404's
When building the compiler exits with error:
In member function 'const void* fcp::aggregation::VectorData::data() const':
./fcp/aggregation/core/vector_data.h:39:52: error: 'const fcp::aggregation::VectorData::vector' has not been declared
full output is:
ERROR: /home/doug/federated-compute/fcp/aggregation/core/BUILD:150:11: Compiling fcp/aggregation/core/federated_sum.cc failed: (Exit 1): gcc failed: error executing command
(cd /root/.cache/bazel/bazel_root/a2d1cfe108fb6ecf8584c66084ab27f5/sandbox/processwrapper-sandbox/11649/execroot/com_google_fcp &&
exec env -
PATH=/root/.cache/bazelisk/downloads/bazelbuild/bazel-5.3.0-linux-x86_64/bin:/home/doug/federated-compute/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/proc/self/cwd
TF2_BEHAVIOR=1
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/fcp/aggregation/core/objs/federated_sum/federated_sum.pic.d '-frandom-seed=bazel-out/k8-opt/bin/fcp/aggregation/core/objs/federated_sum/federated_sum.pic.o' -fPIC -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib '-std=c++17' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE="redacted"' '-D__TIMESTAMP_="redacted"' '-D__TIME__="redacted"' -c fcp/aggregation/core/federated_sum.cc -o bazel-out/k8-opt/bin/fcp/aggregation/core/_objs/federated_sum/federated_sum.pic.o)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from ./fcp/aggregation/core/agg_vector_aggregator.h:26,
from fcp/aggregation/core/federated_sum.cc:20:
./fcp/aggregation/core/vector_data.h: In member function 'const void* fcp::aggregation::VectorData::data() const':
./fcp/aggregation/core/vector_data.h:39:52: error: 'const fcp::aggregation::VectorData::vector' has not been declared
const void* data() const override { return this->vector::data(); }
^~~~~~
Target //fcp/demo:federated_program_test failed to build
The demo uses an http federated compute protocol. There is also a gRPC federated API that the client supports. It would be good to also include a demo server that supports a gRPC client.
How to combine PCS (Private-compute-services) and FCP effectively for federated learning and analytics? could you please share any helpful blogs or resources on this topic?
Hi folks!
When installing the latest available commit in this repository (8d4b013673f87ccd35c9ec95a97749a5ff83f467
), I get the following dependency conflict error:
ERROR: Cannot install -r requirements.txt (line 27), -r requirements.txt (line 28) and absl-py==2.* and >=2.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested absl-py==2.* and >=2.0
tensorflow 2.14.0 depends on absl-py>=1.0.0
tensorflow-federated 0.68.0 depends on absl-py==1.* and >=1.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
This happens both on a x86_64 Linux with Ubuntu 22.04, and on a M1 macOS
To reproduce, from your shell (I tested with Docker):
docker run --rm -it ubuntu:22.04
apt update
apt install -y git gcc python3 python3-dev python3-venv curl
curl --output /usr/local/bin/bazelisk-linux-amd64 https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-linux-amd64
chmod a+x /usr/local/bin/bazelisk-linux-amd64
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
git clone https://github.com/google/federated-compute.git
cd federated-compute
pip install -r requirements.txt
Thanks for your support!
I followed the GETTING_STARTED guide and got a successful build.
But the demo test (bazelisk //fcp/demo:federated_program_test --config=clang
) failed locally. The test log is pasted as following:
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //fcp/demo:federated_program_test
-----------------------------------------------------------------------------
2023-04-07 17:14:13.652917: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-07 17:14:13.690128: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-04-07 17:14:13.690429: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-07 17:14:14.381273: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:From /usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py:576: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
Instructions for updating:
experimental_relax_shapes is deprecated, use reduce_retracing instead
WARNING:tensorflow:From /usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py:576: calling function (from tensorflow.python.eager.polymorphic_function.polymorphic_function) with experimental_relax_shapes is deprecated and will be removed in a future version.
Instructions for updating:
experimental_relax_shapes is deprecated, use reduce_retracing instead
Traceback (most recent call last):
File "/usr/local/google/home/ryangu/.cache/bazel/_bazel_ryangu/6e3f6b9903df237029824c625696458e/sandbox/linux-sandbox/12836/execroot/com_google_fcp/bazel-out/k8-opt/bin/fcp/demo/federated_program_test.runfiles/com_google_fcp/fcp/demo/federated_program_test.py", line 24, in <module>
import tensorflow_federated as tff
File "/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/__init__.py", line 78, in <module>
backends.native.set_sync_local_cpp_execution_context()
File "/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/python/core/backends/native/execution_contexts.py", line 164, in set_sync_local_cpp_execution_context
context = create_sync_local_cpp_execution_context(
File "/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/python/core/backends/native/execution_contexts.py", line 148, in create_sync_local_cpp_execution_context
factory = executor_factory.local_cpp_executor_factory(
File "/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/executor_factory.py", line 103, in local_cpp_executor_factory
_decompress_file(compressed_path, binary_path)
File "/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/executor_factory.py", line 55, in _decompress_file
with open(output_path, 'wb') as binary_file:
OSError: [Errno 30] Read-only file system: '/usr/local/google/home/ryangu/venv/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executor_stacks/../../../../data/worker_binary'
I only used sudo
on two apt install
commands, as instructed:
sudo apt install -y git gcc python3 python3-dev python3-venv
and
sudo apt install -y clang lld libc++-dev libc++abi-dev
win11
3.10
bazelisk test //fcp/demo:federated_program_test
Repository rule http_archive defined at:
C:/users/corgi/_bazel_corgi/hiuillts/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: repository @org_tensorflow' used the following cache hits instead of downloading the corresponding file.
* Hash '99c732b92b1b37fc243a559e02f9aef5671771e272758aa4aec7f34dc92dac48' for https://github.com/tensorflow/tensorflow/archive/v2.11.0.tar.gz
If the definition of 'repository @org_tensorflow' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'org_tensorflow':
Traceback (most recent call last):
File "C:/users/corgi/_bazel_corgi/hiuillts/external/bazel_tools/tools/build_defs/repo/http.bzl", line 143, column 10, in _http_archive_impl
patch(ctx, auth = auth)
File "C:/users/corgi/_bazel_corgi/hiuillts/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 180, column 21, in patch
fail("Error applying patch %s:\n%s%s" %
Error in fail: Error applying patch @//fcp/patches:tensorflow_googletest.patch:
The system cannot find the path specified.
ERROR: E:/workspace/repo/federated-compute/WORKSPACE:147:13: fetching http_archive rule //external:org_tensorflow: Traceback (most recent call last):
File "C:/users/corgi/_bazel_corgi/hiuillts/external/bazel_tools/tools/build_defs/repo/http.bzl", line 143, column 10, in _http_archive_impl
patch(ctx, auth = auth)
File "C:/users/corgi/_bazel_corgi/hiuillts/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 180, column 21, in patch
fail("Error applying patch %s:\n%s%s" %
Error in fail: Error applying patch @//fcp/patches:tensorflow_googletest.patch:
The system cannot find the path specified.
ERROR: Error computing the main repository mapping: no such package '@org_tensorflow//tensorflow': Error applying patch @//fcp/patches:tensorflow_googletest.patch:
The system cannot find the path specified.
Loading:
E:\WorkSpace\repo\federated-compute>
any ideas?😁
where can i find the Class which name is fcp.client.tasks.Task?
Follow the GETTING_STARTED guide to run the test with Clang.
The full error output is:
Compiling fcp/tensorflow/delete_file_op.cc failed: (Exit 1): clang failed: error executing command (from target //fcp/tensorflow:_delete_file_op.so_lib) /usr/lib/llvm-14/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 ... (remaining 48 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from fcp/tensorflow/delete_file_op.cc:16:
In file included from external/system_provided_tf/headers/tensorflow/core/framework/op_kernel.h:26:
In file included from external/system_provided_tf/headers/tensorflow/core/framework/allocator.h:26:
In file included from external/system_provided_tf/headers/tensorflow/core/framework/numeric_types.h:27:
In file included from external/system_provided_tf/headers/tensorflow/core/platform/types.h:23:
In file included from external/system_provided_tf/headers/tensorflow/core/platform/tstring.h:24:
In file included from external/system_provided_tf/headers/tensorflow/core/platform/cord.h:25:
In file included from external/system_provided_tf/headers/tensorflow/core/platform/default/cord.h:22:
In file included from external/system_provided_tf/headers/absl/strings/cord.h:82:
external/system_provided_tf/headers/absl/strings/internal/cord_rep_btree.h:734:5: error: use of undeclared identifier 'ABSL_INTERNAL_ASSUME'
ABSL_INTERNAL_ASSUME(new_end <= kMaxCapacity);
My configs:
Python: Python 3.10.9
Clang:
Debian clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Bazelisk: bazel 6.1.1
tensorflow & tensorflow_federated are installed via requirements.txt
Following the GETTING_STARTED guide, I successfully installed all dependencies in requirements.txt
and verified with python3 -c "import tensorflow_federated"
:
2023-10-20 23:47:49.801507: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-10-20 23:47:49.862608: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-10-20 23:47:49.863466: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-10-20 23:47:51.225016: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Then, I run bazel build //fcp/artifact_building:federated_compute_plan_builder
and got the following build error:
INFO: Analyzed target //fcp/artifact_building:federated_compute_plan_builder (181 packages loaded, 15873 targets configured).
INFO: Found 1 target...
ERROR: /usr/local/google/home/ryangu/.cache/bazel/_bazel_ryangu/15955890b671f3e842c09329bd3a9df5/external/org_tensorflow/tensorflow/tsl/lib/io/BUILD:204:11: Compiling tensorflow/tsl/lib/io/cache.cc [for tool] failed: (Exit 1): gcc failed: error executing command (from target @org_tensorflow//tensorflow/tsl/lib/io:cache) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 49 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from external/org_tensorflow/tensorflow/tsl/lib/io/cache.cc:16:
external/org_tensorflow/tensorflow/tsl/lib/io/cache.h:99:11: error: 'uint64_t' does not name a type
99 | virtual uint64_t NewId() = 0;
| ^~~~~~~~
external/org_tensorflow/tensorflow/tsl/lib/io/cache.h:20:1: note: 'uint64_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
19 | #include "tensorflow/tsl/platform/stringpiece.h"
+++ |+#include <cstdint>
20 |
external/org_tensorflow/tensorflow/tsl/lib/io/cache.cc:391:12: error: 'uint64_t tsl::table::{anonymous}::ShardedLRUCache::NewId()' marked 'override', but does not override
391 | uint64_t NewId() override {
| ^~~~~
Target //fcp/artifact_building:federated_compute_plan_builder failed to build
Note:
3.9.17
15379115139fca22e6c2ea1e047826ca2e3f3fa5
I commented out the Werror flag in the make files to get past the last issue, but now compilation fails with a linker error:
bazel-out/k8-opt/bin/fcp/client/cache/_objs/file_backed_resource_cache/file_backed_resource_cache.o:file_backed_resource_cache.cc:function fcp::client::cache::FileBackedResourceCache::CleanUp(std::optional, fcp::client::cache::CacheManifest&): error: undefined reference to 'std::filesystem::file_size(std::filesystem::__cxx11::path const&, std::error_code&)'
bazel-out/k8-opt/bin/fcp/client/opstats/_objs/pds_backed_opstats_db/pds_backed_opstats_db.o:pds_backed_opstats_db.cc:function fcp::client::opstats::PdsBackedOpStatsDb::Create(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, absl::lts_20220623::Duration, fcp::client::LogManager&, long): error: undefined reference to 'std::filesystem::__cxx11::path::has_filename() const'
(and many similar errors). looks like the c++ std lib need to be added to the linker options for file_backed_resource_cache.o and pds_backed_opstats_db.o?
build exits with errors:
ERROR: /root/.cache/bazel/_bazel_root/a2d1cfe108fb6ecf8584c66084ab27f5/external/boringssl/BUILD:161:11: Compiling src/crypto/evp/p_x25519.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 40 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
cc1: error: command line option '-std=c++17' is valid for C++/ObjC++ but not for C [-Werror]
cc1: all warnings being treated as errors
ERROR: /root/.cache/bazel/_bazel_root/a2d1cfe108fb6ecf8584c66084ab27f5/external/boringssl/BUILD:161:11: Compiling src/crypto/asn1/a_mbstr.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 40 arguments skipped)
looks like an easy fix.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.