Giter Site home page Giter Site logo

peiwenhu / fledge-key-value-service Goto Github PK

View Code? Open in Web Editor NEW

This project forked from privacysandbox/protected-auction-key-value-service

0.0 0.0 0.0 6.39 MB

FLEDGE Key/Value Service

License: Apache License 2.0

Shell 7.07% JavaScript 0.98% C++ 69.73% Python 1.14% JSONiq 0.01% LLVM 0.08% Yacc 0.08% HCL 10.34% Dockerfile 0.38% Starlark 10.17% jq 0.01% Go 0.02%

fledge-key-value-service's Introduction

Important

The main branch hosts live code with latest changes. It is unstable and is used for development. It is suitable for contribution and inspection of the latest code. The release-* branches are stable releases that can be used to build and deploy the system.


FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the blog post.


Privacy Sandbox Logo FLEDGE Key/Value service

Background

FLEDGE API is a proposal to serve remarketing and other custom-audience ads without third-party cookies. FLEDGE executes the ad auction between the buyers (DSP) and the sellers (SSP) locally, and receives real-time signals from the FLEDGE K/V servers. To learn more about

When the auction is executed, separate FLEDGE K/V servers are queried for the buyers and sellers. When a buyer is making a bid, the DSP K/V server can be queried to receive real-time information to help determine the bid. To help the seller pick an auction winner, the SSP K/V server can be queried to receive any information about the creative to help score the ad.

State of the project

The current codebase represents the initial implementation and setup of the Key/Value server. It can be integrated with Chrome and Android with the Privacy Sandbox unified origin trial and Privacy Sandbox on Android Developer Preview. Our goal is to present the foundation of the project in a publicly visible way for early feedback. This feedback will help us shape the future versions.

The implementation, and in particular the APIs, are in rapid development and may change as new versions are released. The query API conforms to the API explainer. At the moment, to load data, instead of calling the mutation API, you would place the data as files into a location that can be directly read by the server. See more details in the data loading guide.

Currently, this service can be deployed to 1 region of your choice. Multi-region configuration is up to the service owner to configure.

Current features

Build and deployment

  • Source code is available on Github
  • Releases are done on a regular basis
  • Binaries can be built from source code
    • C++ binary
    • [AWS & GCP] Docker container image
    • [AWS]: Nitro EIF
    • [AWS]: Reference AMI
    • Other tools
  • Server can run as a standalone local process for testing without any cloud dependency or TEE-related functionality
  • Server can be deployed to AWS Nitro enclave
  • Server can be deployed to GCP Confidential Space
  • Reference terraform available for a clean and comprehensive deployment to AWS or GCP
    • Clean: assumes the cloud environment has no preset infrastructure
    • Comprehensive: deploys all dependencies and some recommended (but not necessarily required) configuration
    • Many server behaviors can be configured by parameter flags without having to rebuild

Data loading

  • Server loads key/value data from cloud file storage
  • Server loads key/value data from cloud pub/sub services
  • Server loads data into an in-RAM representation for query serving
  • Server continuously monitors for new data and incrementally updates ("delta files") the in-RAM representation
  • Support independent data ingestion pipelining by monitoring directories in cloud file storage independently
  • Supports Flatbuffers as the data event format
  • Supports Avro and Riegeli as the data file format
  • Supports snapshot files for faster server start up
  • Users can perform compactions of delta files into snapshot files in an offline path

Read request processing

  • Support Protected Audience Key Value Server query spec: can be used as a BYOS server to serve requests from Chrome
  • Support simple key value lookups for queries
  • Users can write "user defined functions" to execute custom logic to process queries
  • User defined functions can be written in JavaScript or WASM
  • User defined functions can call "GetValues" to look up key value from the dataset

Advanced features

  • Set-as-a-value is supported
    • A key "value" pair in the dataset can be a key and a set of values
  • UDF can call "RunQuery" API to run set operations on sets (intersection, union, difference)
  • For GCP, Terraform supports deploying into an existing VPC, such as used by the Bidding and Auction services Non-prod Server logs are persisted after server shutdown
  • Data can be sharded and different servers may load and serve different shards (subset) of the dataset.
  • Sharding supports data locality, where the operator specifies "sharding key" for key value pairs so different key value pairs can have the same sharding key.

Breaking changes

While we make efforts to not introduce breaking changes, we expect that to happen occasionally.

The release version follows the [major change]-[minor change]-[patch] scheme. All 0.x.x versions may contain breaking changes without notice. Refer to the release changelog for the details of the breaking changes.

Once the codebase is in a more stable state that is version 1.0.0, we will establish additional channels for announcing breaking changes and major version will always be incremented for breaking changes.

Key documents

Contribution

Contributions are welcome, and we will publish more detailed guidelines soon. In the meantime, if you are interested, open a new Issue in the GitHub repository.

Feedback

The FLEDGE K/V feature set, API and system design are under active discussion and subject to change in the future. If you try this API and have feedback, we'd love to hear it:

Announcements

General announcements

General announcements will be made on the FLEDGE mailing list on chromium.org and the Privacy Sandbox progress update page on the Android developer site, if needed, an accompanying article will be published on the Chrome Developers blog and Android Developers blog.


OpenSSF Scorecard

fledge-key-value-service's People

Contributors

3rwww1 avatar a-shruti avatar bjschnei avatar dave-garred avatar emmafu2022 avatar fhoering avatar formgit avatar kelvintatendagorekore avatar kevinbnaughton avatar kevinkiklee avatar lusayaa avatar lx3-g avatar michaelkleber avatar p-j-l avatar peiwenhu avatar pmeric avatar roscoejp-ibm avatar sevaorlov avatar yangwang-dev avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.