mercari/grpc-federation

gRPC Federation generates a gRPC server by writing a custom option in Protocol Buffers

gRPC Federation

gRPC Federation auto-generates a BFF (Backend for Frontend) server, aggregating and returning results from microservices using the gRPC protocol, configured through Protocol Buffer options.

Motivation

Imagine a system with a backend of multiple microservices. Instead of the client directly communicating with each microservice, it's more efficient to use a dedicated service (BFF - Backend for Frontend) to aggregate information and send it back to the client. However, as the system grows, determining which team should handle the BFF service becomes unclear. This is where Federated Architecture, like GraphQL (Apollo) Federation, comes in.

With GraphQL Federation, each microservice has its own GraphQL server, and the BFF focuses on aggregating resources. However, integrating a new GraphQL server into a large system is challenging, especially in systems developed with the gRPC protocol.

To streamline, we're exploring the idea of automatically generating a BFF using the gRPC protocol with simple custom options. By leveraging existing Protocol Buffers schema for the BFF, we aim to keep using gRPC while drastically reducing maintenance costs.

Why Choose gRPC Federation?

1. Reduce the Boilerplate Implementation Required to Create a BFF

1.1. Automate Type Conversions for the Same Message Across Different Packages

Consider defining a BFF using a proto, assuming the BFF acts as a proxy calling multiple microservices and aggregating results. In this scenario, it's necessary to redefine the same messages to handle microservices responses on the BFF. Without defining these messages on the BFF, the BFF client must be aware of different packages, which is not ideal.

Redefining messages on the BFF results in a considerable number of type conversions for the same messages across different packages.

Using gRPC Federation removes the need for tedious type conversions. Just define custom options in the proto, and gRPC Federation will automatically handle these conversions.

1.2. Optimize gRPC Method Calls

When making multiple gRPC method calls, it is crucial to enhance performance by analyzing dependencies between method calls and processing requests in parallel whenever possible. However, identifying dependencies becomes more challenging as the number of method calls increases.

gRPC Federation simplifies this process by automatically analyzing dependencies and generating code to optimize request flows. It removes the need for manual consideration.

1.3. Simplify Retries and Timeouts on gRPC Method Calls

Setting timeouts or retry counts for gRPC method calls in dependent services can be complicated. However, gRPC Federation allows for a declarative approach to implementing retries and timeouts.

These are just a few examples of how gRPC Federation simplifies work. By creating a BFF with minimal proto definitions, you can significantly reduce the development cost of BFFs.

2. Explicitly Declare Dependencies on Services

Using gRPC Federation enables a clear understanding of the services the BFF depends on by reading the proto file. It uncovers a relationship between each microservice and BFF.

Furthermore, gRPC Federation offers functionality to extract dependencies as a Go library, enabling the extraction of service dependencies through static analysis of the proto file. This capability proves valuable for various types of analysis and automation.

Features

1. Code Generation Assistance Tools

gRPC Federation automatically generates a gRPC server by adding custom options to Protocol Buffers. It supports the protoc-gen-grpc-federation CLI, accessible through protoc. Various other tools are available to assist in code generation:

• protoc-gen-grpc-federation: protoc's plugin for gRPC Federation
• grpc-federation-linter: Linter for gRPC Federation
• grpc-federation-language-server: Language server for gRPC Federation
• grpc-federation-generator: Standalone code generation tool, which monitors proto changes and interactively generates codes

2. CEL Support for Complex Operations

gRPC Federation integrates CEL to enable support for more advanced operations on BFFs.

gRPC Federation CEL API References

3. Extensible System with WebAssembly

gRPC Federation features three extension points:

1. Code generation pipeline
2. Complex processes not expressible by Protocol Buffers
3. CEL API

We plan to make all three extension points available through WebAssembly in the future. Currently, the code generation pipeline and the CEL API are extensible.

3.1 Code Generation Pipeline

This feature allows you to run a custom auto-generated process using the results from gRPC Federation.

How to Run Your Own Code Generation Process

3.2 Complex Processes Not Expressible by Protocol Buffers

gRPC Federation uses a hybrid system, with logic in Go managing operations that can't be expressed in Protocol Buffers. We plan to add support for WebAssembly in the future.

How to Extend the API with the Go

3.3 CEL API

While gRPC Federation inherently supports various CEL APIs, users can use this functionality to use internal domain logic as a CEL API.

How to Extend the CEL API

Installation

Currently, there are three ways to use gRPC Federation:

1. Use buf generate
2. Use protoc-gen-grpc-federation
3. Use grpc-federation-generator

For detailed information on each method, please refer to Installation Documentation.

Usage

• Getting Started
• Examples

Documents

• Feature References
• CEL API References

Contribution

Please read the CLA carefully before submitting your contribution to Mercari. Under any circumstances, by submitting your contribution, you are deemed to accept and agree to be bound by the terms and conditions of the CLA.

https://www.mercari.com/cla/

License

Copyright 2023 Mercari, Inc.

Licensed under the MIT License.

grpc-federation's Issues

Write Engineering Blog

Write blog about gRPC Federation

Code Completion for CEL

Support code completion for CEL expression in language server

Plugin system for custom resolver

Make custom resolver run by WebAssembly as well.

FYI: https://github.com/mercari/grpc-federation?tab=readme-ov-file#32-the-complex-processes-that-cannot-be-expressed-by-protocol-buffers

protoc-gen-grpc-federation no longer works without options in v0.12.0

Describe the bug
When protoc-gen-grpc-federation is executed in v0.12.0 without options, it fails with the error:

$ buf generate
2024/02/08 14:08:34 grpc-federation: unexpected option format: 
Failure: plugin mercari-grpc-federation: exit status 1

This worked in previous versions of the plugin and was relied on by upstream unit tests in github.com/bufbuild/plugins.

** Related Components **

• protoc-gen-grpc-federation
• grpc-federation-linter
• grpc-federation-language-server
• grpc-federation-generator
• others

To Reproduce
Run the protoc-gen-grpc-federation plugin with a buf.gen.yaml file with no opt: settings for the plugin.

Expected behavior
The plugin should complete without errors (unless options are required now, in case we'll need to tweak how this plugin is invoked for testing.

Screenshots
n/a

Additional context
n/a

Support IntelliJ Plugin

Add support for the IntelliJ Plugin. Currently, only Visual Studio Code is supported.

Optimization of redundant CEL expressions

For example, when the following CEL is written, we want to verify if there is a more optimized expression, and if so, we want to prompt for it or automatically modify it.

Example

• foo is a boolean type

by: "foo == true ? true : false"

The above expression should be the following.

by: "foo"

Oneof is Reporting Compilation Errors though Using Examples

Describe the bug

Compiler still requires to specify grpc.federation.field even if specified in oneof fields.

Related Components

• protoc-gen-grpc-federation
• grpc-federation-linter
• grpc-federation-language-server
• others

To Reproduce
Steps to reproduce the behavior.

grpc-federation % cd _examples/10_oneof/proto
proto % protoc -I../proto_deps -I. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative --grpc-federation_out=. --grpc-federation_opt=paths=source_relative federation/federation.proto

2023/10/20 18:48:05 grpc-federation: value must be specified: value must be specified: ERROR: <input>:1:4: undefined field 'user'
 | sel.user
 | ...^: "user_a" field in "org.federation.UserSelection" message needs to specify "grpc.federation.field" option: "user_b" field in "org.federation.UserSelection" message needs to specify "grpc.federation.field" option
--grpc-federation_out: protoc-gen-grpc-federation: Plugin failed with status code 1.

(snippet from the example)

oneof user {
    User user_a = 1 [
      (grpc.federation.field).oneof = {
        expr: "m.value != $.value"
        messages {
          name: "ua"
          message: "User"
          args: { name: "user_id", string: "a" }
        }
        by: "ua"
      }
    ];
}

Expected behavior

Successful compilation without errors.

Additional context

I am trying to utilize grpc.federation.field.oneof to achieve conditional message calls based on the previous message call response. Described in Golang:

firstResponse := client.CallA()
switch firstResponse.GetRes() {
case "SUCCESS":
  secondResponse := client.CallB()
case "FAILED":
  // do nothing
}

I am not sure if using oneof is the correct way, but the expr in FieldOneof rule is the only place that I can find for condition judgements. Or is the current version does not support complicated logic?

Looking forward to your responses!

Buf dropped alpha support to Wasm plugins

Describe the bug

PR bufbuild/buf#2789 dropped support to buf Wasm plugins, so this is a warning that, when you will upgrade this component, you might need some extra work to remove the related code. For instance _examples/16_code_gen_plugin will likely need to be deleted

Related Components

• protoc-gen-grpc-federation

Colorize validator's output

Enable syntax highlight when errors are output by generator or linter

Register a protoc plugin for Buf ecosystem

Created an Issue to make protoc-gen-grpc-federation plugin available as a remote plugin for Buf.

ref: bufbuild/plugins#991

Once registered, I'll update the README

Add automated CI job to push protos to BSR

It appears baca78b updated the proto extension numbers, however, this is not yet reflected in https://buf.build/mercari/grpc-federation

Suggestion, add a push action or script to update this anytime proto changes land on main.

Remove singleflight

Improve the dependency tree building and remove singleflight.