carousell/aggproto

A code generator that enables exposing custom client facing gRPC APIs (Protobuf APIs + server side code) by consuming registered upstream services as defined by their Protobuf specifications.

AggProto [Pre-Release]

A code generator that enables exposing custom client facing gRPC APIs (Protobuf APIs + server side code) by consuming registered upstream services as defined by their Protobuf specifications.

The intent of this library is to generate APIs and server code that:

Given:

• A repository of internal APIs described in Protobuf format
• API Specs defining the behavior of the desired Aggregate endpoints

Generates:

• Protobuf APIs that describe the generated endpoint and that can be used by clients to invoke the endpoints
• Server side code that implements the said endpoint and internally invokes the internal apis in the order determined

General workflow

Step 1: Register upstream Protobuf

Registration of upstream protos, is required to identify the various message definitions, the sub-definitions and the operations defined by the proto. This step requires an empty output directory that will be used in later steps to create custom APIs

Usage:

protoc --registerproto_out=registry_path=<PATH_TO_REGISTRY>:. -I=<PATH_TO_PROTO_DIR>  <PATH_TO_PROTO_FILE>

For example, when registering one of the example proto files in this directory from the root of this project, run:

protoc --registerproto_out=registry_path=examples/registry:. -I=examples/protos examples/protos/listing/listing.proto

Step 2: Define a new API spec

An API Spec is defined as follows:

api:
  name: <name of the custom api>
  group: <a package or a team that this api is grouped with>
  version: <an int representing the current version of the spec>
meta:
  goPackage: <the go package name that you want all imports to use>
input:
  - <each line defines either an input/ an alias for the input/ or an output redirection>
output:
  - <each line defines either an output selection or an aliasing on the output>
operations:
  - <you can optionally whitelist operations>

For a more detailed description of the api spec look at the details below

Step 3: Sync the spec directory

This step processes all the specs found in a directory and generates the client facing protobuf files and the server side go module implementing the generated gRPC service.

Usage:

aggproto sync --api_specs_path <PATH_TO_SPECS> --registry_path <PATH_TO_REGISTRY> --go_out_path <PATH_TO_GENERATED_GO> --proto_out_path <PATH_TO_GENERATED_PROTO>

For example, when generating the example specs, run the following from the root of this repo:

aggproto sync --api_specs_path examples/specs --registry_path examples/registry --go_out_path examples/goOut --proto_out_path examples/protoOut

Examples

Check the examples directory for a detailed list of examples. In order to build the examples, run make examples from the root of the repo.

Basic example

The most basic example involves creating a static endpoint that has static values and takes nothing as input. When we add listing.title="iPhone", this creates a message called listing with a string field called title and the server code will always return the value "iPhone".

api:
  name: mock_listing
  group: static_primitives
  version: 1
meta:
  goPackage: github.com/carousell/aggproto/examples/goOut
output:
  - listing.title="iPhone"
  - listing.description="BNIB iPhone X"

A more complex example

This is a more real example where we take an id of a listing from the request, use that to fetch a listing. From the listing we then extract a `media_id` which is used to fetch a media object. Finally the listing response and media response are merged to create the output response desired.

Alias operation
The = operator is referred to in the documentation as an aliasing operation. In this example, we specify get_listing.id=listing.GetListingRequest.listing_id . This indicates that we expect one of the operations inferred from the output field selection to require listing.GetListingRequest.listing_id as an input parameter and that we are creating an alias for this field to instead read get_listing.id in the client facing proto. This is valid in both input and output field specifications.

Pipe operation
The <- operator is referred to as the pipe operation, wherein the output of one operation is used to form the input of another operation. In this case, we pass the media_id fetched from the listing response to the request of the get media call. This is valid in the input specifications only.

api:
  name: masked_listing_w_media
  group: inferred_input
  version: 1
meta:
  goPackage: github.com/carousell/aggproto/examples/goOut
input:
  - get_listing.id=listing.GetListingRequest.listing_id
  - media.GetMediaRequest.media_id<-listing.GetListingResponse.listing.media_id
output:
  - listing.title=listing.GetListingResponse.listing.title
  - listing.description=listing.GetListingResponse.listing.description
  - listing.photo_url=media.GetMediaResponse.media.photo_url

About

AggProto is created and maintained by Carousell. Help us improve this project! We'd love the feedback from you.

We're hiring! Find out more at http://careers.carousell.com/

License

AggProto is released under Apache License 2.0. See LICENSE for more details.

aggproto's Issues

Support for enums and remaining primitives

Support for static primitives when invoking upstream services

Some upstream services may need static values in the request object such as client-id, support for the same in the manner done for output fields.

regeneration should avoid changes

currently import order changes
order of input parameters change

Support for optional output fields

Should optional output fields be allowed? If so use the ! operator possibly to indicate that an output field is a optional (best-effort)

Support for comments sourced from upstream protos and spec file

Generated Protobuf file should have comments that are sourced from either upstream Message/Field/Operation or from the api spec.

add a generated marker

Support for one-of

In output fields oneof should be translated in an as-is manner
In input fields or request fields selecting a particular subset of one-of should be supported.