google / gnostic-grpc Goto Github PK

A gnostic plugin that converts an OpenAPI API description into a description of a gRPC service that can be used to implement that API using gRPC-JSON Transcoding.

License: Apache License 2.0

Go 99.70% Shell 0.30%

gnostic-grpc's Introduction

gnostic gRPC plugin

This plugin has two functionalities:

Conversion of an OpenAPI v3.0 API description into a description of a gRPC service that can be used to implement that API using gRPC-JSON Transcoding. gRPC services are described using the Protocol Buffers language. Continue reading this README for more information.
Scanning OpenAPI v3 documents for equivalent gRPC service incompatibilities. For simple per file incompatibility scanning see the README file in the directory. For analysis over a set of OpenAPI documents see the README file in the directory.

OpenAPI descriptions are read and processed with gnostic, and this tool runs as a gnostic plugin.

High level overview:

Under the hood the plugin first creates a FileDescriptorSet (bookststore.descr) from the input data. Then protoreflect is used to print the output file.

How to use:

Install gnostic and the plugin before Go 1.17:

go get -u github.com/google/gnostic
go get -u github.com/google/gnostic-grpc

with Go >= 1.17:

go install github.com/google/gnostic@latest
go install github.com/google/gnostic-grpc@latest

Run gnostic with the plugin:

gnostic --grpc-out=examples/bookstore examples/bookstore/bookstore.yaml

This generates the gRPC service definition examples/bookstore/bookstore.proto.

End-to-end example

This directory contains a tutorial on how to build a gRPC service that implements an OpenAPI specification.

What conversions are currently supported?

Given an OpenAPI object following fields will be represented inside a .proto file:

Object	Fields	Supported
OpenAPI object
	openapi	No
	info	No
	servers	No
	paths	Yes
	components	Yes
	security	No
	tags	No
	externalDocs	No

Disclaimer

This is prerelease software and work in progress. Feedback and contributions are welcome, but we currently make no guarantees of function or stability.

Requirements

gnostic-grpc can be run in any environment that supports Go and the Google Protocol Buffer Compiler.

Copyright

License

Released under the Apache 2.0 license.

gnostic-grpc's People

Contributors

Stargazers

Watchers

Forkers

renovate-bot dhiemaz shubhsherl franperezsevilla timburks idoqo lorenzhw winwisly100 krishna-birla josephalvear valery-barysok fideltak rodherz ronnylt isabella232 ivanferianda spacedsweden b25 rizalgowandy abhisheknagvanshi carmo-evan cryptopossum newbie67 penthaapatel markelog will232 morphar zchee chetanugale karim42benhammou sayanta66 42crunch unionj-cloud andrewstucki chepchaf raw34 manikishore-s jxmallett zukobronja deependra-nunet msidana rkamudhan pmtabe1 brainlab-vied thlib emcfarlane ghas-results ilaleksin etam-pro

gnostic-grpc's Issues

Add GitHub action to run tests in CI pipeline

Add a GitHub action to run tests on PR and merge into main

CC: #89

Warn user in case an OpenAPI description with inline schema is provided as input

Regarding Tim's comment:

Since schema references often correspond to named structs, I'm inclined to force all inlined schemas to be converted to references, either with a linter rule or a transformation.

As long as this linter rule or transformation does not exist we should display a warning that the output of gnostic-grpc might be faulty in case the user provides an OpenAPI description with inline schemas.

Example input API with inline schema:

paths:
  /inlineSchema:
    get:
      operationId: inlineSchema
      responses:
        200:
          description: succes
          content:
            application/json:
              schema:
                type: integer
                format: int32

Expected console output after running gnostic --grpc-out...:

level:WARNING  code: "" text: "An inline schema is provided. The output of gnostic-grpc might be incorrect. Consider using a reference for your schema" keys:"paths" keys:"/inlineSchema" keys:"get" keys:"responses" keys:"200" keys:"content" keys:"application/json" keys:"schema"

Example input API with reference schema:

paths:
  /referenceSchema:
    get:
      operationId: referenceSchema
      responses:
        200:
          description: success
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Person'
components:
  schemas:
    Person:
      type: object
      properties:
        age:
          type: integer
          format: int64
        name:
          type: string
          example: Peter

Expected console output:
No warning regarding inline schema

Generator ignores additionalProperties: true

Hey there,
I have this openapi spec component:

    Response:
      properties:
        uid:
          type: string
        data:
          type: object
          additionalProperties: true

oapi generator would translate data field into map[string]interface{}

gnostic-grpc can not produce ad working proto, here's the output after I converted to binary pb using gnostic:

message Data {
}

Thanks

application_json should not be exposed in API types

In the end-to-end example, the generated bookstore.proto contains the following messages:

message ListShelvesResponse {
  repeated Shelf shelves = 1;
}

message ListShelvesOK {
  ListShelvesResponse application_json = 1;
}

message ListShelvesResponses {
  ListShelvesOK ok = 1;
}

As a result, the generated API includes messages with a field named "application_json". For example, to create a shelf, we suggest using this curl command:

curl -X POST \
  http://localhost:51051/shelves \
  -H 'Content-Type: application/json' \
  -d '{
  "application_json": {
    "name": "Books I need to read",
    "theme": "Non-fiction"
    }
}'

After looking more carefully at the OpenAPI Spec, I believe this is incorrect. Specifically the description of Response Objects says that content is "a map containing descriptions of potential response payloads."

Based on that, I think a more appropriate call would be this:

curl -X POST \
  http://localhost:51051/shelves \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Books I need to read",
  "theme": "Non-fiction"
}'

I recently implemented an alternate bookstore API that uses the Google Cloud Datastore API to store books and shelves (see server.go for the Datastore-based implementation). This required some small changes to bookstore.yaml but did not significantly change the messages produced by gnostic-grpc - they still include application_json fields. But here I also hand-wrote a bookstore.proto that seems to me to produce a more appropriate REST API for the associated spec.

Do we agree that this is a problem? If so, what changes are needed to produce the hand-written bookstore.proto instead of the current gnostic-grpc output?

Unnecessary imports of google/protobuf/empty.proto generate warnings.

Running generated .protos through protoc I often get the following warning:

tiny.proto:7:1: warning: Import google/protobuf/empty.proto but not used.

Investigating, I find that this occurs for simple APIs like the following:

# Specification for the Foo Service API
swagger: '2.0'
info:
  title: Foo Service API
  description: Foo Service API
  version: '0.0.1'
host: api.foo.com
schemes:
  - https
  - http
produces:
  - application/json
basePath: /
paths:

  /{foo-id}:
    get:
      tags:
        - Foo
      description: Retrieve metadata about the foo.
      parameters:
        - in: path
          name: foo-id
          type: string
          required: true
      responses:
        '200':
          description: Description of the foo
          schema:
            $ref: '#/definitions/Foo'

definitions:
  Foo:
    type: object
    required:
      - name
    properties:
      name:
        description: Name of the foo. Required in order to create the foo.
        type: string

Here the generated .proto file doesn't actually use google.protobuf.Empty. It would be a nice improvement to avoid adding this import when it isn't needed.

Bind Parameters to Request suffix

Currently parameters are converted to an object with suffix Parameters.
Proposing if we can change this behaviour to

If its 1 parameter, pass it directly to rpc method args
Or add a Request suffix. Its seems more natural to understand.

"/pay/v1/transactions/{transactionId}/callback/pgs/payu": {
      "post": {
        "tags": [
          "callback-controller"
        ],
        "operationId": "payU",
        "parameters": [
          {
            "name": "transactionId",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "default response"
          }
        }
      }
    }

creates

 rpc PayU ( PayUParameters ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { post:"/pay/v1/transactions/{transactionId}/callback/pgs/payu"  };
  }

Excepcted

 rpc PayU ( string ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { post:"/pay/v1/transactions/{transactionId}/callback/pgs/payu"  };
  }

 rpc PayU ( PayURequest ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { post:"/pay/v1/transactions/{transactionId}/callback/pgs/payu"  };
  }

Getting the documentation from OAS files in to the proto

Hey

Is it possible to get the description and summary text into the generated proto file?

Use GitHub actions instead of travis

Similar to gnostic GitHub actions should be used.

How to set data type float

I'm not sure the way to set float as data type.
My OAS3 yaml is defined some prameters as float.

        location:
          type: object
          properties:
            longitude:
              type: number
              format: float
            latitude:
              type: number
              format: float
              example: 139.599194

However when creating proto file by gnostic, they are turned to int.

message Location {
  int32 longitude = 1;

  int32 latitude = 2;
}

Please give some advices...

Plugin fails when there's a name collision between a message and a service name.

gnostic-grpc exits with the following error:

$ gnostic foo.yaml --grpc-out=.
panic: interface conversion: desc.Descriptor is *desc.ServiceDescriptor, not *desc.MessageDescriptor

goroutine 1 [running]:
github.com/jhump/protoreflect/desc.(*FieldDescriptor).resolve(0xc000173980, 0xc0000c3860, 0x4, 0x8, 0xc00015f7d0, 0x2, 0x2, 0x2, 0x0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/descriptor.go:552 +0x749
github.com/jhump/protoreflect/desc.(*MessageDescriptor).resolve(0xc00017a2d0, 0xc0000c3860, 0x3, 0x8, 0xc00015f7d0, 0x2, 0x2, 0x0, 0x0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/descriptor.go:326 +0x498
github.com/jhump/protoreflect/desc.createFileDescriptor(0xc000134000, 0xc00000f800, 0x3, 0x3, 0x0, 0x2, 0xc0000c3b88, 0xc000116cf0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:88 +0x100c
github.com/jhump/protoreflect/desc.createFromSet(0xc000022d30, 0x9, 0x0, 0xc00015f750, 0x1, 0x1, 0xc0000c3b88, 0xc000116cf0, 0xc0000f8380, 0x0, ...)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:225 +0x338
github.com/jhump/protoreflect/desc.createFileDescriptors(0xc00000f0c0, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:131 +0x1d3
github.com/jhump/protoreflect/desc.createFileDescriptorsFromSet(0xc000032680, 0x0, 0x0, 0x0, 0x0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:193 +0xa7
github.com/jhump/protoreflect/desc.createFileDescriptorFromSet(0xc000032680, 0x0, 0x0, 0x0, 0x0)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:169 +0x39
github.com/jhump/protoreflect/desc.CreateFileDescriptorFromSet(...)
        /usr/local/google/home/timburks/go/src/github.com/jhump/protoreflect/desc/convert.go:165
github.com/googleapis/gnostic-grpc/generator.(*Renderer).RenderProto(0xc000032640, 0xc000032680, 0xc000022d20, 0x9, 0xc000022d20, 0xc000022d23, 0xc0000c3e98)
        /usr/local/google/home/timburks/go/src/github.com/googleapis/gnostic-grpc/generator/renderer.go:82 +0x49
github.com/googleapis/gnostic-grpc/generator.(*Renderer).Render(0xc000032640, 0xc000106bd0, 0xc000022d20, 0x9, 0x6, 0xc000022d20)
        /usr/local/google/home/timburks/go/src/github.com/googleapis/gnostic-grpc/generator/renderer.go:62 +0x96
github.com/googleapis/gnostic-grpc/generator.RunProtoGenerator()
        /usr/local/google/home/timburks/go/src/github.com/googleapis/gnostic-grpc/generator/main.go:66 +0x47c
main.main()
        /usr/local/google/home/timburks/go/src/github.com/googleapis/gnostic-grpc/main.go:22 +0x20
Errors reading foo.yaml
exit status 2

for the following foo.yaml input file:

# Specification for the Foo Service API
swagger: '2.0'
info:
  title: Foo Service API
  description: Foo Service API
  version: '0.0.1'
host: api.foo.com
schemes:
  - https
  - http
produces:
  - application/json
basePath: /
paths:

  /{foo-id}:
    parameters:
      - in: path
        name: foo-id
        type: string
        required: true
    get:
      tags:
        - Foo
      description: Retrieve metadata about the foo.
      responses:
        '200':
          description: Description of the foo
          schema:
            $ref: '#/definitions/Foo'

    delete:
      tags:
        - Foo
      description: Remove the foo.
      responses:
        '200':
          description: Description of the foo
          schema:
            $ref: '#/definitions/Foo'

definitions:
  Foo:
    type: object
    required:
      - name
    properties:
      self:
        description: A link to this foo.
        type: string
        format: uri
      kind:
        description: Should always be "Foo".
        type: string
      name:
        description: Name of the foo. Required in order to create the foo.
        type: string
      created_at:
        description: Timestamp in milliseconds since the Unix Epoch.
        type: integer
        format: int64
      lastmodified_at:
        description: Timestamp in milliseconds since the Unix Epoch.
        type: integer
        format: int64

Notably, if foo.yaml is renamed to bar.yaml, the plugin succeeds.

From the error message and the generated .proto files, I suspect that the problem is related to a message and service both having the name Foo (gnostic-grpc derives the service name from the input file name).

Consider including go_package option in generated protos

I believe that this issue by @egonz is requesting that the go_package option be included in generated proto files; since gnostic-grpc generates protos, we should probably address that here.

Adding badges information in the README.md

Hi, I have read the contribution guideline and have
the readme.md do not have the information about :

Go code version compatibility,
Project code quality report cards
Code coverage

Can we update the readme.md?

Getting Plugin error: cycle in imports while trying bookstore sample yaml file

I'm trying with gnostic-grpc to convert a yaml file to proto file. I tried with my own sample obtained the above error. So i tried with the bookstore.yaml file.

Command
./gnostic --grpc-out=service /Users/piraveena/Documents/test/swagger.yaml

Error logs

level:INFO code:"DOCUMENTFIELDS" text:"Field: 'servers' is not supported for the OpenAPI document with title: Bookstore" keys:"servers"
level:INFO code:"DOCUMENTFIELDS" text:"Field: 'security' is not supported for the OpenAPI document with title: Bookstore" keys:"security"
level:INFO code:"COMPONENTSFIELDS" text:"Field: 'securitySchemes' is not supported for the component" keys:"components" keys:"securitySchemes"
level:INFO code:"SCHEMAFIELDS" text:"Field: 'required' is not supported for the schema: book" keys:"components" keys:"schemas" keys:"book" keys:"required"
level:INFO code:"SCHEMAFIELDS" text:"Field: 'required' is not supported for the schema: listBooksResponse" keys:"components" keys:"schemas" keys:"listBooksResponse" keys:"required"
level:INFO code:"SCHEMAFIELDS" text:"Field: 'required' is not supported for the schema: shelf" keys:"components" keys:"schemas" keys:"shelf" keys:"required"
level:INFO code:"SCHEMAFIELDS" text:"Field: 'required' is not supported for the schema: error" keys:"components" keys:"schemas" keys:"error" keys:"required"
level:INFO code:"REQUESTBODYFIELDS" text:"Field: 'required' is not supported for the request: createShelf" keys:"paths" keys:"/shelves" keys:"post" keys:"requestBody" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}" keys:"get" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}" keys:"delete" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}/books" keys:"get" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}/books" keys:"post" keys:"parameters" keys:"required"
level:INFO code:"REQUESTBODYFIELDS" text:"Field: 'required' is not supported for the request: createBook" keys:"paths" keys:"/shelves/{shelf}/books" keys:"post" keys:"requestBody" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}/books/{book}" keys:"get" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: book" keys:"paths" keys:"/shelves/{shelf}/books/{book}" keys:"get" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: shelf" keys:"paths" keys:"/shelves/{shelf}/books/{book}" keys:"delete" keys:"parameters" keys:"required"
level:INFO code:"PARAMETERFIELDS" text:"Field: 'required' is not supported for parameter: book" keys:"paths" keys:"/shelves/{shelf}/books/{book}" keys:"delete" keys:"parameters" keys:"required"
Errors reading /Users/piraveena/Documents/test/swagger.yaml
Plugin error: [cycle in imports: swagger.proto -> swagger.proto]%

Go version: go version go1.19 darwin/amd64
And I'm using latest gnostic-grpc

Any idea why I'm getting above errors

Error parsing go.mod. Unable to install gnostic and the plugin.

The go.mod file lists github.com/googleapis/gnostic as the current module’s module path, whereas it should be updated to github.com/google/gnostic instead it seems. While installing gnostic and the plugin, the following error appears and the installation is aborted.

Steps to reproduce the error:
On running the command

$  go get -u github.com/google/gnostic

the error produced is:

go get: github.com/google/[email protected] updating to
	github.com/google/[email protected]: parsing go.mod:
	module declares its path as: github.com/googleapis/gnostic
	        but was required as: github.com/google/gnostic

On running the command

$  go get -u github.com/google/gnostic-grpc

The error produced is:

go get: github.com/google/gnostic-grpc@none updating to
	github.com/google/[email protected]: parsing go.mod:
	module declares its path as: github.com/googleapis/gnostic-grpc
	        but was required as: github.com/google/gnostic-grpc

Enable master branch protection

      This repository does not seem to have master branch
      protection enabled, at least in the way I'm expecting.
      I was hoping for:

      - master branch protection
      - requiring at least one code reviewer
      - requiring at least two status checks
      - enforcing rules for admins

      Please turn it on!

Adding service address and auth info

I think there are some things that will need to be added (service address, auth info)

@timburks you mentioned here that we should add additional information to the generated output.

Service address:
An example input file might look like this:
input.yaml: (OpenAPI spec)

server:
  - url: https://development.gigantic-server.com/v1
    description: Development server
  - url: https://staging.gigantic-server.com/v1
    description: Staging server
  - url: https://api.gigantic-server.com/v1
    description: Production server

An example output file generated by gnostic-grpc might look like this:
output.proto: (Proto Buf file)

service SomeService {
    // Use the first one if multiple addresses are given
    option (google.api.default_host) = "development.gigantic-server.com/v1";
    // Other code...
}

We can use the default_host service option for that.

Auth info:
Essentially, we want the information of a security scheme object reflected inside the generated proto file.
How should this object be represented inside the generated output file?

As far as I understand, for Google Cloud, one could define a service configuration where additional information (i.e.: authentication) can be provided. This config maps to the service.proto where also the mentioned auth.proto is defined.

We see that for HTTP rules it does not matter whether we define them inside the service config or inside the proto file, however this does not seem to be possible for other configurations.

@timburks and @noahdietz additional input would be helpful.

Tests fail when updating version of gnostic to v0.5.5

All tests fail when the v0.5.5 for gnostic is used.

Broken proto on fields with numbers or hyphens

When parsing an enum that has fields which start in a number it generates a broken proto file.

enum:
  - 5FOO_BAR

translates to

enum Value {
    5FOO_BAR = 0;
}

which is not a valid field.

Something like:

enum:
  - FOO-BAR

Get parsed as

enum Value {
  FOO-BAR = 0;
}

message names that starts with numbers have the same problem:

properties:
  5FooBar:
    ...

message 5FooBar {
  ...

Possible fix

Escape some characters to underscores or add undescore at the beggining of enum fields to make them valid

Duplicate message descriptors generated for request schemas whose names end with "Request"

Steps to reproduce:

Add the following to generator/testfiles/requestBodies.yaml:

/testRequestBodySchemaReference:
    post:
      operationId: testRequestBodySchemaReference
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TestRequestBodySchemaReferenceRequest'
      responses:
        "201":
          description: OK
components:
  schemas:
    TestRequestBodySchemaReferenceRequest:
      type: object
      properties:
        person:
          $ref: '#/components/schemas/Person'

Execute TestFileDescriptorGeneratorRequestBodies to generate an updated .proto file
Observe the following duplication:

message TestRequestBodySchemaReferenceRequest {
  Person person = 1;
}

message Person {
  int64 id = 1;

  int64 age = 2;

  string name = 3;

  repeated string photo_urls = 4;
}

//TestRequestBodySchemaReferenceParameters holds parameters to TestRequestBodySchemaReference
message TestRequestBodySchemaReferenceRequest {
  TestRequestBodySchemaReferenceRequest test_request_body_schema_reference_request = 1;
}

service Requestbodies {
  rpc TestRequestBody ( TestRequestBodyRequest ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { get:"/testRequestBody" body:"person"  };
  }

  rpc TestRequestBodyReference ( TestRequestBodyReferenceRequest ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { get:"/testRequestBodyReference" body:"person"  };
  }

  rpc TestRequestBodySchemaReference ( TestRequestBodySchemaReferenceRequest ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { post:"/testRequestBodySchemaReference" body:"test_request_body_schema_reference_request"  };
  }
}

Add test case for OneOf, AnyOf, and AllOf

The file other.yaml should be extended:

test AnyOf, OneOf, and AllOf
test additionalProperties with an object. E.g.:

  /testAdditionalPropertiesObject:
    get:
      operationId: testAdditionalPropertiesObject
      responses:
        '200':
          description: successful operation
          content:
            application/json:
              schema:
                type: object
                additionalProperties:
                  properties: 
                    id:
                      type: number

Install fails while parsing go.mod

Observed behavior

Following the install instructions results in an error while parsing go.mod

Steps to replicate

run go install github.com/google/gnostic@latest
run go install github.com/google/gnostic-grpc@latest

result

go install github.com/google/gnostic-grpc@latest
go install: github.com/google/gnostic-grpc@latest: github.com/google/[email protected]: parsing go.mod:
	module declares its path as: github.com/googleapis/gnostic-grpc
	        but was required as: github.com/google/gnostic-grpc

Possible resolution

I was able to install gnostic-grpc using go install github.com/googleapis/gnostic-grpc@latest

Minor problems in warning messages

There are a few small issues in the warning messages generated for the examples.

For reference, here are the results for the bookstore example:

~/Desktop/gnostic-grpc/examples/bookstore$ gnostic --grpc-out=. bookstore.yaml 
2019/08/16 15:38:10 Response has no content: deleteShelvesDefault
2019/08/16 15:38:10 Response has no content: deleteShelfDefault
2019/08/16 15:38:10 Response has no content: deleteBookDefault
level:WARNING code:"DOCUMENTFIELDS" text:"Fields: Servers, Security are not supported for Document with title: Bookstore" keys:"Document" 
level:WARNING code:"COMPONENTSFIELDS" text:"Fields: SecuritySchemes are not supported for the component" keys:"Component" 
level:WARNING code:"SCHEMAFIELDS" text:"Fields: Required are not supported for the schema: book" keys:"book" keys:"Schema" 
level:WARNING code:"SCHEMAFIELDS" text:"Fields: Required are not supported for the schema: listBooksResponse" keys:"listBooksResponse" keys:"Schema" 
level:WARNING code:"SCHEMAFIELDS" text:"Fields: Required are not supported for the schema: shelf" keys:"shelf" keys:"Schema" 
level:WARNING code:"SCHEMAFIELDS" text:"Fields: Required are not supported for the schema: error" keys:"error" keys:"Schema" 
level:WARNING code:"REQUESTBODYFIELDS" text:"Fields: Required are not supported for the request: createShelf" keys:"RequestBody" keys:"createShelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"REQUESTBODYFIELDS" text:"Fields: Required are not supported for the request: createBook" keys:"RequestBody" keys:"createBook" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: book" keys:"Parameter" keys:"book" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: shelf" keys:"Parameter" keys:"shelf" 
level:WARNING code:"PARAMATERFIELDS" text:"Fields: Required are not supported for parameter: book" keys:"Parameter" keys:"book"

In the "PARAMATERFIELDS" code, "parameter" is misspelled. Please change this to "PARAMETERFIELDS".
The keys field is intended to hold a keypath to the related item in the OpenAPI document, beginning at the root. This is intended to support IDEs or other tools that could navigate directly to the related item, so it should be complete and exact.

a. The first message (with code "DOCUMENTFIELDS") should either be on the root (with no keys) or split into three messages, one for each key, with each key being the only element in the path for the corresponding message.

b. For the second message (with code "COMPONENTSFIELD"), the keys should be ["components", "securitySchemes"].

c. For the third message ("SCHEMAFIELDS"), the keys should be ["components", "schemas", "book", "required"].

Other messages should be treated similarly, with complete paths of keys that correspond (case-sensitively) to keys in the OpenAPI document. For more examples, see this sample linter in the main gnostic project.
[less-important] We should consider how to replace the log.Printf() calls in the surface builder with messages. This is an issue with gnostic and not gnostic-grpc.

File included an unresolvable reference

I'm trying to convert the following Json file: https://github.com/oracle/hospitality-api-docs/blob/main/rest-api-specs/property/lov.json

Using this command:

gnostic --grpc-out=api ../hospitality-api-docs/rest-api-specs/property/lov.json

and i'm getting this error:

Errors reading ../hospitality-api-docs/rest-api-specs/property/lov.json
Plugin error: [file "lov.proto" included an unresolvable reference to "GetChannelParametersLOVResponses"]

If I then try the --resolve-refs option like this:

gnostic --grpc-out=api --resolve-refs ../hospitality-api-docs/rest-api-specs/property/lov.json

I get another slightly different error:

Errors reading ../hospitality-api-docs/rest-api-specs/property/lov.json
Plugin error: [file "lov.proto" included an unresolvable reference to "lov.GetChannelParametersLOVOK"]

There's another file that has problems too:
https://github.com/oracle/hospitality-api-docs/blob/main/rest-api-specs/property/fofcfg.json

but the rest convert correctly. Any ideas on what is happening here as the Json files seem to be valid?

Thanks.

OpenAPI enum-definitions are not supported in Proto-File generation

Why are enums not generated in Proto-File?

Upgrade golang protobuf api to v2

Hey,

gnostic-grpc has a few references to 'github.com/golang/protobuf' (api v1) in the generator package and in the end-to-end test example.

API v1 is deprecated and it is recommended to use 'google.golang.org/protobuf/proto' (api v2) instead.

What do you think about switching to protobuf api v2?

I would be glad to help with this upgrade so I have prepared PR #97 with the required changes.

Please review it and share your thoughts about these changes 😉

Methods without 'operationId' cause errors

I generated a .proto from one OpneApi file I had but the result was inconsistent regarding the responses.
I then discovered that the field operationId is required for each method to generate the correct .proto. Here is the bookstore.proto generated from the bookstore.yaml without operationId (response messages and rpc returns are wrong, but there was no warning in the generation):

syntax = "proto3";

package bookstore;

import "google/api/annotations.proto";

import "google/protobuf/empty.proto";

import "google/protobuf/descriptor.proto";

message Book {
  string author = 1;

  string name = 2;

  string title = 3;
}

message ListBooksResponse {
  repeated Book books = 1;
}

message ListShelvesResponse {
  repeated Shelf shelves = 1;
}

message Shelf {
  string name = 1;

  string theme = 2;
}

message Error {
  int32 code = 1;

  string message = 2;
}

message OK {
  ListShelvesResponse listshelvesresponse = 1;
}

message RequestBody {
  Shelf shelf = 1;
}

message POSTShelvesParameters {
  Book book = 1;
}

message OK {
  Shelf shelf = 1;
}

message GETShelvesShelfParameters {
  int64 shelf = 1;
}

message OK {
  Shelf shelf = 1;
}

message Default {
  Error error = 1;
}

message DELETEShelvesShelfParameters {
  int64 shelf = 1;
}

message GETShelvesShelfBooksParameters {
  int64 shelf = 1;
}

message OK {
  ListBooksResponse listbooksresponse = 1;
}

message Default {
  Error error = 1;
}

message POSTShelvesShelfBooksParameters {
  int64 shelf = 1;

  Book book = 2;
}

message OK {
  Book book = 1;
}

message Default {
  Error error = 1;
}

message GETShelvesShelfBooksBookParameters {
  int64 shelf = 1;

  int64 book = 2;
}

message DELETEShelvesShelfBooksBookParameters {
  int64 shelf = 1;

  int64 book = 2;
}

service Bookstore {
  rpc GET_shelves ( google.protobuf.Empty ) returns ( Book ) {
    option (google.api.http) = { get:"/shelves"  };
  }

  rpc POST_shelves ( POSTShelvesParameters ) returns ( Book ) {
    option (google.api.http) = { post:"/shelves" body:"book"  };
  }

  rpc DELETE_shelves ( google.protobuf.Empty ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { delete:"/shelves"  };
  }

  rpc GET_shelves_shelf ( GETShelvesShelfParameters ) returns ( Book ) {
    option (google.api.http) = { get:"/shelves/{shelf}"  };
  }

  rpc DELETE_shelves_shelf ( DELETEShelvesShelfParameters ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { delete:"/shelves/{shelf}"  };
  }

  rpc GET_shelves_shelf_books ( GETShelvesShelfBooksParameters ) returns ( Book ) {
    option (google.api.http) = { get:"/shelves/{shelf}/books"  };
  }

  rpc POST_shelves_shelf_books ( POSTShelvesShelfBooksParameters ) returns ( Book ) {
    option (google.api.http) = { post:"/shelves/{shelf}/books" body:"book"  };
  }

  rpc GET_shelves_shelf_books_book ( GETShelvesShelfBooksBookParameters ) returns ( Book ) {
    option (google.api.http) = { get:"/shelves/{shelf}/books/{book}"  };
  }

  rpc DELETE_shelves_shelf_books_book ( DELETEShelvesShelfBooksBookParameters ) returns ( google.protobuf.Empty ) {
    option (google.api.http) = { delete:"/shelves/{shelf}/books/{book}"  };
  }
}

Don't know if this is a bug or not, but adding some documentation or warnings maybe could help someone else.

Main README link to envoy is broken

implement that API using [gRPC-JSON Transcoding](https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_json_transcoder_filter

Invalid proto message gets generated when using anyOf, oneOf, allOf

Given schema:

components:
  schemas:
    TestOneOfApiResponse:
      oneOf:
        - $ref: "#/components/schemas/Person"
        - $ref: "#/components/schemas/Order"
    Person:
      properties:
        id:
          type: integer
       name:
          type: string
    Order:
      properties:
        id:
          type: integer

currently results in the following proto file:

message TestOneOfApiResponse {
  int32 id = 1;

  string name = 2;

  int32 id = 3;
}

TestOneOfApiResponse is not a valid message, because id exists twice.

panic when a $ref refers to a local file

So I have my various services split up into their own yaml files and a common.yaml file they they will refer to

common.yaml

openapi: 3.0.3

info:
  version: "v1"
  title: "Common Components"
  description: Common Components used by many endpoints

paths: {}
components:
  ##########################
  # Common Parameters
  ##########################
  parameters:

    InteractionIdParam:
      in: path
      name: interaction_id
      required: true
      schema:
        $ref: "common.yaml#/components/schemas/InteractionId"
  ##########################
  # Common Data Models
  ##########################
  schemas:
    InteractionId:
      description: >-
        The primary Identifier for the call.
      type: string
      format: uuid
      example: "fdb6816a-a281-412f-ae47-a40fc1033e13"

test.yaml

openapi: 3.0.3

info:
  version: "v1"
  title: "Test Service"
  description: The Test Service

paths:
  /v1/test/{interaction_id}/results:
    get:
      summary: Test Service
      description: |-
        Render the test results
      parameters:
        - $ref: "common.yaml#/components/parameters/InteractionIdParam"
      responses:
        "200":
          $ref: "test.yaml#/components/responses/GetTestSuccess"

components:
  schemas:
    TestGetResponse:
      type: object
      properties:
        data:
          type: string

  responses:
    GetTestSuccess:
      description: OK
      content:
        application/json:
          schema:
            $ref: "test.yaml#/components/schemas/TestGetResponse"

so when I run gnostic --resolve-refs --grpc-out=generated/proto/ common.yaml it works fine
when I run gnostic --resolve-refs --grpc-out=generated/proto/ test.yaml I get

gnostic --resolve-refs --grpc-out=generated/proto/ openapi/test.yaml

2023/10/26 12:19:29 Not able to find parameter information for: _ref:"common.yaml#/components/parameters/InteractionIdParam"
level:WARNING  code:"OPERATION"  text:"One of your operations does not have an 'operationId'. gnostic-grpc might produce an incorrect output file."  keys:"paths"  keys:"/v1/test/{interaction_id}/results"  keys:"get"
Errors reading openapi/test.yaml
Plugin error: [file "test.proto" included an unresolvable reference to "test.InteractionIdParam"]

running gnostic --resolve-refs --pb_out=generated/pb/ openapi/test.yaml works fine and when I run gnostic-grpc not as a plugin I get

gnostic-grpc -input ./generated/pb/openapi/test.pb
2023/10/26 12:25:24 Not able to find parameter information for: _ref:"common.yaml#/components/parameters/InteractionIdParam"
2023/10/26 12:25:24 Plugin error: [file "test.proto" included an unresolvable reference to "test.InteractionIdParam"]

when I run the full test.yaml file I get

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x38 pc=0x105050384]

goroutine 1 [running]:
github.com/google/gnostic/surface.(*OpenAPI3Builder).buildFromResponse(0x8?, {0x140002349e0, 0xf}, 0x1f?)
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:308 +0x114
github.com/google/gnostic/surface.(*OpenAPI3Builder).buildFromResponseOrRef(0x14000169ac8?, {0x140002349e0, 0xf}, 0x14000036ea0?)
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:292 +0x104
github.com/google/gnostic/surface.(*OpenAPI3Builder).buildFromComponents(0x14000169ac8, 0x140001c2120)
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:89 +0x3a8
github.com/google/gnostic/surface.(*OpenAPI3Builder).buildFromDocument(0x12c3b2928?, 0x140001b60c0)
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:61 +0x2c
github.com/google/gnostic/surface.(*OpenAPI3Builder).buildModel(0x14000169ac8, 0x140001b60c0, {0x16b2634cb, 0x18})
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:51 +0xb4
github.com/google/gnostic/surface.NewModelFromOpenAPI3(0x140001b60c0, {0x16b2634cb, 0x18})
        /Users/devlon.gilcrease/projects/gnostic/surface/model_openapiv3.go:32 +0x50
github.com/google/gnostic/lib.(*pluginCall).perform(0x140000787a0, {0x10521c890?, 0x140001b60c0?}, 0x3, {0x16b2634cb, 0x18}, 0x0, 0x0)
        /Users/devlon.gilcrease/projects/gnostic/lib/gnostic.go:174 +0x378
github.com/google/gnostic/lib.(*Gnostic).performActions(0x1400017a1c0, {0x10521c890?, 0x140001b60c0})
        /Users/devlon.gilcrease/projects/gnostic/lib/gnostic.go:611 +0x234
github.com/google/gnostic/lib.(*Gnostic).Main(0x1400017a1c0)
        /Users/devlon.gilcrease/projects/gnostic/lib/gnostic.go:683 +0x4c8
main.main()
        /Users/devlon.gilcrease/projects/gnostic/gnostic.go:44 +0xb0

Broken proto when enum field has hyphen

Something like:

enum:
  - FOO-BAR

Get parsed as

enum Value {
  FOO-BAR = 0;
}

Which is not a valid field name.

Possible solution

Replace all hyphen on field name to an underscore

Issue using the same OpenSpec attribute name when an enum is involved

Hello,

I'm using gnostic-grpc to generate a protobuf from my OpenAPI spec.

I'm running into an issue when enums are involved.

Here is a contrived OpenAPI spec snippet, please note the two occurrences of the attribute family:

components:
  schemas:
    Pet:
      required:
      - id
      - name
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        family:
          type: string
          enum:
            - dog
            - cat
        detailedFamily:
          type: object
          properties:
            family:
              properties:
                name:
                  type: string
                kind:
                  type: string
                  enum:
                    - dog
                    - cat

When I run:
gnostic-grpc --grpc_out=. enum-issue.yml

I see the following error:

panic: interface conversion: desc.Descriptor is *desc.MessageDescriptor, not *desc.EnumDescriptor

goroutine 1 [running]:
github.com/jhump/protoreflect/desc.(*FieldDescriptor).resolve(0xc000440080, 0xc000265828, 0x4, 0x8, 0xc00041d780, 0x2, 0x2, 0x0, 0x0)
	/Users/nbaliga/workspace/protoreflect/desc/descriptor.go:558 +0x79a
github.com/jhump/protoreflect/desc.(*MessageDescriptor).resolve(0xc000425b30, 0xc000265828, 0x3, 0x8, 0xc00041d780, 0x2, 0x2, 0x0, 0x0)
	/Users/nbaliga/workspace/protoreflect/desc/descriptor.go:339 +0x4c5
github.com/jhump/protoreflect/desc.createFileDescriptor(0xc000278e00, 0xc0002f7b60, 0x3, 0x3, 0x0, 0x2, 0xc000265b68, 0xc00034bb60)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:88 +0x1098
github.com/jhump/protoreflect/desc.createFromSet(0xc000338060, 0x10, 0x0, 0xc00041d690, 0x1, 0x1, 0xc000265b68, 0xc00034bb60, 0xc0001b5960, 0x0, ...)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:225 +0x358
github.com/jhump/protoreflect/desc.createFileDescriptors(0xc00034bb30, 0x4, 0x6, 0x0, 0x18, 0x1bb9320, 0x0)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:131 +0x1fd
github.com/jhump/protoreflect/desc.createFileDescriptorsFromSet(0xc000415000, 0x0, 0xc00034bb01, 0xc000415000, 0xc000265db8)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:193 +0xad
github.com/jhump/protoreflect/desc.createFileDescriptorFromSet(0xc000415000, 0x0, 0x3, 0x4, 0xc00034bb30)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:169 +0x39
github.com/jhump/protoreflect/desc.CreateFileDescriptorFromSet(...)
	/Users/nbaliga/workspace/protoreflect/desc/convert.go:165
github.com/googleapis/gnostic-grpc/generator.(*Renderer).RenderProto(0xc000306f80, 0xc000415000, 0xc000338050, 0x10, 0xc000338050, 0xc00033805a, 0xc000265e78)
	/Users/nbaliga/workspace/gnostic-grpc/generator/renderer.go:82 +0x49
github.com/googleapis/gnostic-grpc/generator.(*Renderer).Render(0xc000306f80, 0xc00024aaf0, 0xc000338050, 0x10, 0x6, 0xc000338050)
	/Users/nbaliga/workspace/gnostic-grpc/generator/renderer.go:62 +0xac
github.com/googleapis/gnostic-grpc/generator.RunProtoGenerator(0xc00026c420)
	/Users/nbaliga/workspace/gnostic-grpc/generator/main.go:58 +0x33e
main.main()
	/Users/nbaliga/workspace/gnostic-grpc/plugin/main.go:30 +0x145
Errors reading enum-issue.yaml
exit status 2

Based on 91dd378, I put in some debug statements in the relevant section of the code and I see the following:

Plugin error: 
[could not convert 
desc name:"Family"  
field:{name:"name"  number:1  label:LABEL_OPTIONAL  type:TYPE_STRING  type_name:""}  
field:{name:"kind"  number:2  label:LABEL_OPTIONAL  type:TYPE_ENUM  type_name:"Kind"}  
enum_type:{name:"Kind"  value:{name:"DOG"  number:0}  value:{name:"CAT"  number:1}} 

to 

EnumDescriptor. TypeName: enum_issue.Family. Type: TYPE_ENUM]

If I rename either of the family attributes in the spec, there is no issue.

Furthermore, suppose I change the type of of the 1st family attribute to anything other than enum, this also works and there is no issue.

The 2nd occurrence of the family attribute is an object that happens to contain an enum. The 1st occurrence of the family attribute is a straight enum.

And this seems to cause issues.

Any help will be appreciated!

Entire YAML:
enum-issue.txt

Polymorphism doesn't work correctly

Something like

"UPI": {
        "type": "object",
        "allOf": [
          {
            "$ref": "#/components/schemas/PaymentMode"
          },
          {
            "type": "object",
            "properties": {
              "vpa": {
                "type": "string"
              },
              "status": {
                "type": "string"
              }
            }
          }
        ]
      }

Generates

message UPIAllOf2 {
  string vpa = 1;

  string status = 2;
}

message UPI {
  PaymentMode all_of_1 = 1;

  UPIAllOf2 all_of_2 = 2;
}

In place of

message UPI {
  PaymentMode mode = 1;
  string vpa = 1;
  string status = 2;
}

Similar issues with discriminator and oneOf

Any plans on releasing with latest master?

Hi 👋 and thanks for your work.
There were several improvements since v0.1.0. Could you make a release with the latest version?

Regards