ed-xcf/protobuf2pydantic

generate pydantic model by protobuf.pb2 file

protobuf2pydantic

Generate a file which include pydantic models by using protobuf.pb2 file

Installation

pip3 install protobuf2pydantic

Getting Started

in CLI

pb2py ../test_pb2.py > wow.py

in Python

from protobuf2pydantic import msg2py
from pydantic import validator

import transaction_pb2


class AmountResponse(msg2py(transaction_pb2.AmountResponse)):
    @validator("amount")
    def non_negative(cls, v):
        assert v >= 0
        return v

* Required proto file syntax = "proto3";

protobuf2pydantic's Issues

2024.4.18 generation broken

We tried to upgrade from version 2023.12.21 to 2024.4.18 (the interim releases gave us the problems that have subsequently been fixed in the latest version).

For some reason, the output files are missing large chunks of code. Here's an example:

enum EventType {
    EVENT_TYPE_UNSPECIFIED = 0;
    EVENT_TYPE_ONE = 1;
    EVENT_TYPE_TWO = 2;
}

message GetEventsRequest {
    uint32 limit = 1;
    string status_filter = 2;
    EventType events_filter = 3;
}

In the previous version, the output is as follows:

class GetEventsRequest(BaseModel):
    limit: int = Field(default_factory=int)
    status_filter: str = Field(default_factory=str)

    class EventType(IntEnum):
        EVENT_TYPE_UNSPECIFIED = 0
        EVENT_TYPE_ONE = 1
        EVENT_TYPE_TWO = 2

    events_filter: EventType = Field(default_factory=int)

In the latest version, the output is as follows:

class GetEventsRequest(BaseModel):
    tenant_id: str = Field(default_factory=str)
    limit: int = Field(default_factory=int)
    status_filter: str = Field(default_factory=str)

So the event type class and events filter just got dropped.

In our response method, the same thing happened and all we were left with is the following, which isn't just wrong but invalid:

class GetEventsResponse(BaseModel):

My naive suspicion is that this is related to the fix with the inner class indentation.

Create instance from proto

Hi there,

Very nice pkg, I was testing it and quickly (for my use case [1]) I needed to convert from proto instance to pydantic, then I was wondering what do you think about adding a helper method to the generated classes such .from_proto? if you agree in the idea I can jump in in the implementation (or if you prefer implementing all good) .

[1] just playing in a proof of concept project that from a proto definition I generate the following services rpc (grpc), rest(fastapi), graphql, I basically 1) compile the proto and add the grpc server 2) use your lib to generate pydantic that empowers fastapi 3) from the generated pydantic I use pydantic2graphene to expose for graphql

from protobuf2pydantic import msg2py
from pydantic import validator

import transaction_pb2


class AmountResponse(msg2py(transaction_pb2.AmountResponse)):
    @validator("amount")
    def non_negative(cls, v):
        assert v >= 0
        return v


msg_proto = transaction_pb2.AmountResponse(amount=10.01, currency="USD")

amount_pydantic = AmountResponse.from_proto(msg_proto)

KeyError in Mappings

Hello,

Thanks a lot for this library, it is perfect to interface ProtoBufs and FastApi.

I got troubles with the following proto:

syntax = "proto3";

message Entity {
    int32 start=1;
    int32 stop=2;
    string type=3;
    string text=4;
    string value=5;

}

message EntityList {
    repeated Entity entities = 1;
}

message EntityOutput {
    map<string, EntityList> entities_per_lang = 1;
}

Got a KeyError : 11 as the field in the dict is another proto field.

This could be solved by taking the corresponding class.
(I could make a PR but as the code is small I'm pasting it here. If it's easier for you I can make a PR)

Changing the convert field function to

def convert_field(level: int, field: FieldDescriptor) -> str:
    level += 1
    field_type = field.type
    field_label = field.label
    was_mapping = False
    extra = None

    if field_type == FieldDescriptor.TYPE_ENUM:
        enum_type: EnumDescriptor = field.enum_type
        type_statement = enum_type.name
        class_statement = f"{tab * level}class {enum_type.name}(IntEnum):"
        field_statements = map(
            lambda value: f"{tab * (level + 1)}{value.name} = {value.index}",
            enum_type.values,
        )
        extra = linesep.join([class_statement, *field_statements])
        factory = "int"

    elif field_type == FieldDescriptor.TYPE_MESSAGE:
        type_statement: str = field.message_type.name
        if type_statement.endswith("Entry"):
            key, value = field.message_type.fields  # type: FieldDescriptor
            if value.type != 11:
                type_statement = f"Dict[{m(key)}, {m(value)}]"
            else:
                was_mapping = True
                type_statement = f"Dict[{m(key)}, {value.message_type.name}]"
            factory = "dict"
        elif type_statement == "Struct":
            type_statement = "Dict[str, Any]"
            factory = "dict"
        else:
            extra = msg2pydantic(level, field.message_type)
            factory = type_statement
    else:
        type_statement = m(field)
        factory = type_statement

    if field_label == FieldDescriptor.LABEL_REPEATED and not was_mapping:
        type_statement = f"List[{type_statement}]"
        factory = "list"

    default_statement = f" = Field(default_factory={factory})"
    if field_label == FieldDescriptor.LABEL_REQUIRED:
        default_statement = ""

    field_statement = f"{tab * level}{field.name}: {type_statement}{default_statement}"
    if not extra:
        return field_statement
    return linesep + extra + one_line + field_statement

Have a great day

add cli support for folders and output to a file/folder

Hi.
I've just found your tool and seems magic :) very cool

as I am working with a "big" schema definition (~100 .proto files, distributed in a 3/4 level nested tree) I was wondering if you could enhance pb2py to support the input from a folder (and implicitly convert all the *_pb2.py files inside) and output to a target folder (and not on stdout)

at the moment I am putting in place a shell script based on find, mkdir -p and redirection, but it's very fragile and I was wondering if you could help on the tool side

Error running CLI - unsupported operand type(s)

Hi

I think there's some issue with the CLI.
I did the following:

python3.9 -m venv .venv
source .venv/bin/activate
pip3 install protobuf2pydantic
pb2py --help 

and I got:

Traceback (most recent call last):
  File "/Users/my_user/repos/tmp/.venv/bin/pb2py", line 5, in <module>
    from protobuf2pydantic.main import app
  File "/Users/my_user/repos/tmp/.venv/lib/python3.9/site-packages/protobuf2pydantic/__init__.py", line 11, in <module>
    from protobuf2pydantic.biz import msg2pydantic
  File "/Users/my_user/repos/tmp/.venv/lib/python3.9/site-packages/protobuf2pydantic/biz.py", line 64, in <module>
    def convert_field(level: int, field: FieldDescriptor) -> str | None:
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

I'm getting a similar error when I run pb2py /path/to/protobuf/dummy_pb2.py > wow.py command, and when I simply import protobuf2pydantic.

Note the issue seem to exist in the newest version only, 2024.1.8.
The version one before it 2023.12.21 seem to be working fine.

I'm using Python 3.9.13, on a Mac with Sonoma 14.1 OS, with protobuf2pydantic version 2024.1.8

Make use of message enum number instead of index.

In the below reference, you make the value number in the generated Pydantic model equal to the index of the elements in enum. This can cause problems when enum set in the proto file is not in a running number.

It can be solved by placing value.number instead of value.index.

https://github.com/Ed-XCF/protobuf2pydantic/blob/d184c118eac463af22bc762519edf5086cb654ac/protobuf2pydantic/biz.py#L53C13-L53C78