How to generate pool methods from buf cli

[planetscale]

How to generate pool methods from buf cli

[mattrobenolt]

I do this? https://github.com/planetscale/vitess-types/blob/main/buf.gen.yaml#L8-L22

So I'm not exactly sure what is different in your setup, but you can try to follow how it's working there.

[johnsiilver]

I added all the:

      - pool=github.com/planetscale/vitess-types/gen/vitess/query/dev.Row
      - pool=github.com/planetscale/vitess-types/gen/vitess/query/dev.BoundQuery
      - pool=github.com/planetscale/vitess-types/gen/vitess/binlogdata/dev.VStreamRowsResponse
      - pool=github.com/planetscale/vitess-types/gen/vitess/query/v15.Row
      - pool=github.com/planetscale/vitess-types/gen/vitess/query/v15.BoundQuery
      - pool=github.com/planetscale/vitess-types/gen/vitess/binlogdata/v15.VStreamRowsResponse
      - pool=github.com/planetscale/vitess-types/gen/vitess/query/v16.Row
      - pool=github.com/planetscale/vitess-types/gen/vitess/query/v16.BoundQuery
      - pool=github.com/planetscale/vitess-types/gen/vitess/binlogdata/v16.VStreamRowsResponse

But still no bueno. I can't use the out gen part, but since it seems to be rendering everything else, in the file, I'm doubting it is that.

Is there anything I have to add to the proto file itself to enable the pooling feature?

And thanks for helping with this.

[mattrobenolt]

Oh, I see your confusion then. You need to specify in the pool arguments the types you want to create pools for. :)

[johnsiilver]

Oh, I see. For some reason I was thinking those were some type of includes that were required, on second look it should have been obvious to me.

Thanks for the help. Out of curiosity, is there a way to specify generating a pool for all protos?

[mattrobenolt]

I don't believe so. You typically don't want it for every possible message type since pooling isn't objectively better.

You can add the vtproto.mempool annotation as well in your .proto files, as you suggested, which might help you out there. Both of these options should work fine. I will say though, I have never personally used that option, so I'm not sure how it works without reading through the source code. It's likely that might be worth it's own issue.

I'm going to close this issue now though unless there's something I'm missing. :)

[johnsiilver]

I'm curious why pooling isn't objectively better, or did you mean just not better for sub-protos and not parent objects?

I saw the annotation, but that seemed to generate an error on not know what the option was. I'll look at see if I can figure out what it takes to make that work, maybe checkout the source code myself.

Feel free to close the issue, and again, thanks for your time.

[mattrobenolt]

It's hard to quantify. If it's low frequency objects, pooling is just unnecessary and doesn't add any value. Likely would result in slowdown for objects that never get reuse between GC cycles.

It's just hard to objectively say in every case if it's better or not without profiling. Arguably, if it was objectively better in every situation, we'd use a pool for every object we ever allocate, and that's not true.

Protobufs also tend to a lot of times have difficult to track down lifetimes, so it's often difficult to determine when it's safe to return to the pool after use. This may not be the case for you, but in something like grpc-go, you don't control the lifetime, so using a pool is not really possible.

[johnsiilver]

Makes sense. As a note, I was looking for it to avoid a tooling problem. buf cli is wrapped in another layer for me (there are no actual buf.yaml files, everything is built dynamically and those files are rendered only for the lifetime of generation). So a little extra pool overhead that may or may not get used isn't a big deal, but I'd have to retool some to do the pool= opt stuff. I'm hoping the annotation might help.

And I get you with the lifetime. Was experimenting with allocating a bunch of protos with the arenas package, but I found that to be a nightmare to track. My takeaway was that an arena needs to belong to a type that never gives access to the arena data without returning a copy, and the object clears the arena on closure.

Thanks again for all the help Matt. Have yourself a good rest of the week.