Metadata fields

[benthosdev]

Metadata fields

[Jeffail]

Reviewed all the inputs and added metadata from the ones I felt made sense.

[chrisriley]

I've been working on a version of the amqp input that includes all but Acknowledger and MessageCount from the amqp.Delivery struct. Still working out the nesting of amqp.Table in Delivery.Headers. I'll send you a PR when I'm done to see if you can use any of it.

[Jeffail]

I've just pushed an API change that allows per message metadata in batches and updated all the existing components to use it.

[seanlaff]

What are your thoughts on allowing metadata on messages in addition to parts? I have a processor that is turning a multi-part message into N new messages (basically performing a group-by) and it'd be handy if I could enrich each outgoing message with metadata about the grouping.

[Jeffail]

Hey @seanlaff, how would you intend on using the metadata? For cases like that I would usually either suggest setting and referring to the batch wide metadata on the first part, or just set the field on all parts.

[seanlaff]

@Jeffail Inspecting the metadata on the first part is the solution I opted for in the end, and works pretty well. My use case may be a little contrived, but this is what it looks like:

Data is streamed from pubsub to a benthos batch processor, and then to a database sink. However the database table that the data is written into depends on a dynamic field in the message. Additionally, the database does not support writing to multiple tables in one request.

My first solution was to spin up goroutines in the output to handle writing to the N discrete tables concurrently- but that felt like work I could relegate to benthos itself. Instead, I opted to create a processor (executed after the batch processor) that creates N new multi-part message, with each message containing all parts that share the same keying field. Then in my output I inspect the metadata of the first part, and make a singular database write for the correct table for all parts of that message at once (thus leaning on benthos to do the concurrent work I was doing by hand before)

This may be a non-idomatic use of benthos- and is only a minor inconvenience since I can achieve the same by checking the metadata on the first part

[Jeffail]

@seanlaff, sounds like a reasonable approach. Let me know if you hit any issues with doing it this way and I'll take another look.

[ghballiet]

@Jeffail I've raised a PR for a candidate implementation of metadata with the Kafka output type - happy to address any comments there.

[Jeffail]

Closing this as it's become stale. The bulk of the work is done, anything else can be tackled if/when people need it.