kazanzhy/kafka-connect-transform-record2row

kafka-connect-transform-record2row's Issues

Add option to directly replace field with JSON-serialized value

This SMT seems super helpful, I was surprised something like this didn't exist already. However, there is one issue I have with it right now that would require some workarounds to bring our data into a nice shape.

Before the field value is converted to JSON, you wrap it with a struct first, which would transform a field called "nested" with a value like the following

{
  "obj": { "a": "foo", "b": "bar" },
  "arr": [1, 2, 3]
}

into the string value

"{\"nested\":{\"obj\":{\"a\":\"foo\",\"b\":\"bar\"},\"arr\":[1,2,3]}}"

that gets written back to the nested field in the output record. When this data now gets inserted into e.g. a Postgres JSONB column, I'd have to query the obj.a value like this:

SELECT nested -> 'nested' -> 'obj' -> 'a' FROM table;

That seems like unnecessary duplication to me and is a little unergonomic for the user. It would be nicer if they could just write

SELECT nested -> 'obj' -> 'a' FROM table;

Thus, an option to skip the struct-nesting step of the transformation would be much appreciated. I might give implementing it myself a go if that's welcome.