Where is Broken Input going

[snowflakedb]

Where is Broken Input going

[yarmiganosca]

Is there any way to hook into the standard Connector failure handling behavior and send "bad" messages to a dead-letter topic?

We need to look into the null issue, but the design is to move all problematic data to the table stage for the target table. You can read more about table stages here, but the short of it is that each table has an associated stage named "@%mytable".

[madisoncoder15]

I found the mentioned table stage where the faulty messages are going, but it appears that the message contents is showing up under the record_metadata rather than the record_content, and the actual record_metadata is missing. I was required to create a file format with the error_on_column_count_mismatch configuration set to false in order to display the table stage.

[binglihub]

The file content on table stage is what connector received from Kafka. Since it is invalid (not a JSON or Avro), Connector can't process. So connector directly write it into a file and save on table stage

[sfc-gh-zli]

This #171 PR would enable the connector to support non-snowflake converters. If you use native json/avro converters, DLQ is naturally supported.