String Format Decimals Converting To Unknown Codes

[snowflakedb]

String Format Decimals Converting To Unknown Codes

[sfc-gh-zli]

This is a bug inside SnowflakeAvroConverter. We internally used org.apache.avro.generic.GenericDatumReader to convert AVRO bytes to JSON in SnowflakeAvroConverter, which does not handle byte array and BigDecimal well. I have a walk around:

1. change your value.converter from com.snowflake.kafka.connector.records.SnowflakeAvroConverter to io.confluent.connect.avro.AvroConverter.

2. Make sure your Snowflake connector version >= 1.4.2

@drobert FYI the code you commented on is only triggered when non-snowflake-converters are used. If you see further problem after changing to confluent's converter, we can look deeper on the lines that you commented on. Previously if you are using SnowflakeAvroConverter, the conversion from AVRO to JSON happens here: https://github.com/snowflakedb/snowflake-kafka-connector/blob/master/src/main/java/com/snowflake/kafka/connector/records/SnowflakeAvroConverter.java#L175

[sfc-gh-zli]

please try to upgrade to 1.4.0, this should have been fixed in #131

[mjreiss]

@sfc-gh-zli Is this issue still ongoing? We are currently on version 1.4.1, and all messages are going into snowflake with this unusable format.

[drobert]

Looking at the referenced PR (#131) I commented on a few things that look like they may be contributing: https://github.com/snowflakedb/snowflake-kafka-connector/pull/131/files#r463259581

Notably:

• the conversion code casts the byte array to BigDecimal without specifying scale or precision
• json Number types don't support arbitrary precision. I would expect only a String representation to be able to represent a BigDecimal accurately