Implement the topics.regex configuration parameter

[snowflakedb]

Implement the topics.regex configuration parameter

Thanks for the suggestion -- we are looking at this.

A question, though. Without an entry in the tables2topicsmap, we ingest topics into a table with the same name. This can cause problems with Snowflake's limitations on table names. E.g., we can't have a table with a hyphen, so we need to remap "my-topic" to something. But we can't extend the map to dynamically-added topics.

What would you prefer:

1. We don't ingest new topics whose names are invalid Snowflake identifiers.
2. We strip any non-conforming characters, and append a disambiguator to the name. E.g., "foo-bar" might become "foobar_12349876".

[yarmiganosca]

I think as long as you logged that the attempt had been made, with the offending topic name, and the logs weren't flooded with that (so like no more than once every five seconds per offending topic plz plz), we can handle that on our end. We're controlling topic names with a ruby gem that handles Avro schema encoding and publishing to a topic, so we're already abiding by the Snowflake table name rules, but I think logging is at least a good first step for a good general failure behavior.

[iexos]

We auto-generate the table name in the map anyway. The corresponding part in the shell script is:

table=$(echo $topic | tr '.[:lower:]' '_[:upper:]')

This works pretty well for us, as we only have dots as special characters in topic names. In my view the second option is preferred, as our team is just one of many users of Kafka and cannot control the naming convention just for this connector.

[binglihub]

4db8759