Read from multiple topics with multiple schemas

[absaoss]

Read from multiple topics with multiple schemas

[felipemmelo]

For multiple topics with multiple schemas, you'll need multiple jobs, even if only using Spark Structured Streams directly.

However, if you're using TopicNameStrategy (as in here) for Schema Regsitry, you can reuse the configuration and only change the topic name, as the schema name will defined from the topic, as explained here.

[felipemmelo]

Hi @mxolisiz , if the topics have the same schema, it should work. You can separate the topics by using RETAIN_ORIGINAL_SCHEMA as explained here

If you however want to read topics containing different schemas, then it should not work from a limitation on Spark Structured Streams, as the schema for the dataframes needs to be defined before the job is started.

[mxolisiz]

Hi @felipemmelo thanks for the response
"as the schema for the dataframes needs to be defined before the job is started" That makes sense thank you... I am reading from more than one topic which all have different schemas

Stupid question :) - does that mean if i want to use this library with my multiple topics & multiple schemas I'd probably have to read each topic to it's own separate dataframe and have separate schemaRegistryConfs (if writing that type of code in a clean way is possible)

[mxolisiz]

@felipemmelo Oh ok, Thanks a lot for the assistance. learnt a lot here.