Invalid SCHEMA messages are produced for deselected streams

[laurentS]

Invalid SCHEMA messages are produced for deselected streams

[laurentS]

Thanks @edgarrmondragon for these details. selected_by_default seems to have appeared in sdk v0.27 but this tap cannot upgrade past v0.22 due to meltano/sdk#1704 so I guess we'll have to wait for meltano/sdk#1708 to be merged.

selected_by_default seems to be a good way to go though, and this bug is not immediately blocking for us. I'll wait on #209 to propose a fix for this.

[laurentS]

I'm reopening this issue, as we are still seeing the bug with tap-github on the Traffic* streams.
The bug is happening for streams that have selected_by_default = False. For instance:

{"type": "SCHEMA", "stream": "traffic_clones", "schema": {"properties": {"repo": {"type": ["string", "null"]}, "org": {"type": ["string", "null"]}, "repo_id": {"type": ["integer", "null"]}, "timestamp": {"format": "date-time", "type": ["string", "null"]}, "count": {"type": ["integer", "null"]}, "uniques": {"type": ["integer", "null"]}}, "type": "object"}, "key_properties": ["repo", "org", "timestamp"], "bookmark_properties": ["timestamp"]}

is generated with selected_by_default = True but if that option is flipped to False then we get:

{"type": "SCHEMA", "stream": "traffic_clones", "schema": {"properties": {}, "type": "object"}, "key_properties": ["repo", "org", "timestamp"], "bookmark_properties": ["timestamp"]}

The first SCHEMA message is valid, the second one isn't. This is with singer-sdk = 0.32.0.

I think the problem is that the test case added in #1698 does not cover the use case described here.

[edgarrmondragon]

@laurentS thanks for reporting this. The latest SDK was shipped with Stream.selected_by_default, which has the same use cases in mind. That involved adding unit tests in the SDK, so it may be more robust. We can give that a try.

Also, I don't think this behavior is spilling into normal sync runs since tap-github --test=schema ignores any selection.

PS: meltano/sdk#1698 might help