Incremental replication doesn't respect the current state

[emishas]

Incremental replication doesn't respect the current state

[laurentS]

Hi @emishas thanks for reporting this. I'm not entirely sure how meltano calls the tap from the above config, as I don't use meltano at all, but my initial hunch is that the repositories stream is not really incremental. The stream returns a single record for a repo, at most (if the repo name does not exist, the tap skips it and returns nothing).

I'm not entirely sure that updated_at, which is set as the replication key for the stream, is actually updated for all changes of the record.

Reading the docs:

Using the state, the tap should then skip returning rows where the replication key comes strictly before than previous maximal replication key value stored in the state.

As you only have a single record for each repo, the newer record value for updated_at cannot be strictly before itself. So I think it makes sense that all records are returned by the tap.

Then my guess is that in your case the deduplication logic is incorrect for the stream, which would explain why all records end up being duplicated. There is no primary_key nor state_partitioning_keys on the stream, which might explain this. Can you share what (a couple of) your records look like, and what the state file content is? It might help understand what is happening.