May be it is incorrect way of filtering duplicates?

[dbt-labs]

May be it is incorrect way of filtering duplicates?

[drewbanin]

Hey @sphinks - we adopted this logic from the snowplow web data model.

Did you find this because of a problem with your data? Or were you just scanning the code? I think the point here is that there are two duplication steps:

1. group by every column to "distinct" the dataset
2. exclude records for page views with multiple, different user agents

There's a subtle distinction here. The first step squashes identical/duplicate records into a single row. The second step will intentionally throw away context for pageviews with multiple, different user agent contexts.

If a browser is reporting that a single pageview is represented by two different user agents, then there's probably something bad happening either in the browser or in the tracking implementation.

I think the alternative of just picking one is also a reasonable approach, but it's not how the original Snowplow web data models were built, so I'm hesitant to change it without good reason! I would however be super happy to have better documentation around some of this code.

Let me know if all of this makes sense

[sphinks]

@drewbanin I have find it just by scanning code and you reference to original snowplow model is ok, to understand that it should go like it is implemented. Thanks!