wrong page_view_index if session alive across days

[dbt-labs]

wrong page_view_index if session alive across days

[drewbanin]

hey @tromika - i think you're totally right. To calculate this field accurately, we'd need to start counting from the max page_view_index for the given user. That gets really difficult when you account for identity stitching, so I'm not sure that there's a super easy answer here. If anything, I'd be in favor possibly removing (or renaming) this field or adding another pageview model that is built in a full-refresh fashion. Keen to hear your thoughts

[tayloramurphy]

Just FYI, a full refresh fixed this. For now we're just going to schedule a weekly full refresh.

@drewbanin I do wonder if it'd be possible to do date-windowed incremental refreshes. It seems like refreshing data from the past 2-3 weeks would solve most of the issues.

[tromika]

@drewbanin hopefully this is the case only for few users so with regular refresh this could be solved. Anyway, would be great if this could be solved somehow but as you said I know it's not easy.

[jtcohen6]

We've gone back and forth on this genre of issue over the past 1.5 years (!). Ultimately, I think we've decided that this package is going to privilege performance and clarity of transformation over 100% correctness in incremental runs. The operative assumption is that the user could run a full (or partial) refresh of historical data to fix issues like the one raised here. If that's not the case, the user can override the snowplow_page_views model and implement a slower-but-always-correct approach.