Performance issue with model snowplow_page_views

[dbt-labs]

Performance issue with model snowplow_page_views

[drewbanin]

Hey @sphinks - I think you're totally right here - the changes we introduced in 0.7.x were optimized for correctness and not for speed. I think this is one case where it's better to be 99% correct and fast than 100% correct and slow. We have plans to revisit these models to make them a lot more performant, and I think a big part of that will be minimizing the "lookback" period as you're describing!

[sphinks]

Is it possible to make fix for this part of code write now? It will allow to upgrade to new branch and use lot of fixes, while waiting for big refactoring? I can prepare PR in case my idea is ok.

[drewbanin]

Sure, feel free to submit a PR! I'd love to take a look at your approach here.

[sphinks]

@drewbanin have created PR: #69 Any idea how to check it 30 days are ok? I was checking plans with original and new code. Looks like new one is much better. But the issues is every project has it is own amount of events per month, so it is better to setup as small interval for session as possible in any case.

[jtcohen6]

In my experience on a previous project, where we had 2.5 months of Snowplow data (285mm events):

99.82% arrived within 1 hr of firing (diff b/w derived_tstamp + collector_tstamp)
99.88% arrived within 24 hr

We want to remove the full look-back and rebalance speed and correctness in 0.8.x. I wonder if we should even look back 30 days?

[drewbanin]

@jtcohen6 @sphinks I'd be inclined to make it less than 30 days for sure. We can either do:

1. Always reprocess the last 2-3 days of data
2. Dynamically find the last day that dbt ran, then reprocess data in that interval

I think the second approach is more aligned with how dbt usually works. You guys buy that?

[sphinks]

@drewbanin but taking second approach is not reflecting the idea of initial commit as far as I understand. We talking events starting from last date dbt was running, but take a look in the whole history to find match for session. In case we replace history match with last date dbt was running, do we have any new info? I mean history interval we are reprocessing will be equal to interval of non processed events. Am I misunderstand second approach?

[sphinks]

Have created a PR #71 to fix the issue.

[jtcohen6]

Closing this issue, since we merged #71 and released 0.8.0