Comments (1)
Hey @philipherrmann, that logic comes straight from Snowplow's original web data model for Redshift. In particular, this comment:
-- exclude all root ID with more than one page view ID
We interpreted that to mean, "Events associated with multiple different page view IDs are considered noise and should be excluded." Why it's there is a fair question, though: snowplow/snowplow-web-data-model#43
To be honest, that's a place where we've deferred to the Snowplow folks. They've just released a new web model (SQL transformations for Redshift); I took a look and couldn't find this exact logic replicated there, though there were several other steps in which duplicated event IDs + page view IDs are removed entirely.
Also, for bigquery it would be quite easy to implement an incremental model for this since all the timestamps are there, right? Should I try to submit a PR?
The snowplow_web_page_context
model actually does not run on BigQuery. Instead, page view IDs are just pulled straight into the snowplow_page_views
model, without any duplication:
https://github.com/fishtown-analytics/snowplow/blob/f24a2bf91d4ce44f789d1cae0e33d85aa7f8eb58/models/page_views/bigquery/snowplow_page_views.sql#L62-L66
Why? At this point, I'm not quite sure. I think we should aim for consistency, and presently that feels like removing this logic across the board.
from snowplow.
Related Issues (20)
- NoneType has no len() error. HOT 7
- Support new snowplow data model HOT 2
- Compilation Error in model snowplow_web_ua_parser_context (models/page_views/optional/snowplow_web_ua_parser_context.sql) at path ['enabled']: None is not of type 'boolean' Code: 10004 HOT 3
- When are the todos in bigquery going to be implemented? Columns missing HOT 2
- no snowplow_users model - lookml should be removed HOT 1
- Testing for late-arriving facts HOT 1
- Update package readme
- Use enabled config to implement cross-database support HOT 2
- Map inferred_user_id in snowplow_page_views HOT 1
- Out of range exception HOT 4
- Performance issue with model snowplow_page_views HOT 9
- "Unrecognized name: root_id" and "No matching signature for function TIMESTAMP" HOT 3
- Fails to exclude page views from pingdom HOT 1
- snowplow_web_events_internal_fixed is not used
- Syntax Error in BigQuery snowplow_page_views
- Package tidy up
- Note undesired behavior from setting enabled: true in root project
- Add Spark support HOT 1
- Snowflake web page context is unnecessarily separated and is not incremental HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snowplow.