Comments (4)
I think I can do this a slightly coarser grain - which is weekly for football sims and daily for nba. The basic way to do this will be hold some folders in s3, and then based on when the most recent actual is recorded, we can use write the new results into a folder.
something like s3://some_bucket/sim_version/year/week/model.parquet
, or for the nfl model it would be currently this path: s3://some_bucket/nfl/2023/0/nfl_reg_season_summary.parquet
from nba-monte-carlo.
Pipeline then basically has a final step of dropping new files into S3, then building a duckdb file that can hit the entire directory. A thin copy of the parquet files. Then evidence hits that duckdb file instead of the local file.
from nba-monte-carlo.
related to this conversation here: #102
from nba-monte-carlo.
Actually this is a dupe of #102 and will be solved as part of that.
from nba-monte-carlo.
Related Issues (20)
- Rework external materializations to allow persistence HOT 4
- What to do about Rill & Superset?
- load dbt exposures to evidence sources HOT 2
- IOError from DuckDB when run with Superset HOT 3
- add conference pages to ncaaf model
- add team pages to ncaaf model
- add actuals rfwd for both ncaaf & nfl models HOT 2
- add team pages to nfl model
- Upgrade to evidence universal SQL HOT 2
- bring NFL model up to parity with NCAAF
- Malloy? HOT 1
- automate ingestion HOT 5
- fix bug on nba main page HOT 2
- misc model updates HOT 5
- Clean up cosmestic issues
- missing matchups don't calculate (known issue - source is future games, instead of generic teams table) EX: BRK @ UTA HOT 1
- year filter - year isn't holding sort order HOT 1
- team filter doesn't update when you change the year (sql query updates, filter does not)
- DBT build errors HOT 5
- fix misc build issues HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nba-monte-carlo.