bobheadxi/timelines

🏷 Historical analysis of Git repositories and Git host activity as a service

What is this?

timelines's Issues

db: `res.RowsAffected() != itemCount` causes havok

https://github.com/bobheadxi/projector/blob/3138160/db/repos.go#L143

this:

• reports 1 row has been updated, when 30 items were provided
• causes worker to hang in an uninterruptible state. this probably hints at a more serious issue with all the WaitGroups - stacktrace in comments

db: add postgres

will probably need it for users, projects, etc. that wont be indexed all the time

analysis: language burndown

can't find examples, but looks like there is language detection floating around, and a commit about language reports. We could analyze the growth of languages over time based on additions/deletions, to stick with git clone --bare

web: POC graphql query

web: set up typescript

https://facebook.github.io/create-react-app/docs/adding-typescript

chore: set up tslint for web

git: garbage collection, loading existing repos

web: scaffold data visualizations

some libraries:

• https://github.com/hshoff/vx I like this one (based purely on examples)
• https://vis.gl/ has a collection of pretty fancy tools ⭐️
  • possibly overkill but damn it looks cool - https://medium.com/vis-gl
  • will probably be using deck.gl as of e76b177
• https://github.com/uber/react-vis tbh this is pretty generic and a little inflexible, I think it's pretty similar to:
  • https://github.com/FormidableLabs/victory
  • https://github.com/recharts/recharts

github: add support for milestones and tags as Items

analysis: only analyze new commits

design: branding

• logo: https://www.figma.com/file/PKNIT3TZiFpR3RQ8uOVBAt8F/Logo
• colors: https://www.figma.com/file/gPFrzttllHsLZdUzsFdWFyGP/Colors

graphql: set up graphql codegen

https://github.com/dotansimha/graphql-code-generator

https://github.com/graph-gophers/graphql-go

deployment: set up postgres, worker on a separate vps's

heroku provides 10k rows free, which is almost certainly not enough. might use for dev, but probably host separately on launch - admin might be painful but whatever

digital ocean

DO offers nicer and more understandable pricing, but no free tier.

google cloud

has free tier

alternatively, g1-small is ~$14/mo and n1-standard is ~$24/mo

github: plan out sync story

need to plan this out better. blocks #3

github: sync stars history

eg https://github.com/timqian/star-history

github: scaling for large, large repos

At 5000 requests per hour, a repo like https://github.com/Microsoft/vscode will take more than 14 hours to sync all 70,000+ issues and PRs (at time of writing). Some things to do:

• if rate limit is hit, need to put the job back on the queue, to be picked up after an hour
• improve how PRs are fetched (right now we list all issues, which is pretty efficient, but if an issue is a PR a separate API request is made to grab it. repos with 5000+ PRs will hit the limit fast)

also:

• probably can't naively sync everything per sync - will probably have to leverage webhooks, and only ever do a full sync exactly once (#39)

feature: organization and user overviews

deployment: set up Procfile

for now, try deploying both to a single heroku instance, ie with 2 procs. if that doesn't perform too well, can run:

• server on heroku dyno
• worker(s) on some other cloud instnaces, using Inertia

analysis: work out burndown data format, pipe into db

what is this???

[
  [1086,  0, 0,0, 0,0],
  [1022,389, 0,0, 0,0],
  [1022,389,29,0, 0,0],
  [1022,389,29,0, 0,0],
  [1011,365,29,0,58,0],
  [1011,365,29,0,56,2],
]

need to:

• finalize burndown format
• insert into db

deps: drop gocloud.dev dependency

sounds fancy, but doesnt really add much value right now

deployment: investigate Cloudflare

ci: cache dependencies

~3min per build is spent on dependency fetch

git: scaling repo storage

low-priority, but at scale will probably need some kind of compaction:

https://github.com/src-d/borges

db: drop existing analysis on insert (?)

still pondering if old analysis are worth holding onto. each on generates a lot of rows however, and old data is of dubious usefulness (since scaling ticks will render the bands / samples incomparable)

for now, might be a good idea to drop any old data before analysis

github: webhooks

see #26

testing: set up goleak to check for leaked goroutines

i have lots of goroutines floating around

https://github.com/uber-go/goleak

also #22

ci: flesh out pipeline

worker: clean up error reporting

right now a lot of stuff doing:

1. log error
2. pass error to channel

pretty wordy

analysis: source insights

https://github.com/src-d/awesome-machine-learning-on-source-code

https://docs.google.com/presentation/d/e/2PACX-1vQpGOxoE094t-cU21Auij0GT3pybvY5yHlp1txFjarMYjx1Siyuryw4sfKN3pOj3Z9hHu2s_TARkviY/pub?start=false&loop=false&delayms=3000&slide=id.p

all: rename to “timelines”

feature: sign in to see self in project timelines

all: set up a worker and a service

service: API access, etc.
worker: chron jobs, sync tasks, git stuff, etc.

feature: periodic reports

e.g. https://github.com/icecrime/docker-bulletin/blob/master/repositories/docker.docker/2016-03-12.md

graphql: define API schema

https://github.com/bobheadxi/timelines/blob/master/graphql/schema.graphql

probably want to refer to #41

web: graphql client

https://github.com/apollographql/react-apollo

probably need to update the codegen as well (follow-up to #1)

db: burndown CRUD

db, graphql: top-n files, contributor queries

files: files with active changes
contributors: most active contributors

web: set up UI template or framework

Some options:

• https://blueprintjs.com/
• https://semantic-ui.com/
• https://github.com/uikit/uikit ⭐️

docs: project documentation

• project overview
• architecture diagram
• go: build + dev instructions
• web: build + dev instructions
• cmd tooltips (most are currently blank)
• makefile help text (ie https://github.com/ubclaunchpad/inertia/blob/master/Makefile)

db: use go-cloud pg

https://godoc.org/gocloud.dev/postgres

git, analysis: explore basic insight possibilities

deployment: postgres

https://www.heroku.com/postgres

analysis: scaling burndowns for short-term (eg hackathon) vs long-term (eg linux) projects

• short-term, eg https://github.com/lunchpals/open-now: <100 commits, across 48 hours
  • suitable timescale: every hour
  • probably takes negligible amount of time to process
• long-term, eg https://github.com/torvalds/linux: >648,000 commits, across >27 years
  • suitable timescale: every month or 2 months
  • can take over an hour to process (example in https://github.com/src-d/hercules notes a Linux burndown took 1h 40min to generate)

will need to scale timelines dynamically.

• implement src-d/hercules#238 (PR at src-d/hercules#245) (pulled in 013252a)
• project "duration" detection of some sort - need to work out how commits are clustered
  • for now, probably just go with the implementation in f22e174 etc - commit cluster detection can come later, if it's actually an issue
• update DB to account for potentially volatile timescales

data: decide on store

probably with Google Cloud: https://cloud.google.com/free/

db: pipe github data into postgres

web: hand-draw some potential visualizations

github: set up sync

store: set up redis

redis as job queue, can later use as rate limiter as well - probs more versatile overall

db, git: account for potential other git hosts

• add host field in database repositories table
• add github.com to git repo paths
• move package github to host/github
• rename github_item to meta_item or something

etc.

graphql: burndown query