cncf/devstats.archive

📈CNCF-created tool for analyzing and graphing developer contributions

GitHub archives and git Grafana visualization dashboards

Authors: Łukasz Gryglicki [email protected], Justyna Gryglicka [email protected].

This is a toolset to visualize GitHub archives using Grafana dashboards.

GHA2DB stands for GitHub Archives to DashBoards.

More information about Kubernetes dashboards here.

Kubernetes and Helm

Please see example Helm chart for an example Helm deployment.

Please see Helm chart for a full Helm deployment.

Please see LF Helm chart for the LF Helm deployment (it is a data deployment, has no Grafana, and uses ElasticSearch in addition to Postgres to store data).

Please see GraphQL Helm chart for GraphQL foundation DevStats deployment.

Please see Kubernetes dashboard if you want to enable a local dashboard to explore the cluster state.

Please see bare metal example to see an example of bare metal deployment.

The rest of this document describes the current bare metal deployment on metal.equinix.com used by CNCF projects.

Presentations

• Presentations are available here.
• Direct link.
• Another direct link.

Talks

• Who What How: Understanding Kubernetes Development through DevStats.
• A Kubernetes Application End-to-End: DevStats.

Architecture

DevStats is deployed using Helm on Kubernetes running on bare metal servers provided by Equinix.

DevStats is written in Go, it uses GitHub archives, GitHub API and git as its main data sources.

Under the hood, DevStats uses the following CNCF projects:

• Helm (for deployment).
• containerd (as a Kubernetes container runtime, CRI).
• cert-manager (for HTTPS/SSL certificates).
• OpenEBS (for local storage volumes support).
• MetalLB (as a load balancer for bare metal servers).
• CoreDNS (Kubernetes internal DNS).

And other projects, including:

• Equinix (bare metal servers provider).
• Ubuntu (containers base operating system).
• kubeadm (for installing Kubernetes).
• NFS (for shared write network volumes support).
• NGINX (for ingress).
• Calico (as networking for Kubernetes, CNI).
• Golang (DevStats is written in Go).
• PostgreSQL (DevStats database is Postgres).
• patroni (HA deployment of PostgreSQL database, tweaked for DevStats).
• GitHub archives (main data source).
• GitHub API (data source).
• git (data source).
• Grafana (UI).
• Let's Encrypt (provides HTTPS/SSL certificates).
• Travis CI (continuous integration & testing).

Please check this for a detailed architecture description.

Deploying on your own project(s)

See the simple DevStats example repository for single project deployment (Homebrew), follow instructions to deploy for your own project.

Goal

We want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).

Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.

The only requirement is that project must be hosted on a public GitHub repository/repositories.

Data hiding

If you want to hide your data (replace with anon-#) please follow the instructions here.

Forking and installing locally

This toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases, and multiple Grafana instances. It is written in Go and can be forked and installed by anyone.

Contributions and PRs are welcome. If you see a bug or want to add a new metric please create an issue and/or PR.

To work on this project locally please fork the original repository, and:

• Compiling and running on Linux Ubuntu 18 LTS.
• Compiling and running on Linux Ubuntu 17.
• Compiling and running on Linux Ubuntu 16 LTS.
• Compiling and running on macOS.
• Compiling and running on FreeBSD.

Please see Development for local development guide.

For more detailed description of all environment variables, tools, switches, etc, please see Usage.

Metrics

We want to support all kinds of metrics, including historical ones. Please see requested metrics to see what kind of metrics are needed. Many of them cannot be computed based on the data sources currently used.

Repository groups

There are some groups of repositories that are grouped together as a repository groups. They are defined in scripts/kubernetes/repo_groups.sql.

To setup default repository groups:

• PG_PASS=pwd ./kubernetes/setup_repo_groups.sh.

This is a part of kubernetes/psql.sh script and kubernetes psql dump already has groups configured.

In an All CNCF project repository groups are mapped to individual CNCF projects scripts/all/repo_groups.sql:

Company Affiliations

We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.

There is a project that attempts to create such mapping cncf/gitdm.

DevStats has an import tool that fetches company affiliations from cncf/gitdm and allows to create per company metrics/statistics. It also uses companies.yaml file to map company acquisitions (any data generated by a company acquired by another company is assigned to the latter using a mapping from companies.yaml).

If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on https://k8s.devstats.cncf.io a couple of days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.

New affiliations are imported into DevStats about 1-2 times/month.

Architecture

For architecture details please see architecture file.

Detailed usage is here

Adding new metrics

Please see metrics to see how to add new metrics.

Adding new projects

To add a new project on a bare metal deployment follow adding new project instructions.

See cncf/devstats-helm:ADDING_NEW_PROJECTS.md for information about how to add more projects on Kubernetes/Helm deployment.

Grafana dashboards

Please see dashboards to see a list of already defined Grafana dashboards.

Exporting data

Please see exporting.

Detailed Usage instructions

• USAGE

Servers

The servers to run devstats are generously provided by Equinix bare metal hosting as part of CNCF's Community Infrastructure Lab.

One line run all projects

• Use GHA2DB_PROJECTS_OVERRIDE="+cncf" PG_PASS=pwd devstats.
• Or add this command using crontab -e to run every hour HH:08.

Checking projects activity

• Use: PG_PASS=... PG_DB=allprj ./devel/activity.sh '1 month,,' > all.txt.
• Example results here - all CNCF project activity during January 2018, excluding bots.

devstats.archive's Issues

Custom Date Ranges for Dashboards

It would be helpful to be able to pick a custom date range for all dashboards.

Use Case: We are creating programs around mentoring and contributor ladder where we will need to measure the value from multiple angles and the programs may not start or end on a specific 7 day/month/etc. period. It may be something like the last three days or two weeks, etc.

Fix shrinking icons when window size too small horizontally

Please fix logos on https://all.cncftest.io/dashboard/db/dashboards?refresh=15m&orgId=1 so that they shrink proportionally instead of getting squished.

Update bot exclusion lists

After adding many new projects, we need to update bot exclusion lists, some of the most active "developers" are now jenkins and test bots:
https://grpc.cncftest.io/dashboard/db/developers-summary?orgId=1&var-period_name=Last%20decade&var-metric=github_events&var-period=y10

possible to add release timeframes to Quick Ranges?

I want to to use devstats to answer things like:

• how has X changed over the past 3 releases?
• is Y better or worse for the pending release vs. the last release we shipped?
• what is the histogram of people who did Z for the release we're about to ship?

Something that would be helpful is to have Quick Ranges that are defined by specific kubernetes releases, eg:

• specific releases: 1.8 release cycle, 1.7 release cycle
• relative release: last release, last two releases, last three releases, (to now, or to their end)

How can we make answers these sorts of release-date based questions easier?

Suggested approvers stopped working about 12th December 2017

Last data is 11th Dec 2017:
https://k8s.devstats.cncf.io/dashboard/db/suggested-approvers?orgId=1
Workflow changed?
I've run metric SQL manually for (replacing {{from}} and {{to}}):
2017-10-01 - 2017-11-01 --> data is OK
2017-11-01 - 2017-12-01 --> data is OK
2017-12-01 - 2018-01-01 --> data ending at 12th Dec
2018-01-01 - 2018-02-01 --> no data.

SQL is here: https://github.com/cncf/devstats/blob/master/metrics/kubernetes/other_approver.sql

create temp table suggested_approvers as
select distinct i.id as issue_id,
  substring(
    c.body from '(?i)META={"approvers":\["([^"]+)"\]}'
  ) as approver,
  i.dup_repo_name as repo_name
from
  gha_comments c,
  gha_payloads pl,
  gha_issues i
where
  i.is_pull_request = true
  and c.event_id = pl.event_id
  and i.event_id = pl.event_id
  and i.created_at >= '{{from}}'
  and i.created_at < '{{to}}'
  and c.dup_actor_login = 'k8s-merge-robot'
  and c.body like '%APPROVALNOTIFIER%'
  and substring(
    c.body from '(?i)META={"approvers":\["([^"]+)"\]}'
  ) is not null
;

create temp table actual_approvers as
select distinct i.id as issue_id,
  c.dup_actor_login as approver
from
  gha_comments c,
  gha_payloads pl,
  gha_issues i
where
  i.is_pull_request = true
  and c.event_id = pl.event_id
  and i.event_id = pl.event_id
  and i.created_at >= '{{from}}'
  and i.created_at < '{{to}}'
  and c.dup_actor_login not in ('googlebot')
  and c.dup_actor_login not like 'k8s-%'
  and c.dup_actor_login not like '%-bot'
  and c.dup_actor_login not like '%-robot'
  and substring(
    c.body from '(?i)(?:^|\n|\r)\s*/approve\s*(?:\n|\r|$)'
  ) is not null
;

select
  'other_approvers;All;all_suggested_approvers,no_approver,other_approver,suggested_approver' as name,
  round(count(distinct sa.issue_id) / {{n}}, 2) as all_suggested_approvers,
  round(count(distinct sa.issue_id) filter (where aa.issue_id is null) / {{n}}, 2) as no_approver,
  round(count(distinct sa.issue_id) filter (where sa.approver != aa.approver) / {{n}}, 2) as other_approver,
  round(count(distinct sa.issue_id) filter (where sa.approver = aa.approver) / {{n}}, 2) as suggested_approver
from
  suggested_approvers sa
left join
  actual_approvers aa
on
  aa.issue_id = sa.issue_id
union select 'other_approvers;' || r.repo_group || ';all_suggested_approvers,no_approver,other_approver,suggested_approver' as name,
  round(count(distinct sa.issue_id) / {{n}}, 2) as all_suggested_approvers,
  round(count(distinct sa.issue_id) filter (where aa.issue_id is null) / {{n}}, 2) as no_approver,
  round(count(distinct sa.issue_id) filter (where sa.approver != aa.approver) / {{n}}, 2) as other_approver,
  round(count(distinct sa.issue_id) filter (where sa.approver = aa.approver) / {{n}}, 2) as suggested_approver
from
  gha_repos r
join
  suggested_approvers sa
on
  sa.repo_name = r.name
  and r.repo_group is not null
left join
  actual_approvers aa
on
  aa.issue_id = sa.issue_id
group by
  repo_group
order by
  all_suggested_approvers desc,
  name asc
;

drop table suggested_approvers;
drop table actual_approvers;

Any suggestions?

CI?

Would it be possible to use Travis CI or CircleCI to get automated testing?

I note this because I'm willing to help set that up if there's interest.

Introduce metrics about issues

Please show issue filing/closing rates for example. Ideally broken down by kind and SIG.

Approver numbers for k8s dropped sharply

There's no description but assuming this chart is tracking the number of active reviewers (and not number of reviews done):

https://k8s.devstats.cncf.io/dashboard/db/approvers?orgId=1&var-period=q&var-repogroups=All

Kubernetes went from around 100 approvers to 42, any insight on what happened here?

Heptio missing from company stats dashboard

Heptio seems to be missing from:

https://devstats.k8s.io/dashboard/db/companies-stats?orgId=1
https://devstats.k8s.io/dashboard/db/companies-velocity?orgId=1

However if I URL hack I can find our activity:

https://devstats.k8s.io/dashboard/db/companies-stats?orgId=1&var-period=d7&var-metric=activity&var-repogroup_name=All&var-repogroup=all&var-companies=Heptio

Any ideas?

Create playlist based on tags

I just wanted to make sure that you know about the playlist feature. Starting a playlist will cycle dashboards at a set interval. Kinda nifty for tv monitors in the office or a booth at conferences. You can start a playlist by going to a specific url ex http://play.grafana.org/playlists/play/1

DevStats

Devstats at the bottom of https://k8s.devstats.cncf.io/ should be DevStats.

Min+max+avg in legends is a lie

If min+max+avg is enabled in the legend its important to know that the values are in fact the min/max/avg of the values returned from the query.

So if the query sent to Influxdb is aggregated by avg()
min will be the min of the avg values
max will be the max of the avg values
avg will be the correct value.

So the values might be off if you compare them to raw data.

Add support for /lifecycle bot-command

OK this one should be easy :)

A new bot-command /lifecycle was added within the last month or so, and is being used quite a bit. I'd like to see that reflected in the bot-commands dashboard.

I was going to try PR'ing this myself but I'm not sure if I understand the development flow, my guess was I'd need to modify:

• metrics/kubernetes/bot_commands.sql
• metrics/kubernetes/bot_commands_tags.sql
• metrics/kubernetes/gaps.yaml

If I've got an example to follow I may be able to do this for other bot commands we may have missed

Add exporting options

Would it be possible to add an exporting feature to dashboards? If it's a matter of selecting a 'top 10' dashboards of importance to do this, I can select those. Looking for excel, csv, etc. options. json is fine for devs and the like but people using this tool may not have experience with that format. This will allow for others to easily consume the data in the format that they feel comfortable with.

containerd logo

Please use the white logo to make it legible.

https://containerd.devstats.cncf.io/dashboard/db/all-cncf-projects?refresh=15m&orgId=1
https://github.com/cncf/artwork/blob/master/containerd/icon/white/containerd-icon-white.png

Favicon looks good, though.

Query by individual

Querying for company stats exists, but individuals does not. There are multiple workflows where this is useful, but a particular near term need is to measure activity towards membership ladder progression. Devstats appears to offer a more powerful path to insights than looking at a user's activity on a github page.

Adding multiple repo selection to PR Authors Dashboard

I'd like to see the capability to select multiple repositories to the PR Authors Dashboard. Only one at a time can be selected now.
https://devstats.k8s.io/dashboard/db/prs-authors?orgId=1

Approvers and Reviewers histograms should have release milestone information

They seem to have a releases checkbox but for example I can't seem to find "who did the most reviews for 1.8". Am I looking in the right place?

Consider SIGs as a top level dashboard object

I think it'd be a useful tool for the SIGs if the metrics being measured could be made available to them on a SIG level. So for example it'd be nice for SIGs to know how they'd been doing cycle to cycle or perhaps allow people to ascertain which SIGs need more reviewers or more hands on deck, which ones are running smooth, those sorts of things.

"But wait, there's no way for us to tell what which SIGs own which parts of the codebase!" Perhaps we can start encouraging people to start moving towards that model.

People missing from sig-cluster-lifecycle reviewers graph

Checking to see who has reviewed PRs from 1.8 to now, and it's missing at least @timothysc and @luxas

https://devstats.k8s.io/dashboard/db/approvers-histogram?orgId=1&var-period_name=v1.8.0%20-%20now&var-period=anno_9_now&var-repogroup_name=Cluster%20lifecycle&var-repogroup=cluster_lifecycle

Test server links should point to the test server projects

• Links to all test server projects "*.cncftest.io" should point to other projects on the test server.
• Links to all prod server projects "*.devstats.cncf.io" should point to other projects on the prod server.

Combining dashboards

Should we continue the work of #41 should we combine more dashboards?

Lukasz, on a dashboard like https://k8s.devstats.cncf.io/dashboard/db/companies-velocity?orgId=1 is it possible to add single line sections between each graph (like the please report section on the homepage) that would include an A NAME and so clicking on it would produce a URL like https://k8s.devstats.cncf.io/dashboard/db/companies-velocity?orgId=1#companies_active that would jump you to that section?

If so, shall we combine the three companies dashboards? Is there a logical order?

Can we combine the Issues and PRs dashboards?

The 4 SIGs ones?

Anything else?

@spiffxp @parispittman @castrojo Please chime in what you would find useful.

req: issue age dashboard

It would be nice to have an issues-age dashboard, parallel to the prs-age dashboard currently in place

It would be nice, if possible, to be able to filter by: kind, sig, priority

The question I'm trying to answer is: how many auto-filed issues (author:k8s-merge-robot,label:kind/flake,"Failure cluster") have been addressed by humans vs. left untouched?

Add remaining CNCF projects: rkt, CNI, Envoy, Jaeger, Notary, TUF, Rook, All, CNCF

Add remaining CNCF projects:

• rkt
• CNI
• Envoy
• Jaeger
• Notary
• TUF
  With their websites:
• rkt.cncftest.io, rkt.devstats.cncf.io
• cni.cncftest.io, cni.devstats.cncf.io
• envoy.cncftest.io, envoy.devstats.cncf.io
• jaeger.cncftest.io, jaeger.devstats.cncf.io
• noraty.cncftest.io, notary.devstats.cncf.io
• tuf.cncftest.io, tuf.devstats.cncf.io

Contributors by Release

Similar to the Top Comments Dashboard, it would be very beneficial for easily accessible data on who contributed to each release by their handle, including # of PRs, and the repo that they went into. We can then filter by the repo and the release date/number.

Are "All PRs ..." dashboards redundant?

Trying to reduce cognitive load from too many dashboards.

Comparing PRs merged to All PRs merged it seems like the same information is available. Ditto for Need Rebase PRs vs All Need Rebase PRs. I prefer the more detailed non-all variants.

What do folks think about dropping the "All" dashboards? If people use them or prefer it to the more detailed dashboards, could we instead move the "all" graph to be another graph in the non-all dashboards?

Dashboards as code

Some day in the future we really want to improve the experience for people who want to store their dashboards as files primarily rather then storing them in the database primarily. Until then I recommend using https://github.com/weaveworks/grafanalib great way of writing your dashboards are code instead of bloaty json files :)

All Dashboards

Can you get the panel of All CNCF Projects (with their logos) to appear below the list of all dashboards on the homepage?

If not, please add it to the list of all views visible on the homepage.

Categories items on the home page

The items displayed on the home page (https://devstats.k8s.io) are several and they probably going to increase. We may end up with lots of scrolling. This can be confusing. Can we Categories related items e.g. have all issues related items under "Issue" or companies related items under 'Companies' (for example, have 'Companies stats", "Companies summary" and "Companies velocity" under main item "Companies". Thanks!

User Guide

Add to readme or github wiki a DevStats User Guide describing how to use the site and some example workflows to find and interpret data.

Rename self

Please rename independent to self.

Some enhancements and corrections required for - ubuntu16 setup guide.

I am trying to setup a local devstats environment using the instructions provided in:
https://github.com/cncf/devstats/blob/master/INSTALL_UBUNTU16.md

It will be good to include the following:

• Minimum System Requirements and recommended settings (like sysctl) for running postgres and influxdb on the same machine. (Currently I got into an issue where influx db is not starting the http service due to default limits on the openfiles etc. Still trying to get past this stage.)

Also, noticed the following minor items that needs to be updated:

• GOPATH needs to be fixed throughout the doc. For example /data/dev or $HOME/data/dev
• Need Go 1.7+. Using 1.6 default on Ubuntu 16.04 causes error at step : go get github.com/google/go-github/github
• sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_recycle: No such file or directory (not sure if this is expected)
• /etc/github/aoauth -> /etc/github/oauth

Prometheus dashboard has "Kubernetes" mention in graph title

The graph in https://prometheus.cncftest.io/dashboard/db/all-prs-merged?orgId=1&from=now-5y&to=now has the title "PRs merged in all Kubernetes repositories", while it should mention Prometheus instead.

[dashboards] repository groups are unclear

What concrete repositories do the repository groups map to?

Personally I would prefer to have the ability to select individual repositories everywhere I can only use repository groups

Drop down menu of projects

Would it be possible to modify Grafana so that when you click the project logo at the top left, under the Pin option would be entries for every CNCF project (with logo and name). Selecting one would redirect you to that's project's page.

All Need Rebase PR's dashboard is misleading or incorrect

I would expect "All Need Rebase PR's" to reflect the total number of open PR's that have the needs-rebase label applied on any given day

So for example, for kubernetes, I would expect the value for today to match the count returned by this github query which as of writing this issue is 278. The All Need Rebase PR's dashboard shows 0.

Increase size of OpenTracing logo

Please make sure that You follow instructions from CONTRIBUTING

Please increase the size or center the existing logo.

https://opentracing.devstats.cncf.io/dashboard/db/project-statistics?orgId=1&var-period=y10&var-repogroup_name=All&var-repogroup=all

Change default role to readonly editor

I suggest you edit the default anonymous role to Read Only Editor ( http://docs.grafana.org/installation/configuration/#auto-assign-org-role )

This makes it possible for viewer to inspect queries and settings but without giving them the power so save the dashboards. Makes it easier for people like me to find issues or just see how you the build the dashboards.

Missing Company

It appears as though Samsung is missing from the company listings. For example, from the company velocity page:

Yet, Samsung is listed as a company for the velocity metrics. See (under the values):
https://github.com/cncf/devstats/blob/5a08b45d33f4723e2746da45508586291c99481d/metrics/kubernetes/gaps.yaml#L110-L117

Not yet sure what's going on but wanted to report the bug.

When to use bar or line charts.

I guess its a matter of taste but when visualizing time spent and percentiles I think line charts are more explicit then bar charts. ( ex https://cncftest.io/dashboard/db/opened-to-merged ) Bar charts are more suitable when the values can be stacked. (like https://cncftest.io/dashboard/db/companies-stats ) Otherwise the viewer always have to consider if the values are stacked or not.

One company at the bottom

On https://devstats.k8s.io/dashboard/db/companies-stats?orgId=1

Sort by avg. Caicloud is still on the bottom, instead of the top.

Lyft should be #1 company for Envoy, probably missing affiliations, now shows Google

• Please look into why Google is #1 on https://envoy.cncftest.io/dashboard/db/companies-velocity?orgId=1. I expected Lyft to be since the software was created there

Partially blocked - if I have time I'll just run with manually added affiliation for top author.
If I get new affiliations in the meantime, I'll do full reimport of affiliations.

Add "Contributors" and "Contributions" everywhere when you can choose metrics like "Authors", "Issues", ... etc

• Add "Contributors" and "Contributions" everywhere when you can choose metrics like "Authors", "Issues", ... etc.

Add "All" project and a metric for the unique number of people who have made a commit, PR or issue

• Add "all.cncftest.io" project that aggregates data from all CNCF projects
• Add a metric for the unique number of people who have made a commit, PR or issue - especially to be used on 'All' project, but also usable on all other projects

Dashboard item to display open issues and PR for SIGs

In the last ContribX SIG meeting (11/15/2017), we had a discussion about Kubernetes issues that are currently open but should have been closed based on some investigation done by a non-SIG member. Ideally SIG member should quickly close such an issues without spending much time per findings added as comments in the issue. However due to bandwidth limitation it's not always the case. One of the possible solutions was to display Open Issues and PR in the devstats per SIG to give a SIG idea about them. As I was looking at devstats, I guess the good place to display such stats for open issues can be under 'SIG issues' with new box for 'Open issue'? and something similar to PRs. (I am also working on possibilities of new labels to identify close candidates something that can be displayed on devstats in future)

ContribX meeting note for related action item under 11/15/2017:
https://docs.google.com/document/d/1qf-02B7EOrItQgwXFxgqZ5qjW0mtfu5qkYIF1Hl4ZLI/edit?ts=5a0c7fbb

Dashboard for "PR Merged by Companies"

Is it possible to have a dashboard for "PR Merged by Companies" ?

What are events?

On https://devstats.k8s.io/dashboard/db/project-statistics?orgId=1 you show 3333 events.

Does this mean unique people who have done something on GitHub?

If so, event authors would be clearer. Similarly Commits would be Commit authors.

Can we also show the number of actual commits? Since most commits are done by bots, this does not seem like a good metric.

Adjust pagination

20 results per page doesn't make a lot of sense, especially compared to the amount of data loaded in a typical graph. I would suggest paginating to 1000 rows per page or more, to facilitate on-page search (i.e., finding your company).

https://devstats.k8s.io/dashboard/db/prs-authors-companies-histogram?orgId=1&var-period=y&var-repogroup_name=All&var-repogroup=all

Dashboard request: New/Episodic contributors

Here's some graphs I'd like to have:

Top request:

New Contributors: count of the number of PRs and count of unique individuals submitting a PR for the first time this month/week/day in repo history.

Also like to have, but less important:

Episodic Contributors: people/PRs from contributors who have not submitted a PR for a prior 3 months, and not more than 12 PRs ever.

New Issue Filers: count of the number of people and the number of issues from folks who are filing an issue for the first time in their history.

It would also be really nice if all of the above was divisible by SIG label.

Improve Readme

Please link to https://www.githubarchive.org/ and explain the data is there. Please show the approximate sizes of downloads.