GitHub Elasticsearch Analytics
This node.js app will crawl the GitHub issues and comments API to then save them in Elasticsearch.
Install
git clone https://github.com/grafana/github-to-es.git
cd github-to-es
npm install
Configure
Copy config.sample.json to config.json.
{
"github_token": "mytoken",
"elasticsearch": {
"host": "localhost",
"port": 9200
},
"repos": [
{
"repo": "grafana/grafana-docker",
"comments": true,
"issues": true,
"sinceDays": 2
}
]
}
Specify your Elasticsearch details (no auth options available at this point). For the repository entries you can specify if comments and/or issues should be fetched. If you want a complete (from the start) import of all issues and comments remove the sinceDays option or set it to 0. After the complete import is done you can do incremental updates by setting this to 1.
Init & Create Elasticsearch index
node app.js init | bunyan
The above command will create an Elasticsearch index named github
. The | bunyan
part is optional. It's to get nicer console logging (instead of the default json logger). To use bunyan install
it first using npm install -g bunyan
.
You can reset (remove and recreates) the index using:
node app.js reset | bunyan
Start import
node app.js start | bunyan
Add Elasticsearch Data source in Grafana
When you add the data source specify github
as index name and created_at
in the Timestamp field.
Grafana Dashboards
Limitations & Possible improvements
Currently GitHub API limits the number of pages you can fetch to 400. So there is a limit for the initial complete import of issues & comments to 40000 issues and 40000 comments.
It would be nice to get stars & other repo stats as well.