maximilien / ghtrack Goto Github PK
View Code? Open in Web Editor NEWA python tool to keep track of GitHub users stats
License: Apache License 2.0
A python tool to keep track of GitHub users stats
License: Apache License 2.0
I'm getting module not found errors. See below
Install and run ghtrack
Install and run ghtrack (on my machine?)
➜ ghtrack git:(master) ./ght version
Traceback (most recent call last):
File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 65, in <module>
from cli import *
File "/Users/joesepi/code/ghtrack/cli.py", line 15, in <module>
import io, sys, yaml, json, csv, os.path
ModuleNotFoundError: No module named 'yaml'
add output of python --version
➜ ghtrack git:(master) python --version
zsh: command not found: python
➜ ghtrack git:(master) python3 --version
Python 3.11.5
➜ ghtrack git:(master) ./ght --help
Traceback (most recent call last):
File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 64, in <module>
from docopt import docopt
ModuleNotFoundError: No module named 'docopt'
➜ ghtrack git:(master) pip3 install docopt
Collecting docopt
Downloading docopt-0.6.2.tar.gz (25 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: docopt
Building wheel for docopt (pyproject.toml) ... done
Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13705 sha256=ebca8400d58cfd5103868c421e8539cda4694d6b95bac7703ace678fd40af17c
Stored in directory: /Users/joesepi/Library/Caches/pip/wheels/1a/b0/8c/4b75c4116c31f83c8f9f047231251e13cc74481cca4a78a9ce
Successfully built docopt
Installing collected packages: docopt
Successfully installed docopt-0.6.2
➜ ghtrack git:(master) ./ght --help
Traceback (most recent call last):
File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 65, in <module>
from cli import *
File "/Users/joesepi/code/ghtrack/cli.py", line 15, in <module>
import io, sys, yaml, json, csv, os.path
ModuleNotFoundError: No module named 'yaml'
./ght stats september knative --commits --issues \
--users=maximilien,octocat \
--all-repos \
--output=text \
--summarize \
--rate-limit-random \
--rl-max=150 \
--rl-sleep=5m
results in
Traceback (most recent call last):
File "./ghtrack.py", line 70, in <module>
rc = command.execute()
File "/Users/maximilien/github/ghtrack/cli.py", line 630, in execute
rc = func()
File "/Users/maximilien/github/ghtrack/cli.py", line 767, in stats
self._update_users_commits()
File "/Users/maximilien/github/ghtrack/cli.py", line 416, in _update_users_commits
self._update_repo_stats('commits', self.users_commits)
File "/Users/maximilien/github/ghtrack/cli.py", line 326, in _update_repo_stats
self.repos_stats[repo_name][data] += user_data_map[repo_name]
KeyError: 'commits'
no errors
issue command above. If you include --show-all-stats
then command executes fine.
./ght --version
GH Track v0.3.5.3
Python 2.7.15
/kind bug
Use existing well-known in memory DB like Pandas instead of custom
/kind proposal
Documentation to show how to run this tool on a public cloud, e.g., IBM Cloud or Google Cloud Knative services
/kind good-first-issue
Current ./ght —help
does not match the README.md
README needs to reflect current tool feature
./ght --version
GitHub Track v0.2
v2.7.15
/kind cleanup
The following command:
/ght stats july knative --users=maximilien --repos=client-contrib -o txt
should return all stats for user 'maximilien' in july in repo 'client-contrib' in the knative organization and displat in standard out.
output
issue command
./ght --version
GH Track v0.3
Python 3.8.2
/kind bug
Documentation to show how to run this tool on a public cloud, e.g., IBM Cloud or Google Cloud Knative services
/kind good-first-issue
Show how to run ght
as a means to collect stats info from a small team of developers and creating charts from data.
Common use case is to show commits, PRs, issues, and reviews data for a team of developers over a month period.
/kind doc
Current data is by user, that is:
...
org year month data state
------- ------ ------- ------- -------
knative 2020 july commits closed
user repo data count
---------- -------------- ------- -------
maximilien client commits 5
maximilien client-contrib commits 0
octocat client commits 0
octocat client-contrib commits 0
...
Would be great to present different summary view, e.g., by data type and by repo. So something like:
repo data count
------------- ------- -------
client commits 5
client-contrib commits 0
and
data repo count
--------- ------ -------
commits client 5
commits client-contrib 1
prs client-contrib 0
reviews client-contrib 10
Avoids having to compute these summary and very useful for team workflows
./ght stats july knative --commits --issues --prs --summarize --users=maximilien,octocat --repos=client,client-contrib -o text --show-all-stats
/kind good-first-issue
➜ ghtrack git:(master) ✗ ./ght stats july knative --users=julz \
--commits --reviews --issues --prs \
--all-repos
Getting 'commits' for 'julz' in organization: 'knative'
[=============-----------------------------------------------] 21.1% ...processing repos
Getting 'prs' for 'julz' in organization: 'knative'
[=============-----------------------------------------------] 21.1% ...processing repos
...
➜ ghtrack git:(master) ✗ ./ght stats july knative --users=julz \
--commits --reviews --issues --prs \
--all-repos
Getting 'commits' for 'julz' in organization: 'knative'
[===============================================] 100% ...processing repos
...
command above
./ght --version
GH Track v0.3
Python 3.8.2
/kind good-first-issue
Any ./ght stats ...
call on knative-sandbox
results in eventual traceback. Seems to be OK for some of the repos but consistently failed on others.
Traceback (most recent call last):---------------------------] 22.6% ...processing repos
File "/Users/maximilien/github/ghtrack/cli.py", line 630, in execute
rc = func()
File "/Users/maximilien/github/ghtrack/cli.py", line 767, in stats
self._update_users_commits()
File "/Users/maximilien/github/ghtrack/cli.py", line 410, in _update_users_commits
commits_count = self.client.commits_count(repo, user, self.start_date(), self.end_date())
File "/Users/maximilien/github/ghtrack/client.py", line 137, in commits_count
for sc in repo.get_stats_contributors():
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Repository.py", line 2834, in get_stats_contributors
headers, data = self._requester.requestJsonAndCheck(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 318, in requestJsonAndCheck
*self.requestJson(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 410, in requestJson
return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 471, in __requestEncode
status, responseHeaders, output = self.__requestRaw(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
[Previous line repeated 965 more times]
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 498, in __requestRaw
response = cnx.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 110, in getresponse
r = verb(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 543, in get
return self.request('GET', url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1322, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 327, in begin
self.headers = self.msg = parse_headers(self.fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 221, in parse_headers
return email.parser.Parser(_class=_class).parsestr(hstring)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/parser.py", line 67, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/parser.py", line 56, in parse
feedparser.feed(data)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 176, in feed
self._call_parse()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 180, in _call_parse
self._parse()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 295, in _parsegen
if self._cur.get_content_maintype() == 'message':
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 594, in get_content_maintype
ctype = self.get_content_type()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 578, in get_content_type
value = self.get('content-type', missing)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse
return self._sanitize_header(name, value)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header
if _has_surrogates(value):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/utils.py", line 57, in _has_surrogates
s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python object
No errrors
Any stats
call
./ght --version
GH Track v0.3.5.3
Python 2.7.15
/kind bug
The "new" GraphQL API allows pulling commits / PRs by user (and can probably also pull reviews if the right parameters are passed):
https://docs.github.com/en/graphql/reference/objects#user
This should significantly reduce the number of API calls needed, which in theory should also speed up the data pulls.
Provide a means to slow down API requests to avoid rate limit errors
Some ght queries take a long time and results in 100s of API requests which will end up in rate limit errors. Provide a way to slow down (sleep) after some number of API requests.
./ght stats july knative --commits --issues --reviews \
--rate-limit -rt-max 100 --rt-sleep 30m \
--summarize --users=maximilien,octocat \
--repos=client,client-contrib \
--show-all-stats -o csv -f out.csv
By specifying options: --rate-limit -rt-max 100 --rt-sleep 30m
then the request to ght
will attempt to do some rate limiting by sleeping for 30 minutes after every 100 API requests.
Currently the ./hack/build.sh --e2e
take more than 5 minutes to run
Should run under a minute or faster.
./hack/build.sh --e2e
./ght --version
GitHub Track v0.2.2
Python 3.8.2
/kind cleanup
Currently output is at the command line
Generate output into CSV or JSON files
./ght reviews July knative —users=maximilien —repos=client,client-contrib —output=out.csv
/kind cleanup
./ght stats october knative -o text --filename=fake.txt ...
...
wrote output file: fake.txt
OK
✅ Succes
But file is not created
Should ignore --filename
for text output or create the file
see above
./ght --version
GH Track v0.3.5.3
Python 2.7.15
/kind good-first-issue
/kind bug
All calls to the $ght reviews ...
command take a long long time and typically timeout with rate limit errors. Even when using —rate-limit
flags
Executes without errors
Any calls with $ght reviews ...
./ght --version
GitHub Track v0.4.x
3.x
/kind bug
Use GH action or cron to allow users to run this on their own repos to collect data periodically
/kind proposal
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.