maximilien/ghtrack

A python tool to keep track of GitHub users stats

ghtrack's Issues

Cannot run due to missing modules

Bug report

I'm getting module not found errors. See below

Expected behavior

Install and run ghtrack

Steps to reproduce the problem

Install and run ghtrack (on my machine?)

./ght version

➜  ghtrack git:(master) ./ght version
Traceback (most recent call last):
  File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 65, in <module>
    from cli import *
  File "/Users/joesepi/code/ghtrack/cli.py", line 15, in <module>
    import io, sys, yaml, json, csv, os.path
ModuleNotFoundError: No module named 'yaml'

Python version

add output of python --version

➜  ghtrack git:(master) python --version
zsh: command not found: python
➜  ghtrack git:(master) python3 --version
Python 3.11.5

More output from my issue

➜  ghtrack git:(master) ./ght --help
Traceback (most recent call last):
  File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 64, in <module>
    from docopt import docopt
ModuleNotFoundError: No module named 'docopt'
➜  ghtrack git:(master) pip3 install docopt
Collecting docopt
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: docopt
  Building wheel for docopt (pyproject.toml) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13705 sha256=ebca8400d58cfd5103868c421e8539cda4694d6b95bac7703ace678fd40af17c
  Stored in directory: /Users/joesepi/Library/Caches/pip/wheels/1a/b0/8c/4b75c4116c31f83c8f9f047231251e13cc74481cca4a78a9ce
Successfully built docopt
Installing collected packages: docopt
Successfully installed docopt-0.6.2
➜  ghtrack git:(master) ./ght --help                                                                                                        
Traceback (most recent call last):
  File "/Users/joesepi/code/ghtrack/./ghtrack.py", line 65, in <module>
    from cli import *
  File "/Users/joesepi/code/ghtrack/cli.py", line 15, in <module>
    import io, sys, yaml, json, csv, os.path
ModuleNotFoundError: No module named 'yaml'

Traceback when --show-all-stats not used

Bug report

./ght stats september knative --commits --issues \
						--users=maximilien,octocat \
						--all-repos \
						--output=text \
						--summarize \
						--rate-limit-random \
						--rl-max=150 \
						--rl-sleep=5m

results in

Traceback (most recent call last):
  File "./ghtrack.py", line 70, in <module>
    rc = command.execute()
  File "/Users/maximilien/github/ghtrack/cli.py", line 630, in execute
    rc = func()
  File "/Users/maximilien/github/ghtrack/cli.py", line 767, in stats
    self._update_users_commits()
  File "/Users/maximilien/github/ghtrack/cli.py", line 416, in _update_users_commits
    self._update_repo_stats('commits', self.users_commits)
  File "/Users/maximilien/github/ghtrack/cli.py", line 326, in _update_repo_stats
    self.repos_stats[repo_name][data] += user_data_map[repo_name]
KeyError: 'commits'

Expected behavior

no errors

Steps to reproduce the problem

issue command above. If you include --show-all-stats then command executes fine.

./ght version

./ght --version
GH Track v0.3.5.3

Python version

Python 2.7.15

/kind bug

Switch the backend to Pandas

Feature request

Use existing well-known in memory DB like Pandas instead of custom

/kind proposal

Show how to run ght on public Knative services

Feature request

Documentation to show how to run this tool on a public cloud, e.g., IBM Cloud or Google Cloud Knative services

/kind good-first-issue

docs are not in sync with the current code features

Bug report

Current ./ght —help does not match the README.md

Expected behavior

README needs to reflect current tool feature

./ght version

./ght --version
GitHub Track v0.2

Python version

v2.7.15

/kind cleanup

ght stats ... command not complete

Bug report

The following command:

/ght stats july knative --users=maximilien --repos=client-contrib -o txt

should return all stats for user 'maximilien' in july in repo 'client-contrib' in the knative organization and displat in standard out.

Expected behavior

output

Steps to reproduce the problem

issue command

./ght version

./ght --version
GH Track v0.3

Python version

Python 3.8.2

/kind bug

Show how to run this tool on a public cloud

Feature request

Documentation to show how to run this tool on a public cloud, e.g., IBM Cloud or Google Cloud Knative services

/kind good-first-issue

Using ght to collect periodic data for a small team of developers on a set of repos

Feature request

Show how to run ght as a means to collect stats info from a small team of developers and creating charts from data.

Use case

Common use case is to show commits, PRs, issues, and reviews data for a team of developers over a month period.

/kind doc

Add --summarize flag to present a summary of the data

Feature request

Current data is by user, that is:

...
org        year  month    data     state
-------  ------  -------  -------  -------
knative    2020  july     commits  closed

user        repo            data       count
----------  --------------  -------  -------
maximilien  client          commits        5
maximilien  client-contrib  commits        0
octocat     client          commits        0
octocat     client-contrib  commits        0
...

Would be great to present different summary view, e.g., by data type and by repo. So something like:

repo                 data   count
-------------  ------- -------
client        commits 5
client-contrib commits 0

and

data          repo                count
---------  ------              -------
commits   client               5
commits   client-contrib 1
prs            client-contrib  0
reviews    client-contrib  10

Use case

Avoids having to compute these summary and very useful for team workflows

UI Example

./ght stats july knative --commits --issues --prs --summarize --users=maximilien,octocat --repos=client,client-contrib -o text --show-all-stats

/kind good-first-issue

Progress bar when collecting stats for a user on all repos does not seem to get to 100%

Bug report

➜  ghtrack git:(master) ✗ ./ght stats july knative --users=julz \
                       --commits --reviews --issues --prs \
                       --all-repos
Getting 'commits' for 'julz' in organization: 'knative'
[=============-----------------------------------------------] 21.1% ...processing repos
Getting 'prs' for 'julz' in organization: 'knative'
[=============-----------------------------------------------] 21.1% ...processing repos
...

Expected behavior

➜  ghtrack git:(master) ✗ ./ght stats july knative --users=julz \
                       --commits --reviews --issues --prs \
                       --all-repos
Getting 'commits' for 'julz' in organization: 'knative'
[===============================================] 100% ...processing repos
...

Steps to reproduce the problem

command above

./ght version

./ght --version
GH Track v0.3

Python version

Python 3.8.2

/kind good-first-issue

Traceback when collecting commits in `knative-sandbox`

Bug report

Any ./ght stats ... call on knative-sandbox results in eventual traceback. Seems to be OK for some of the repos but consistently failed on others.

Traceback (most recent call last):---------------------------] 22.6% ...processing repos
  File "/Users/maximilien/github/ghtrack/cli.py", line 630, in execute
    rc = func()
  File "/Users/maximilien/github/ghtrack/cli.py", line 767, in stats
    self._update_users_commits()
  File "/Users/maximilien/github/ghtrack/cli.py", line 410, in _update_users_commits
    commits_count = self.client.commits_count(repo, user, self.start_date(), self.end_date())
  File "/Users/maximilien/github/ghtrack/client.py", line 137, in commits_count
    for sc in repo.get_stats_contributors():
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Repository.py", line 2834, in get_stats_contributors
    headers, data = self._requester.requestJsonAndCheck(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 318, in requestJsonAndCheck
    *self.requestJson(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 410, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 471, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
    return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
    return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 515, in __requestRaw
    return self.__requestRaw(original_cnx, verb, url, requestHeaders, input)
  [Previous line repeated 965 more times]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 498, in __requestRaw
    response = cnx.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/github/Requester.py", line 110, in getresponse
    r = verb(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests-2.23.0-py3.8.egg/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3-1.25.9-py3.8.egg/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 327, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 221, in parse_headers
    return email.parser.Parser(_class=_class).parsestr(hstring)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/feedparser.py", line 295, in _parsegen
    if self._cur.get_content_maintype() == 'message':
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 594, in get_content_maintype
    ctype = self.get_content_type()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse
    return self._sanitize_header(name, value)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header
    if _has_surrogates(value):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/email/utils.py", line 57, in _has_surrogates
    s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python object

Expected behavior

No errrors

Steps to reproduce the problem

Any stats call

./ght version

./ght --version
GH Track v0.3.5.3

Python version

Python 2.7.15

/kind bug

Update to Github GraphQL API

The "new" GraphQL API allows pulling commits / PRs by user (and can probably also pull reviews if the right parameters are passed):

https://docs.github.com/en/graphql/reference/objects#user

This should significantly reduce the number of API calls needed, which in theory should also speed up the data pulls.

Add a way to rate limit since some queries take time and will result in GitHub rate limit errors

Feature request

Provide a means to slow down API requests to avoid rate limit errors

Use case

Some ght queries take a long time and results in 100s of API requests which will end up in rate limit errors. Provide a way to slow down (sleep) after some number of API requests.

UI Example

./ght stats july knative --commits --issues --reviews \
                                       --rate-limit -rt-max 100 --rt-sleep 30m \
                                       --summarize --users=maximilien,octocat \
                                       --repos=client,client-contrib \
                                       --show-all-stats -o csv -f out.csv

By specifying options: --rate-limit -rt-max 100 --rt-sleep 30m then the request to ght will attempt to do some rate limiting by sleeping for 30 minutes after every 100 API requests.

e2e tests take too long to run

Bug report

Currently the ./hack/build.sh --e2e take more than 5 minutes to run

Expected behavior

Should run under a minute or faster.

Steps to reproduce the problem

./hack/build.sh --e2e

./ght version

./ght --version
GitHub Track v0.2.2

Python version

Python 3.8.2

/kind cleanup

Output in CSV, YML, and JSON files

Feature request

Currently output is at the command line

Use case

Generate output into CSV or JSON files

UI Example

./ght reviews July knative —users=maximilien —repos=client,client-contrib —output=out.csv

/kind cleanup

Calling ght with `-o text` and `--filename` prints file created when it's not

Bug report

./ght stats october knative -o text --filename=fake.txt ...

...
wrote output file: fake.txt
OK
✅ Succes

But file is not created

Expected behavior

Should ignore --filename for text output or create the file

Steps to reproduce the problem

see above

./ght version

./ght --version
GH Track v0.3.5.3

Python version

Python 2.7.15

/kind good-first-issue
/kind bug

$ght reviews ... takes a long long time

Bug report

All calls to the $ght reviews ... command take a long long time and typically timeout with rate limit errors. Even when using —rate-limit flags

Expected behavior

Executes without errors

Steps to reproduce the problem

Any calls with $ght reviews ...

./ght version

./ght --version
GitHub Track v0.4.x

Python version

3.x

/kind bug

Automate running with GH action

Feature request

Use GH action or cron to allow users to run this on their own repos to collect data periodically

/kind proposal