snyk-tech-services/snyk-scm-contributors-count

Count distinct contributor of Snyk watched repos across several SCM

Snyk SCM Contributors counting

This repository is in maintenance mode, no new features are being developed. Bug & security fixes will continue to be delivered. Open source contributions are welcome for small features & fixes (no breaking changes)

This tool is used to count contributors with commits in the last 90 days in repositories matching the following criteria:

1. Repo name XYZ (single repo mode if available for SCM command - see help)
2. All repos in given projects/org/group (ex: Bitbucket Server project(s))
3. All repos in given projects/org/group (ex: Bitbucket Server project(s)) AND monitored by Snyk
4. All repos in SCM (varies a little depending on SCM)
5. All repos in SCM (varies a little depending on SCM) AND monitored by Snyk

Example

I want to know the countributors count for Snyk monitored projects in bitbucket server project key AN.

Installation

Require node 14 !

npm i -g snyk-scm-contributors-count

or use corresponding binaries in the release page

Link to full documetation

Snyk Docs

Usage

For Azure, Bitbucket Cloud and Bitbucket Server, you need to export your Snyk Token or apply the --skipSnykMonitoredRepos flag:

export SNYK_TOKEN=<YOUR-SNYK-TOKEN>
snyk-scm-contributors-count <command> <command-options>

If using binaries, chmod +x snyk-scm-contributors-count-<env> to allow execution If you can't type snyk-scm-contributors-count in your terminal, then add you bin folder to the PATH in ~/.bash_profile or equivalent. If using nvm, make sure the bin folder for your node version is also in the PATH.

For running the executable on Windows, you need to set the value of the env variable SNYK_TOKEN first

set SNYK_TOKEN=<YOUR-SNYK-TOKEN>
snyk-scm-contributors-count <command> <command-options>

Examples:

snyk-scm-contributors-count bitbucket-server --token BITBUCKET-TOKEN --url http://bitbucket-server.mycompany.com --projectKeys Key1,Key2 --exclusionFilePath=./snyk.exclude

snyk-scm-contributors-count bitbucket-cloud --user USERNAME --password PASSWORD --workspaces Workspace1,Workspace2 --repo Repo --skipSnykMonitoredRepos

snyk-scm-contributors-count azure-devops --token AZURE-TOKEN --org AZURE-ORG --projectKeys ProjectKey1 --json

snyk-scm-contributors-count gitlab --token TOKEN --url URL --exclusionFilePath PATH_TO_FILE --json

snyk-scm-contributors-count github --token TOKEN --orgs ORG --repo REPO

snyk-scm-contributors-count github-enterprise --token TOKEN --orgs ORG1,ORG2

Walkthroughs:

• Azure-Devops
• Bitbucket-Server
• Bitbucket-Cloud
• Gitlab
• Github
• Github-enterprise

Common options across commands

• --exclusionFilePath pointing to snyk.exclude file, simple text file containing emails of committers to exclude (i.e [email protected], etc...)
• --json output JSON

Additional options might be available depending on the command

Common options across the Bitbucket Cloud, Bitbucket server and Azure Devops commands

• --skipSnykMonitoredRepos to skip checking with repos that are monitored by Snyk (useful for sizing before Snyk rollout). In that case the SNYK_TOKEN is not required (This flag is auto-applied to the Gitlab command)

Run in DEBUG MODE

Use DEBUG=snyk* env var before your command, for example:

DEBUG=snyk* snyk-scm-contributors-count bitbucket-server --token BITBUCKET-TOKEN --url http://bitbucket-server.mycompany.com --projectKeys Key1,Key2 --exclusionFilePath=./snyk.exclude

Run in DEBUG MODE (On WINDOWS)

Set the value of env var as DEBUG=snyk* before your command, for example:

set DEBUG=snyk*
snyk-scm-contributors-count bitbucket-server --token BITBUCKET-TOKEN --url http://bitbucket-server.mycompany.com --projectKeys Key1,Key2 --exclusionFilePath=./snyk.exclude

Development

Add a command and SCM support

1. Create a new ts file under cmds (duplicate cmds/bitbucket-server.ts)

2. Fill out command, desc, and builder options, leaving in:

• exclusionFilePath
• json
• skipSnykMonitoredRepos

3. The handler function will be called with argv which should match the builder options

4. Create a class with your command name extending SCMHandlerClass. It'll require you to implement the abstract method fetchSCMContributors expecting a Promise<ContributorMap> in return

• types can be function in src/lib/types.ts

5. Once create and asbtract function implemented, in handler, instantiate the class you just created

6. Call

<classInstance>.scmContributorCount(argv.url,SourceType["YOUR-SOURCE"],argv.skipSnykMonitoredRepos,argv.exclusionFilePath,argv.json)

7. profit.

Build

npm run build

or in watch mode

npm run build-watch

Best Practices

• Most SCMs have paginated results, fetch all the relevant pages, only what's useful
• Be gentle with rates against SCM. Use client or throttling libs like bottleneck
• Snyk API interaction is using snyk-api-ts-client with built-in throttling and retries

Note of rate and other limitations

• Please be aware that all the SCMs have an API rate limit control. The tool takes that into account.

Release

Releases rely on semantic-release so pull requests that are merged to main with a properly formatted commit message will trigger a new stable release on NPM.

If changes need testing before being included in a stable release version, an alpha pre-release can be rolled out by merging to the alpha branch. Alpha pre-release versions use this format: [major].[minor].[patch]-alpha.[x].

snyk-scm-contributors-count's Issues

Change --token command-line option into an environment variable

Please consider removing the --token command-line option (providing the SCM access token) and changing it to use an environmental variable.
As such the token is visible in "ps" listings and similar.
https://github.com/snyk-tech-services/snyk-repo-diff uses example-secrets.sh for this reason.

Add calcualation for locally cloned repositories

I don't know if this is something that's needed generally, but I'm leaving this here for posterity. At least a one-liner documenting the calculation logic would be convenient if found in the README.md. For example like this, which calculates the amount of contributors in all repositories under and including the current working directory:
find . -name ".git" | xargs -I {} git --git-dir {} log --since="90 days ago" --pretty=format:%ae | sort | uniq | wc -l

With this oneliner for example, one can clearly see that you only calculate contributors on the default branch, not all branches.

But a caveat would be git submodules; in your current calculation logic you do not calculate code contributors in submodules within a repository, although that oneliner would do so..

Return error when SNYK_TOKEN is not set

It would be nice if the script at least warns about SNYK_TOKEN not being set.
It is mentioned in the docs, but took me quite some time to find it back after not having used the script.

README documentation: add brackets for project names

the document does not mention that when there are spaces in a project name, brackets would be required
worth adding to the readme

Bitbucket Cloud - count UUIDs instead of email-addresses

Counting the unique email-addresses attached to commits doesn't seem to be a reliable method of counting active users in our case.

As an example; we have a user that has failed to set his/her default email-address (git config --global user.email).
That user is counted 20+ times because Git defaults to <user>@<hostname>, resulting in:

...
  snyk:count-output     '<user>@ip-XXX-XX-109-8.XX-XXXX-X.compute.internal' => {
  snyk:count-output       email: '<user>@ip-XXX-XX-109-8.eu-west-1.compute.internal',
  snyk:count-output       contributionsCount: 2,
  snyk:count-output       reposContributedTo: [Array]
  snyk:count-output     },
  snyk:count-output     '<user>@ip-XXX-XX-109-248.XX-XXXX-X.compute.internal' => {
  snyk:count-output       email: '<user>@ip-XXX-XX-109-248.XX-XXXX-X.compute.internal',
  snyk:count-output       contributionsCount: 1,
  snyk:count-output       reposContributedTo: [Array]
  snyk:count-output     },
  snyk:count-output     '<user>@ip-XXX-XX-109-244.XX-XXXX-X.compute.internal' => {
  snyk:count-output       email: '<user>@ip-XXX-XX-109-244.XX-XXXX-X.compute.internal',
  snyk:count-output       contributionsCount: 2,
  snyk:count-output       reposContributedTo: [Array]
...

I've created a PR that uses the user account's UUID that is available in the Bitbucket Cloud API and will attach that to this issue :)

retrieveMonitoredRepos and retrieveOrgsAndIntegrations may return incomplete results

Similar issue as #84.

retrieveMonitoredRepos and retrieveOrgsAndIntegrations in https://github.com/snyk-tech-services/snyk-scm-contributors-count/blob/main/src/lib/snyk/index.ts are calling /orgs endpoint with default limit which may not return all Organizations and thus results are incomplete. Also pagination is not supported by these functions currently.

Probably a similar fix as merged from #86 should be implemented to fix this.

Bitbucket cloud 401 not authorized

Hi,

I'm facing issue when try to run script in my bitbucket cloud and return 401, like this:

I could see that bitbucket.org deprecated in your API auth using user and pass in your api.(https://bitbucket.org/blog/deprecating-atlassian-account-password-for-bitbucket-api-and-git-activity)

Incomplete results due to default page size in retrieveMonitoredReposBySourceType

The retrieveMonitoredReposBySourceType in Snyk\Index.js does not specify a result limit or do any paging on the underlying Snyk REST API that retrieves monitored repos. This causes incomplete results when counting contributors, as it is only going back to count contributors in the source control for the first 10 monitored repos; 10 being the default page size of the underlying Snyk API.

The following is used to call the Snyk API (line 117):

/orgs/${orgs[i].id}/targets?origin=${SourceType[sourceType]}&version=${snykApiVersion}

Without a &limit parameter or paging logic, it will return only 10 results.

Slack test

integration test

Azure Devops default URL

Azure Devops default URL is set to cloud, no local TFS option

Azure DevOps - Unescaped Characters

When running this script on ADO, we get the following errors.

Job -yu5c341lgdj failed: TypeError [ERR_UNESCAPED_CHARACTERS]: Request path contains unescaped characters
Retrying job -yu5c341lgdj in 25ms!
Job -yu5c341lgdj failed: TypeError [ERR_UNESCAPED_CHARACTERS]: Request path contains unescaped characters
snyk:azure-devops-count Failed to retrieve repo list from Azure Devops.
snyk:azure-devops-count TypeError [ERR_UNESCAPED_CHARACTERS]: Request path contains unescaped characters +0ms
Failed to retrieve repo list from Azure Devops. Try running with DEBUG=snyk* snyk-contributor
snyk:azure-devops-count Fetching single repo contributor from Azure Devops. Project d38f51da-cd00-4a89-9721-ef5b5a5a500c - Repo

GitHub module ignores contributors with private email

When contributing code using the web interface GitHub can hide their real email and instead [email protected] is used

[email protected] is also supported by GitHub - https://github.blog/2017-04-11-private-emails-now-more-private/

Both email forms can be used from any git tool as well.

In current version these contributions are not counted

snyk-scm-contributors-count/src/lib/github/github-contributors.ts

Line 140 in d4e67c5

!commit.commit.author.email.endsWith('@users.noreply.github.com') &&