sonarsource/sonar-loc-count

LoC Counting Scripts

This is a collection of shell scripts that demonstrate how to count lines of code from repositories and/or local directories.These scripts can be used to estimate LoC counts that would be produced by a Sonar analysis of these projects, without having to implement this analysis.

These scripts that connect to a DevOps platform must be run in a blank workspace (without data).

To exclude directories or files from the scan, you can insert the names of these exclusions into the .clocignore file in the environment where you run the scripts. Example :

bootstrap

test

• Installation
• General usage
• Contributions and Feedbacks

Installation

Requirements:

• bash version 4+
• Git
• curl
• jq
• cloc installed v1.96
• For Mac OSX you need gnu-sed (brew install gnu-sed)

General usage

Most scripts will produce two reports of LoC by language (.lang) and by repository (.file).

github.com:

Counts lines of code from a GitHub.com organization. Requires to pass username, personal access token and the organization. The token must have repo scope.The script generates a report per project(File : ProjectName.txt) that indicates the number of lines of code per branch and indicates the branch that has the highest number of lines of code.As well as a Report_global.txt file that indicates the maximum line of code on the repository.

<github_com.sh> <user> <token> <organization>
github_com.sh myuser 1234567890abcdefgh myGitHubDotComOrg

or

<github_com.sh> <user> <token> <organization> <MyRepoName>
github_com.sh myuser 1234567890abcdefgh myGitHubDotComOrg MyRepoName

bitbucket.org:

Counts lines of code from a Bitbucket.org organization. Requires to pass username, App token password and the workspace slug. The token must have Repositories Read permissions.The script generates a report per project(File : ProjectName.txt) that indicates the number of lines of code per branch and indicates the branch that has the highest number of lines of code.As well as a Report_global.txt file that indicates the maximum line of code on the repository.

<bitbucket_org.sh> <user> <PassordToken> <myWorkspace>
bitbucket_org.sh myuser 1234567890abcdefgh myBBWorkspace

or

<bitbucket_org.sh> <user> <assordToken> <myWorkspace> <MyProjectName>
<bitbucket_org.sh> myuser 1234567890abcdefgh myBBWorkspace MyProjectName

If you have more than 100 repos , you need to change Value of parameter page=Number_of_page on line 53

   1 Page = 100 repos max
   Example for 150 repos : GetAPI="repositories/$wks?pagelen=100&page=2"

Azure DevOps Services:

Counts lines of code from a Azure DevOps Services organization. Requires to pass personal access token and the organization. The token must have Code > Read permissions. The script generates a report per project(File : ProjectName.txt) that indicates the number of lines of code per branch and indicates the branch that has the highest number of lines of code.As well as a Report_global.txt file that indicates the maximum line of code on the repository.

<azure_devops_services.sh> <token> <organization>
azure_devops_services.sh 1234567890abcdefgh myADOOrg 

or

<azure_devops_services.sh> <token> <organization> <MyProjectName>
azure_devops_services.sh 1234567890abcdefgh myADOOrg MyProjectName

Gitlab.com:

Counts lines of code from a GitLab.com Group or Project. Requires to pass personal access token and the group. The token must have read_api and read_repository scopes.The script generates a report per project(File : ProjectName.txt) that indicates the number of lines of code per branch and indicates the branch that has the highest number of lines of code.As well as a Report_global.txt file that indicates the maximum line of code on the repository.

<gitlab_com.sh> <token> <groupName>
gitlab_com.sh 1234567890abcdefgh myGitLabGroupName

or

<gitlab_com.sh> <token> <groupName//MyProjectName> 
gitlab_com.sh 1234567890abcdefgh myGitLabGroupName/MyProjectName

If you have more than 100 repos , you need to change Value of parameter page=Number_of_page on line 58 or 61

   1 Page = 100 repos max
   Example for 150 repos :  GetAPI="/projects/$groupname1?per_page=100&page=2"

Local Filesystem:

Counts lines of code from a local directory or file. The script generates a report file : Report_Name-of-Directory.txt

<filesystem.sh> PathToDirectory

Contributions and feedbacks

Contributions and feedbacks are welcome, as PRs or issues directly with this repository, or through your established Sonar communication channel.

sonar-loc-count's Issues

Cannot loop through a hierachy of groups within Gitlab

When having a hierarchy of groups within Gitlab, the script doesn't work.
Hint : Use full_path instead of path

powershell filesystem script has a minor error

powershell does not use || for or like bash or cmd. you want to replace || with -or. I just had to do this to get the filesystem.ps1 script to work on windows.

Error:

PS C:\Users\<user>\sonar-loc-count\PowerShell> .\filesystem.ps1 "C:\Users\<user>\code\" "C:\Users\<user>\tools\cloc.exe"
At C:\Users\<user>\sonar-loc-count\PowerShell\filesystem.ps1:26 char:34
+         if((Test-Path $CLOCPATH) || (Test-Path ($CLOCPATH+".exe"))) {
+                                  ~~
The token '||' is not a valid statement separator in this version.
    + CategoryInfo          : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : InvalidEndOfLine
    ```

Couting LoC from AWS CodeCommit

Git ignore output files

I see a few issues in the azuredevops.ps1

The tale is not correct. Looks like the clocmax is done at the project level and not at the branch level.
The sorting is done alphabetically because the object is a string and not a decimal.
The script doesn't remove the repo folders.

bitbucket app password token not rendering repo list

Seems that the personal bitbucket app token is not enough to run this script.
The script fails by not creating a cpt.txt file.

./bitbucket_org.sh: line 165: cpt.txt: No such file or directory
rm: cpt.txt: No such file or directory

-------------------------------------------------------------------------------------------
The maximum lines of code on the repository is : <  0 > result in <Report_global.txt>

but the problem is that the repo list is not populated by the curl call. The token is valid and the call succeeded. I reproduced the call using Postman and I will attach a trace of it. The token has all the read permissions and I am an admin on the account.

https://gist.github.com/valugi/270a9f927e33753a9d2192e7d3624e0c

Any recommendations?

Refactor the scripts for more modularity

All our DevOps platform scripts are unstructured.
Let's change that so that we can reuse the common functions and think about multi-threading some parts of it.

azure_devops_services.sh fails for projects with a space in their name

Running azure_devops_services.sh against a project with a space in the name leads to no detected branches and thus no actual counting. This issue is that line 81 should use $name1 rather than $Name.

Patch attached, I can create a PR if you like but for such a simple fix it seems like overkill.

Unable to read sonar-lang-defs.txt file

After each repo it gives me the error "Unable to read sonar-lang-defs.txt." and at the end, after the line that says "Building final report:" I get the same error. I am using the Azure DevOps file. Please help.

Rate limiting on github.com

curl -s -u github_pat_xxxx: https://api.github.com/repos/my_org/my_repo/branches
gives

{                                                                                                        
  "message": "API rate limit exceeded for $MY_IP. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",                               
  "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"                                                                                                             
}

I had to change the curl to
curl -s --header "Authorization: Bearer github_pat_xxxx" https://api.github.com/repos/my_org/my_repo/branches

Can the script use an authentication method with a high rate limit?

Should default language exclusions be applied?

example /vendor/ for GO
Default library folders for JS/TS.

github_com.sh only counting LOC for up to 30 repositories

Hi,

The default API used in github_com.sh (https://api.github.com/orgs/${org}/repos) will only list up to 30 repositories.
Using https://api.github.com/orgs/${org}/reposrepos?per_page=100 we can get up to 100.

I've ended up using Github cli, gh repo list ${org} -L 1000 --json name,defaultBranchRef, it gives us up to 1000.

If folder already exists, ask for confirmation on exit with error, prevent data loss in cleanup

I lost precious data when i used github script and i had same named folders in the folder i ran the script in. As it tried to clone in same folder, gave warning but didn't stop or warn that data can be lost.

Ended up loosing some of my data.

Lots o' Errorz

Hi there, Sonar sent us this tool in order to help us evaluate the product as a potential customer. However I'm so far unable to get it to work in ubuntu. Would love some assistance with what I might be missing.

Git clone for each branch bloating time to run

Running git clone <repo> --branch <branch> for large repos with a high number of branches causes the script to take several hours to complete and high network consumption. Whereas checking out the branches from a single clone would work just as well.

GitHub example does not handle 403 return

When the call to get repos from GitHub reruns a 403, the example should exit gracefully

Add some multi-threading

For large organizations, running the script can be very long.
Let's consider running some parts of it in parallel.

The best first candidate might be repository cloning.

Target only the main repo branches by default

In most cases, there is no need to evaluate the LoC counts of every branch of every project as the variations should be very minor, and well within the error margin for this script.
Let's make the simpler and faster behavior the default one.

Improve Documentation

• Clarify the technical requirements (generally) in README
• Add more comments to scripts to allow better understanding of what is happening in each step so they will be easier to modify

cloc issues with solar-lang-defs.txt file extension duplications.

The solar-lang-defs have upper and lowercase file extensions, and with the newest version of cloc it will throw a fatal error.

The fix would be to remove the duplicated lines.

Just a FYI - I managed to fix it locally but i'm not sure if cloc would only count the lowercase extensions now.

I've used the packed version (.exe) of cloc for windows.

Using LoC on Azure DevOps Services always showing 0 lines of code

I am trying to run the Azure DevOps script in my organization. The script is able to read the name of the repositories but not the branch names. Also for all the repos the count is 0.

Here is an example of a repo report:

---

The maximum lines of code in the < Atlantis.AI.SPO.EditorSurface > project is : < 0 > for the branch : < No >

---

The maximum lines of code in the < Atlantis.AI.SPO.EditorSurface > project is : < 0 > for the branch : < No >

---

The maximum lines of code in the < Atlantis.AI.SPO.EditorSurface > project is : < 0 > for the branch : < No >

I have installed all the installation pre-requisites and I'm running the following command:
./azure_devops_services.sh <my_personal_token> <my_organization>

My personal token has the required permissions Code > Read.

Check prerequisites at startup

Should check the presence of jq, curl, cloc (including version), sed (or gsed if MacOS), and git according to the script being called

Include Support for Bitbucket Server / Datacenter

Would be nice to have an adapted version of the script that is able to determine LoC in Bitbucket Onprem Solutions ( Server / Datacenter ).

Probably these steps are necessary:

• Check if there are changes in API
• No workspaces exist in server
• Base URL can be changed in file or via argument