hhatto/gocloc

A little fast cloc(Count Lines Of Code)

gocloc

A little fast cloc(Count Lines Of Code), written in Go. Inspired by tokei.

Installation

require Go 1.19+

$ go install github.com/hhatto/gocloc/cmd/gocloc@latest

Arch Linux user can also install from AUR: gocloc-git.

Usage

Basic Usage

$ gocloc .

$ gocloc .
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                         3              8              0             18
Go                               1             29              1            323
-------------------------------------------------------------------------------
TOTAL                            4             37              1            341
-------------------------------------------------------------------------------

Via Docker

with dockerhub

$ docker run --rm -v "${PWD}":/workdir hhatto/gocloc .

with GitHub Packages on GitHub Actions

jobs:
  build:
    name: example of code measurement using gocloc
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@master

      - name: Login GitHub Registry
        run: docker login docker.pkg.github.com -u owner -p ${{ secrets.GITHUB_TOKEN }}

      - name: Run gocloc
        run: docker run --rm -v "${PWD}":/workdir docker.pkg.github.com/hhatto/gocloc/gocloc:latest .

Integration Jenkins CI

use SLOCCount Plugin.

$ cloc --by-file --output-type=sloccount . > sloccount.scc

$ cat sloccount.scc
398 Go      ./main.go
190 Go      ./language.go
132 Markdown        ./README.md
24  Go      ./xml.go
18  Go      ./file.go
15  Go      ./option.go

Support Languages

use --show-lang option

$ gocloc --show-lang

Performance

• CPU 3.8GHz 8core Intel Core i7 / 32GB 2667MHz DDR4 / MacOSX 13.3.1
• cloc 1.96
• tokei 12.1.2 compiled with serialization support: json
• gocloc a88edc5
• target repository is golang/go commit:f742ddc

cloc

$ time cloc .
   12003 text files.
   11150 unique files.
    1192 files ignored.

8 errors:
Line count, exceeded timeout:  ./src/cmd/dist/build.go
Line count, exceeded timeout:  ./src/cmd/trace/static/webcomponents.min.js
Line count, exceeded timeout:  ./src/net/http/requestwrite_test.go
Line count, exceeded timeout:  ./src/vendor/golang.org/x/net/idna/tables10.0.0.go
Line count, exceeded timeout:  ./src/vendor/golang.org/x/net/idna/tables11.0.0.go
Line count, exceeded timeout:  ./src/vendor/golang.org/x/net/idna/tables12.0.0.go
Line count, exceeded timeout:  ./src/vendor/golang.org/x/net/idna/tables13.0.0.go
Line count, exceeded timeout:  ./src/vendor/golang.org/x/net/idna/tables9.0.0.go

github.com/AlDanial/cloc v 1.96  T=35.07 s (317.9 files/s, 78679.3 lines/s)
-----------------------------------------------------------------------------------
Language                         files          blank        comment           code
-----------------------------------------------------------------------------------
Go                                9081         205135         337681        1779107
Text                              1194          11530              0         210849
Assembly                           563          15549          21625         122329
HTML                                17           3197             78          24983
C                                  139           1324            982           6895
JSON                                20              0              0           3122
CSV                                  1              0              0           2119
Markdown                            27            674            106           1949
Bourne Shell                        16            253            868           1664
JavaScript                          10            234            221           1517
Perl                                10            173            171           1111
C/C++ Header                        26            145            346            724
Bourne Again Shell                  16            120            263            535
Python                               1            133            104            375
CSS                                  3              4             13            337
DOS Batch                            5             56             66            207
Windows Resource File                4             23              0            146
Logos                                2             16              0            101
Dockerfile                           2             13             15             47
C++                                  2             11             14             24
make                                 5              9             10             21
Objective-C                          1              2              3             11
Fortran 90                           2              1              3              8
awk                                  1              1              6              7
YAML                                 1              0              0              5
MATLAB                               1              1              0              4
-----------------------------------------------------------------------------------
SUM:                             11150         238604         362575        2158197
-----------------------------------------------------------------------------------
cloc .  33.70s user 1.48s system 99% cpu 35.237 total

tokei

$ time tokei --sort code  --exclude "**/*.txt" .
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Go                   9242      2330107      1812147       318036       199924
 GNU Style Assembly    565       159534       127093        16888        15553
 C                     143         9272         6949         1000         1323
 JSON                   21         3122         3122            0            0
 Shell                  16         2785         2267          342          176
 JavaScript             10         1972         1520          218          234
 Perl                    9         1360         1032          170          158
 C Header               27         1222          727          349          146
 BASH                   16          918          521          279          118
 Python                  1          612          421           70          121
 CSS                     3          354          337           13            4
 Autoconf                9          283          274            0            9
 Batch                   5          329          207           66           56
 Alex                    2          117          101            0           16
 Dockerfile              2           75           47           15           13
 C++                     2           49           24           14           11
 Makefile                5           40           20           10           10
 Objective-C             2           21           15            3            3
 FORTRAN Modern          2           12            8            3            1
 Markdown               18         2402            0         1853          549
-------------------------------------------------------------------------------
 HTML                   17        19060        18584           49          427
 |- CSS                  4         2071         1852           10          209
 |- HTML                 1          219          212            0            7
 |- JavaScript           8         6920         6876           16           28
 (Total)                          28270        27524           75          671
===============================================================================
 Total               10117      2533646      1975416       339378       218852
===============================================================================
tokei --sort code --exclude "**/*.txt" .  0.76s user 0.50s system 562% cpu 0.224 total

gocloc

$ time gocloc --exclude-ext=txt .
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                            9096         205242         352844        1764503
Assembly                       563          15555          21624         122324
HTML                            17           3197            212          24849
C                              139           1324            983           6894
JSON                            20              0              0           3122
BASH                            27            345           1106           2122
Markdown                        18            549             28           1825
JavaScript                      10            234            218           1520
C Header                        26            145            346            724
Perl                            10            173            584            698
Python                           1            133            104            375
CSS                              3              4             13            337
Batch                            5             56              0            273
Plan9 Shell                      4             23             50             96
Bourne Shell                     5             28             24             78
C++                              2             11             14             24
Makefile                         5             10             10             20
Objective-C                      2              3              3             15
FORTRAN Modern                   2              1              3              8
Awk                              1              1              6              7
-------------------------------------------------------------------------------
TOTAL                         9956         227034         378172        1929814
-------------------------------------------------------------------------------
gocloc --exclude-ext=txt .  0.65s user 0.51s system 119% cpu 0.970 total

License

MIT

gocloc's Issues

proposal: Split into a command and a library

I would like to write some code to analyze a repository and write the stats into a backend time relational database and having the analysis from gocloc would be nice, but to write some custom code around it I would need it to be importable.

I could fork but what do you think about creating a go package and separate cmd?

Option to ignore code by a pattern

Use case: Exclude test code *_test.go

Installation failure on Windows

When I tried to install it on Windows without Go modules, I encountered the problem that is also mentioned in the repo's README (package github.com/src-d/enry: code in directory /PATH/TO/go/src/github.com/src-d/enry expects import "github.com/src-d/enry/v2").

However, I could not install it using Go modules either. The error message is the following: go: cannot find main module; see 'go help modules'.

Question: how it works when use command "gocloc ."

It's quite useful to calculate how many codes are there in my project. After using it I try to implement a more simple one with a few same project to play. However something disturbed me a lot so I hope some one can help me. My question is how the command gocloc . call function to calculate lines of code.
I knew that go get <this project address> download the package so I can use it in my project. But why gocloc . can invoke this package? What I know is only pass this argument with main args as go run main.go gocloc ..
How this main function get the command info input at the terminal?

Incorrect code count issue

When a single line comment is added next to a piece of code, the particular line is considered as "comment".

Example:

public class Sample {
public static void main(String args[]){
int a; /* A takes care of counts */
int b;
int c;
String d; /*Just adding comments */
/*End of Main*/
}
}

The above code has got 8 lines as code but when gocloc is run the output shown comes as below. .

filename=../Sample.java
[CODE,cd:1,cm:0,bk:0,iscm:false] public class Sample {
[CODE,cd:2,cm:0,bk:0,iscm:false] public static void main(String args[]){
[COMM,cd:2,cm:1,bk:0,iscm:false,iscms:false] int a; /* A takes care of counts */
[CODE,cd:3,cm:1,bk:0,iscm:false] int b;
[CODE,cd:4,cm:1,bk:0,iscm:false] int c;
[COMM,cd:4,cm:2,bk:0,iscm:false,iscms:false] String d; /*Just adding comments */
[COMM,cd:4,cm:3,bk:0,iscm:false,iscms:false] /End of Main/
[CODE,cd:5,cm:3,bk:0,iscm:false] }
[CODE,cd:6,cm:3,bk:0,iscm:false] }

-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Java 1 0 3 6
-------------------------------------------------------------------------------
TOTAL 1 0 3 6
-------------------------------------------------------------------------------

\* Statistical exception

gomod go get installed, version gocloc \ @ v0.3.1
Comprehensive code is regarded as a comment after return "/*" in go code

gomod go get安装，版本[email protected]
在go代码中return "/*" 后全面代码算为注释

// getRouteByName 函数使用函数名称生成路由路径。
func getRouteByName(name string) string {
	names := splitName(name)
	if checkAllowMethod(names[0]) {
		names = names[1:]
	}
	if len(names) == 0 {
		return "/*"
	}
	name = ""
	for i := 0; i < len(names); i++ {
		if names[i] == "By" {
			name = name + "/:" + names[i+1]
			i++
		} else {
			name = name + "/" + names[i]
		}
	}
	return strings.ToLower(name)
}

Is there a verbose option?

Is there a "verbose" option that shows which files are being considered/accounted?

(Sorry for opening an issue to ask a question. )

Thanks

Please update to Go 1.16 and also release for macOS ARM64

Go 1.16 was just released with macOS ARM64 support, see https://blog.golang.org/go1.16

No Windows builds available

Show totals for each language

Currently, the totals are only displayed for each line type. It would be useful to see it for each language as well.

For example, this:

Language                     files          blank        comment       code
-------------------------------------------------------------------------------
Go                            16             306          238           1607
XML                           7               0            0            209
HCL                           5               19           0            134
Bourne Shell                  1               6            21            1
JSON                          2               0            15            0
Markdown                      1              16            38            0 
YAML                          2              10           352            0 
-------------------------------------------------------------------------------
TOTAL                         34            357           664           1951
-------------------------------------------------------------------------------

would become this:

Language                     files          blank        comment       code        TOTAL
-------------------------------------------------------------------------------------------
Go                            16             306          238           1607       2167
XML                           7               0            0            209        216
HCL                           5               19           0            134        158
Bourne Shell                  1               6            21            1         29
JSON                          2               0            15            0         17
Markdown                      1              16            38            0         55
YAML                          2              10           352            0         364
-------------------------------------------------------------------------------------------
TOTAL                         34            357           664           1951      3006
-------------------------------------------------------------------------------------------

Need support for MacOS X

Not covering most file types

Some files do not end with standard extensions and these files are not correctly recognized.
Take the XML files as an example, only the files with the extension xml or XML are counted.
In fact, many files that are not with these extensions may also be XML files, such as .fxml, .plist, .rss, .xproj and so on.

refer here:
https://gist.github.com/ppisarczyk/43962d06686722d26d176fad46879d41#file-programming_languages_extensions-json-L3109
https://github.com/AlDanial/cloc/blob/4d244d29a4b0f3c994ded12f7a0c57b8fe2906ef/cloc#L7581

It is recommended to add these file extensions to improve accuracy.

release gocloc binary

Is the SkipDuplicated option judged to be reversed?

SkipDuplicated ，when the option set false , duplicated content will be skipped.

utils.go

if !opts.SkipDuplicated {
	ignore := checkMD5Sum(path, fileCache)
	if ignore {
		if opts.Debug {
			fmt.Printf("[ignore=%v] find same md5\n", path)
		}
		return nil
	}
}

Some file types showing all lines as comments

Steps to reproduce:

go install github.com/hhatto/gocloc/cmd/gocloc
git clone https://github.com/hhatto/gocloc.git
cd gocloc
gocloc Makefile

produces:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Makefile                         1              4             11              0
-------------------------------------------------------------------------------
TOTAL                            1              4             11              0
-------------------------------------------------------------------------------

Relative to 'cloc':

       1 text file.
       1 unique file.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.02 s (61.8 files/s, 927.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
make                             1              4              0             11
-------------------------------------------------------------------------------

Suspected cause:

Many of the Languages have the multiline quote definition of [][]string{{"", ""}} for language.multiLines

But the test for marking as code is:

if len(language.multiLines) == 0 {
  onCode(clocFile, opts, len(inComments) > 0, line, lineOrg)
  continue scannerloop
}

I've noticed this for Makefiles and bash scripts, so I think it's rather pervasive. I did notice some code to try to deal with Makefiles, not sure if that's part of this though. I think it's that the Language definition should be empty for these types of files that don't support multiline comments or the logic for detecting should deal with empty strings as if it were empty.

Could you change release label from v0.3 to semantic v0.3.0

Exclude file names depending on path

The --not-match option only considers the base name.

Current behaviour:
Let's say I have the file structure:

.
├── app.py
└── test_dir
    └── app.py

And I want to exclude test_dir/app.py but not the root app.py, ideally I would use the supported regex format allowed by the --not-match option like below:

$ gocloc --not-match="(test_dir/app.py)" .     

However, this won't ignore the test file as I would like. This behaviour works the other way around too as if I wanted to ignore app.py but not test_dir/app.py, I might do:

$ gocloc --not-match="(app.py)" . 

But this will now ignore both of the files regardless of their location.

Desired behaviour:
For the file matching to account for the full path the file is being compared to.

0.5.1 release was not completed?

👋 it looks like 0.5.1 release was not completed, see this failed action run. Create this issue to raise some awareness. Thanks!

relates to Homebrew/homebrew-core#137486

Support Git parameters

Scanning code lines in a Git repository is also a common scenario

Support common compression formats such as zip, tar, etc

gocloc --version print blank

steps to reproduce

1. install the tool shell go install github.com/hhatto/gocloc/cmd/gocloc@latest
2. execute gocloc --version
3. output ()

Unable to install in windows 7

I'm trying to install for windows and I'm getting the following error:

$ go get github.com/hhatto/gocloc
# github.com/hhatto/gocloc
..\..\..\..\go_workspace\src\github.com\hhatto\gocloc\file.go:53: scanner.Buffer undefined (type *bufio.Scanner has no field or method Buffer)

Go version = go1.5.1 windows/amd64

Unable to build gocloc on raspberry pi

I'm hitting this issue building gocloc on a raspberry pi.

$ uname -sr
Linux 4.19.55-Re4son-v8+

$ go version
go version go1.12.9 linux/arm64

$ GO111MODULE=on go get -u -ldflags "-s -w" github.com/hhatto/gocloc/cmd/gocloc
go: finding github.com/hhatto/gocloc/cmd/gocloc latest
go: finding github.com/hhatto/gocloc/cmd latest
go: finding gopkg.in/check.v1 latest
go build github.com/src-d/enry/v2/data: /usr/lib/go-1.12/pkg/tool/linux_arm64/compile: signal: killed

Skip files explicitly using .gitignore

It would be great if gocloc could skip files explicitly excluded by a versioning tool like git via .gitignore file.

support output format options?

I am using this library. I want to convert the output format to csv, tsv in order to link with other aggregation devices, but do you plan to support the output format option?

Is it possible to not match multiple directories --not-match-d

Hello, it's a nice project, I wanted to know if there was a way to exclude more than one folder ?

If yes, how ? If no, will it be a big change to implement ?

Incorrect comment count

Consider this code:

package main

func main() {
	var n string /*
		comment
		comment
	*/
}

There are 4 code lines and 3 comment lines, but gocloc counts not as expected:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                               1              1              0              7
-------------------------------------------------------------------------------
TOTAL                            1              1              0              7
-------------------------------------------------------------------------------

scannerloop stops after encountering a very long line

If you have a line longer than the 1MB buffer length (don't ask) the scannerloop's scanner.Scan() for condition will evaluate to false. When this happens line counting for the current file stops where it is and reports incorrect results for that file.

gocloc/file.go

Line 90 in 7b24285

for scanner.Scan() {

I could see a few fixes for this.

1. A new option to set the buffer size with a maximum of 1MB being the default if it is unset:

	if opts.MaxLineLength > 0 {
		scanner.Buffer(buf.Bytes(), opts.MaxLineLength)
	} else {
		scanner.Buffer(buf.Bytes(), 1024*1024)
	}

2. Scanning the files ahead of time to find the longest gap between line endings and then automatically setting that as the buffer size. This does require reading the file twice though.

3. Changing the scannerloop to use something like mmap instead of scanner.

If you're interested in the third one let me know and I'll work on a PR.

The first one probably touches a bit more of the overall design than I should take on for a first PR.

I think the second one is safe but it does double the I/O required. Disk caching may make this less of an issue than doubling the amount of raw data read from disk but still feels like a last resort.

go get instructions fail

go get complains trying to get gocloc.

~$ go get -u github.com/hhatto/gocloc/cmd/gocloc
package github.com/src-d/enry: code in directory /home/martin/go/src/github.com/src-d/enry expects import "github.com/src-d/enry/v2"

Not sure what it means for code to "expect" an import and why that is a problem.

Unable to build gocloc

I get the following error when building gocloc.

$ go get -u -ldflags '-s -w' github.com/hhatto/gocloc/cmd/gocloc
package github.com/src-d/enry: code in directory /Users/jimi/go/src/github.com/src-d/enry expects import "github.com/src-d/enry/v2"

$ go version
go version go1.12.7 darwin/amd64

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G84

sort option doesn't work

According to help, gocloc supports --sort option, but it seems SortTag is not referred from anywhere. In my understanding, currently it is not implemented. Do you have plan to implement it?

Exclude multiple directories

I run with flag '''--not-match-d vendor node_modules''' it doesn't work