gitleaks / gitleaks Goto Github PK

View Code? Open in Web Editor NEW

15.4K 152.0 1.3K 5.09 MB

Protect and discover secrets using Gitleaks 🔑

Home Page: https://gitleaks.io

License: MIT License

Go 99.42% Dockerfile 0.15% Makefile 0.18% Python 0.25%

security security-tools git golang go secret gitleaks devsecops hacktoberfest

gitleaks's Issues

Consumes a lot of memory when running against an org

Interesting project. I was testing it against uber's github org and it effectively run out of memory twice, even when the --disk option was specified. I did not investigate the cause but it is likely to be a mem leak.

Go not building/installing

Hello,

This is my first time installing a go repo. I did sudo go get -u -v -t github.com/zricethezav/gitleaks and it only came back with this response:

github.com/zricethezav/gitleaks (download)

So I'm not quite sure if I'm not building correctly or what else I could be doing wrong. My go version is 1.8.3 and I'm on linux/amd64 Ubuntu 17.10. Sorry about my incompetence!

And I did try installing the binaries but I'm not sure how to do those either.

Persistant/Local Repos

A feature that has been requested by a few folks is the ability to audit local repos as scheduled tasks. @JustinAzoff in #8 suggested using a data store like boltDB to cache commits so we don't run redundant checks on commits that have already been vetted.

I'm thinking we keep the default behavior but add some options like a --dir, -d flag to tell gitleaks run this for a local repo: $ gitleaks --dir {my_repo}. For persistent support we can add a --persist, -p flag that tell gitleaks to skip the rm -rf call and check the data store before running all checks.

[feature request] define disk temp path

Thanks for such amazing tool!
I'm scanning a huge organization and not sure if the space in my disk will be enough to store everything under /tmp/ directory.
So my suggest is include an optional parameter to define the temp directory when --disk is used.
I'd like to help implement that but I don't know golang enough to contribute.

Git leaks - Usability feedback

Hey There,
Thanks again for a really great tool. I'm providing this feedback here, as per our discussions on reddit DM's.

I've got a bit of feedback that i think would help greatly with general usability. I'm slightly selfish in that these would greatly help me, however, I think it would also help other peoples adoption as well.

Customise the datadir / workdir directory on the CLI
Currently all data produced by the tool goes into ~/.gitleaks/ , however, for larger audits, it would be really great to be able to specify this directory so that the user could target a different location for performance or archival reasons.
JSON data output enhancements
This is a really great capability, possibly one of the killer features of your implementation.
Right now, for repositories that contain no results, you create a JSON file with a null value.
Ideally this should either not be created, or it should contain an empty array. This would ease parsing capabilities for those using your tool to generate large batches of results then post process them using something like Ruby (Which is what I am doing). My general feeling is that no report should be created if there are no matches, however, I think consistency is key however.
--quiet mode / --logfile
A quiet mode or logfile capability would help with automating your tool further, alongside more detailed exit codes as detailed below.
Return code definitions / Extended return code capability ?
Right now, from what I understand, the tool returns 0 for successful execution, and non-zero for unsuccessful execution. It would be great to have a bit more detail exposed as exit codes for different circumstances, such as if a scan executed, and returned matches, etc.

This would help greatly with automating your tool further. However, I'd love to hear what you and other users think about this.
A good example of this implemented well is in Puppet's --detailed-exit-codes functionality.

Externalising the regex matches ?
I'm not sure if this is something that will greatly affect performance, but it would be great to externalise the REGEX matches into a configuration file so that you don't have to rebuild the binary when updating rules. Putting them into a runtime configuration file (Maybe YAML) would really help usability. Being able to specify the datadir / workdir in this same file would also be great.

A very basic example could look like this:

---
config:
  workdir: /somewhere/else
regex:
  AWS:
    pattern: <REGEX>

Once again, thanks again for a great tool :)

[Feature Request] - Output to JSON

It would be a great feature to get JSON output of the findings and which we can use for other automations.

Does the docker image work for any git repository?

I'm trying to use the docker image to test a git repository that is hosted on an internal GitLab instance. And I get an error:

$ docker run --rm --name=gitleaks zricethezav/gitleaks https://gitlab.internal/mygroup/myproject

usage: gitleaks [options] <URL>/<path_to_repo>

Options:
 -u --user 		Git user mode
 -r --repo 		Git repo mode
 -o --org 		Git organization mode
 -l --local 		Local mode, gitleaks will look for local repo in <path>
 -v --verbose 		Verbose mode, will output leaks as gitleaks finds them
 --report-path=<STR> 	Report output, default $GITLEAKS_HOME/report
 --clone-path=<STR>	Gitleaks will clone repos here, default $GITLEAKS_HOME/clones
 -t --temp 		Clone to temporary directory
 --concurrency=<INT> 	Upper bound on concurrent "git diff"
 --since=<STR> 		Commit to stop at
 --b64Entropy=<INT> 	Base64 entropy cutoff (default is 70)
 --hexEntropy=<INT>  	Hex entropy cutoff (default is 40)
 -e --entropy		Enable entropy		
 -h --help 		Display this message
 --token=<STR>    	Github API token
 --stopwords  		Enables stopwords

Unknown option https://gitlab.internal/mygroup/myproject

usage: gitleaks [options] <URL>/<path_to_repo>

Options:
 -u --user 		Git user mode
 -r --repo 		Git repo mode
 -o --org 		Git organization mode
 -l --local 		Local mode, gitleaks will look for local repo in <path>
 -v --verbose 		Verbose mode, will output leaks as gitleaks finds them
 --report-path=<STR> 	Report output, default $GITLEAKS_HOME/report
 --clone-path=<STR>	Gitleaks will clone repos here, default $GITLEAKS_HOME/clones
 -t --temp 		Clone to temporary directory
 --concurrency=<INT> 	Upper bound on concurrent "git diff"
 --since=<STR> 		Commit to stop at
 --b64Entropy=<INT> 	Base64 entropy cutoff (default is 70)
 --hexEntropy=<INT>  	Hex entropy cutoff (default is 40)
 -e --entropy		Enable entropy		
 -h --help 		Display this message
 --token=<STR>    	Github API token
 --stopwords  		Enables stopwords

What is this Unknown option https://gitlab.internal/mygroup/myproject error? Where does it come from?

Could it be a proxy related issue? My company is using proxies.

more stopWords please

BEGIN CERTIFICATE
(.crt CERTIFICATE)
BEGIN RSA PRIVATE KEY
(.key PRIVATE KEY)
secret
(Oauth or jwt.io or other API but passed the regx check)

and there some false alarm with

[
	{
		"content": "+[![Cocoapods](https://cocoapod-badges.herokuapp.com/v/DateTools/badge.png)](http://cocoapods.org/?q=datetools)",
		"commit": "70b07b600c79cfaf7dbcc68402dbda73d32e965e"
	},
	{
		"content": "+[![Cocoapods](https://cocoapod-badges.herokuapp.com/v/DateTools/badge.png)](http://cocoapods.org/?q=datetools)",
		"commit": "f6f9cbe91ce9564ffe65c3dede2dbab017d95eb5"
	}
]

Ignored files by default

Hi,

In version 1.17.0 one of the added features was:

Default whitelisting: bin, doc, jpeg, gif

But it seems like that was removed in this commit: 7e3c740#diff-7ddfb3e035b42cd70649cc33393fe32cL193

Was this removal a mistake or am I missing something?

Cheers

Scan for "secret_key"

This should be added to scan beacause we can have some recaptcha_secret_key and stuff like that in the repositories.

Scanning private git repositories

Would be very useful to be able to provide http credentials as an option to authenticate with private git repositories. Currently does not appear to support anything but public projects, unless I have misunderstood the options.

--verbose

--verbose switch giving error
2018/06/11 17:34:15 cannot clone https://github.com/xxx into /root/go/src/gitleaks/xxx

Private Repositories - API Token

Hello!
First I'd like to express my thanks for this awesome project so thank you :)

Is there a reason why I cannot search private repos after providing gitleaks with a read-only API token? It still asks me for my github.com username and password after passing in my --token.

e.g., command:

./gitleaks --temp --token= --verbose --org https://github.com/myOrg

Output URL to specific repo in JSON

It would be nice if the output JSON would contain the URL to the specific repo. Since different orgs can have repos of the same name, it's a bit cumbersome to have to go back and figure out which specific repo has the leaks.

Thanks

Entropy checks

Although the first version did entropy checks apart from regexes (https://github.com/zricethezav/gitleaks/blob/v0.1.0/checks.go#L34), that's no longer the case in the latest version.

I know entropy checks could create lots of false positives, but they can also catch things that regex don't so I see a lot of value in there. Also, using the whitelisting options can reduce a lot of the false positives.

Could this be added again?

more filters and options

Here are some filters and options that would be nice to have:

ignore filetypes
commit depth
commit range (HASH1 - HASH2)
scan PR #
Redact offending string from report/output
~~include commit url in leaks~~

gitleaks scan bypasses RSA key

This happens for some of my repositories when i did scan today. However, the same scan has been done on October 17th, listed all leaks but now it doesn't for few repositories. The same RSA keys have been identified in some repo's and got listed in the report.

I did check the files where it actually found leaks in my last scan. The keys still exists.
No changes made to those repositories from last 4 months.

Can anyone help on this. Will provide any information if required.

Authentication Error

I am running a scan against a repo that we, However it is giving me authentication error message even to I can clone the repo without any issue.

NFO[2018-09-17T11:11:26-04:00] cloning https://github.*********.com/*******/suite.git
WARN[2018-09-17T11:11:26-04:00] skipping audit for repo suite.git due to cloning error: authentication required
INFO[2018-09-17T11:11:26-04:00] writing report to %s/tmp/leaks

git clone command for the above repo runs perfectly fine.

Exclude organization repository

Can we somehow exclude forked repositories or skip the list of repo names?

$ gitleaks --github-org=companyltd --exclude-forks
... cloning: ...
... skipping: [fork] repo

or (combination):

$ gitleaks --github-org=companyltd --exclude=redhat,other-name
... cloning: ...
... skipping: redhat

How to find a file

How to find the file inside the repo from the log below?
OBS.

The https://github.com/organization/xptorepo/commit/abc123 point to a different file than test/xpto/file.go
The https://github.com/organization/xptorepo/tree/master/test/xpto folder doesn't exist (probably exist in the past)
Didn't find anything when searching by "xptoname" or "tralala"
Didn't find the branch "aaaa/heads/master"

{
   "line": "\txptoname = `-----BEGIN PRIVATE KEY-----",
   "commit": "abc123",
   "offender": "-----BEGIN PRIVATE KEY-----",
   "reason": "PKCS8",
   "commitMsg": "tralala",
   "author": "that guy",
   "file": "test/xpto/file.go",
   "branch": "aaaa/heads/master",
   "repo": "xptorepo",
   "date": "2017-11-14T00:09:57-08:00"
}

Output full path in JSON

A number of the results I've come across so far have been been in test type directories. If the full path to the file was available in the JSON it would make it quicker at a glance to easily dismiss results without having to track down the leak manually

Thanks

Gitlab User/Organization Support

Much like --github-user and --github-org I would like to support gitlab and even bitbucket.

gitleaks --verbose

gitleaks --verbose option not working even not listing in all options using help.

Group secrets in the report

I have found the report a bit noisy when one possible secret has been found multiple times and I have had to spend some time reviewing them. I'd say that it'd be better to report a possible secret only once and show an array of commits where it was found, here's an example:

[
    {
        "offender": "AKIALALEMEL33243OLIA",
        "reason": "AWS",
        "commits": [
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "AKIALALEMEL33243OLIAE",
                "message": "Add secrets\n"
            },
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "AKIALALEMEL33243OLIAE",
                "message": "Add secrets\n"
            }
        ]
    },
    {
        "offender": "62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
        "reason": "High Entropy",
        "commits": [
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "secret = 62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
                "message": "Add secrets"
            },
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "secret = 62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
                "message": "Add secrets"
            }
        ]
    }
]

What do you think about this?

Installation Question

Can I install gitleaks without installing Go?
Or maybe a better question is, can you attach the binaries to the GitHub Releases?

Add GCP Service Account Key to regexes list

https://cloud.google.com/video-intelligence/docs/common/auth

shell injection

https://github.com/zricethezav/gitleaks/blob/master/leaks.go#L29 could be pwned by malicious repo urls. need to do some regexing or use https://github.com/src-d/go-git for the clones.

Stop words option to prevent false positives

If we come across a line: conn = client.s3(aws_key=settings.AWS_KEY there will be a regex match on aws_key= which will then check settings.AWS_KEY for entropy. If the entropy is past a certain value, gitleaks determines this is a valid leak which it is not. A simple way to reduce false positives is to add a stop word checker in checks.go.

There could be a default set of stop words, something like: settings, client, connection, setting, ... that are checked if a strict option is set: gitleaks -s <repo url>.

scanning github orgs' private repos

trying to use the --github-org option, but it's only targetting public repos. my API key definitely has full repo access (to public and private), what am I missing?

Add support for scanning an enterprise Github organization

Tried scanning an organization in our enterprise github and I get an error:

gitleaks -o https://github.enterprise.com/enterprisegithub
2018/02/23 09:32:43 Get https://api.github.com/orgs/enterprisegithub/repos: read tcp 10.152.32.109:52205->192.30.253.116:443: read: connection reset by peer

It's clear now it doesn't use the full url provided so when I attempt to provide the full url according the usage from the help message, nothing happens:

gitleaks -o enterprisegithub https://github.enterprise.com/

Uknown option https://github.enterprise.com/

usage: gitleaks [options] <url>

Scanning a single repo does work on the other hand, but has to be done to every repo in the org.

Filter repositories of user/organisation

Hello,

Something I noticed is, if I fork a big project (like node or better, the linux kernel ) and then I scan myself, it will take a while !

From here I can see two areas that can get an enhancement:

Options to scan or not forked project
Options to filter repos by size ( number of commits, project size or number of files ?)
(Or scan repositories ordered by size, so at some point I could kill the process)

panic: runtime error: invalid memory address or nil pointer dereference

Error:

➜  Downloads ./gitleaks-linux-amd64 --repo-path=/home/projects/test 
INFO[2018-07-24T01:09:57+03:00] cloning                                      
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8c4fc5]

goroutine 1176 [running]:
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Patch).FilePatches(...)
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/patch.go:125
main.auditRef.func1.1(0xc4201ac040, 0x0, 0x0, 0xc4200909c0, 0xc432bfc1e0, 0xc432bfc0f0)
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/main.go:355 +0x55
created by main.auditRef.func1
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/main.go:348 +0x1a0

System:

➜  Downloads ./gitleaks-linux-amd64 --version
1.1.2

➜  Downloads uname -a
Linux NB-NIX 4.17.8-1-ARCH #1 SMP PREEMPT Wed Jul 18 09:56:24 UTC 2018 x86_64 GNU/Linux

have other repos, and it works with them

what else do you need?

Unclear: are commit messages audited too?

Note: Gitleaks rewrite coming

Hey all,

First of all, thanks for all the support. I'll be honest - I made gitleaks as an exercise to learn Go so it's a pretty novice code base. Since then I've learned some things and am working on a rewrite that will add some cool features such as branch and PR targeting, as well as support for any remote target (gitleaks will no longer be github specific when cloning remote). Additionally, I'm making use of src-d's go-git plumbing to make the audits quicker with an 'in-memory' option -- no more shelling out git x commands.

I'd like to open the floor for feature requests here. Shout out to @keirans who has already come up with some features:

externalizing regex
cleaner options
white listing commits
regex matching on commit messages

Please add a comment if there is a feature you would like to see in the next gitleaks version

Ability to ignore path

I would really like ability to ignore paths

Timings compared to truffleHog

I would like to add a timing chart to the README comparing times and false positives between truffleHog/similar tools

Add support for github.*

For enterprises that host their own private github (e.g. github.dhs.gov, github.asp.net, etc...), it appears that the github links aren't recognized by the code as valid github links, so the error Unknown option https://github.asp.net shows up. It would be nice if .gov and .net addresses were recognized in addition to the base github.com url.

Is there somewhere in your code that you're validating this? I could try and help implement this...

"fatal error: concurrent map read and map write" scanning repository

Version: Built from master (87d916f as of writing)
OS: macOS 10.14
Go: go version go1.11.1 darwin/amd64

When running 'gitleaks' against a local repository I have (gitleaks --repo-path=path/to/my/repo), gitleaks panics with message listed below.

Unfortunately, the repository is private so I cannot share it, but if there's any information I can give to help narrow this down or steps I can take, I am happy to. Thank you.

fatal error: concurrent map read and map write

goroutine 37 [running]:
runtime.throw(0x166cf3b, 0x21)
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/panic.go:608 +0x72 fp=0xc000d43178 sp=0xc000d43148 pc=0x102c342
runtime.mapaccess2_fast64(0x15888c0, 0xc00001f800, 0x32ec0d, 0xc000d2e240, 0x3f)
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/map_fast64.go:61 +0x1a8 fp=0xc000d431a0 sp=0xc000d43178 pc=0x1012668
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).FindHash(0xc000252880, 0x32ec0d, 0x0, 0x0, 0x0, 0x14, 0xf03e1f0aa05a57fc)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:177 +0x6b fp=0xc000d43200 sp=0xc000d431a0 pc=0x140c47b
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).decodeObjectAt(0xc000148f68, 0x16f44e0, 0xc000d301c0, 0x16f3d60, 0xc000252880, 0x32ec0d, 0x0, 0x0, 0x662e5d6c66d49473, 0x32ec0d)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:301 +0x56 fp=0xc000d43290 sp=0xc000d43200 pc=0x1437676
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).getFromPackfile(0xc000148f68, 0x2f9db7ce84927f1f, 0x66d49473dcedf309, 0x662e5d6c, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:293 +0x2cc fp=0xc000d43398 sp=0xc000d43290 pc=0x143749c
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).EncodedObject(0xc000148f68, 0x9db7ce84927f1f02, 0xd49473dcedf3092f, 0x662e5d6c66, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:174 +0x3f5 fp=0xc000d43438 sp=0xc000d43398 pc=0x1436c05
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0x2f9db7ce84927f1f, 0x66d49473dcedf309, 0x662e5d6c, 0x203000, 0x203000, 0x1020f28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:45 +0x4e fp=0xc000d43488 sp=0xc000d43438 pc=0x13bb95e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*TreeWalker).Next(0xc000d436c0, 0xc000cfe5a0, 0x9, 0xc000d62140, 0x14, 0x84927f1f00004000, 0xdcedf3092f9db7ce, 0x662e5d6c66d49473, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:408 +0x56f fp=0xc000d435c8 sp=0xc000d43488 pc=0x13bdf7f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.transformChildren(0xc000c44230, 0x15efbc0, 0xc001dec701, 0x1b73280, 0x28, 0x38)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/treenoder.go:105 +0x21d fp=0xc000d43720 sp=0xc000d435c8 pc=0x13be9bd
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*treeNoder).Children(0xc000cfe3c0, 0x1b73280, 0x1f41b00, 0xc000001c80, 0x203000, 0x1020f28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/treenoder.go:87 +0x61 fp=0xc000d43760 sp=0xc000d43720 pc=0x13be6b1
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/internal/frame.New(0x16f3760, 0xc000cfe3c0, 0x48, 0x48, 0xc000cfe460)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/internal/frame/frame.go:29 +0x3b fp=0xc000d437e0 sp=0xc000d43760 pc=0x133fcfb
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.newIter(0x16f3760, 0xc000cfe3c0, 0x0, 0x0, 0x0, 0x15efc80, 0x1, 0xc000cfe460)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/iter.go:90 +0x87 fp=0xc000d43840 sp=0xc000d437e0 pc=0x1342db7
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.NewIter(0x16f3760, 0xc000cfe3c0, 0x1, 0x16ec5c0, 0xc00006a060)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/iter.go:71 +0x46 fp=0xc000d43890 sp=0xc000d43840 pc=0x1342c26
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.newDoubleIter(0x16f3760, 0xc000cfe3c0, 0x16f3760, 0xc000cfe410, 0x167ecf8, 0xc000001c80, 0x1000, 0x1000)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/doubleiter.go:43 +0x67 fp=0xc000d43940 sp=0xc000d43890 pc=0x1341ed7
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.DiffTreeContext(0x16f2380, 0xc0000260c0, 0x16f3760, 0xc000cfe3c0, 0x16f3760, 0xc000cfe410, 0x167ecf8, 0x13bb9bc, 0x1f852a0, 0xc000148f50, ...)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/difftree.go:277 +0x96 fp=0xc000d439d0 sp=0xc000d43940 pc=0x1341046
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DiffTreeContext(0x16f2380, 0xc0000260c0, 0xc000c44230, 0xc000cfe320, 0x8a0f72a6f3d94d5e, 0x3faba7ce, 0xc000cfe320, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/difftree.go:28 +0x157 fp=0xc000d43a58 sp=0xc000d439d0 pc=0x13b5b37
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).PatchContext(0xc000c44230, 0x16f2380, 0xc0000260c0, 0xc000cfe320, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:316 +0x4d fp=0xc000d43ab0 sp=0xc000d43a58 pc=0x13bd89d
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf80f0, 0x16f2380, 0xc0000260c0, 0xc000bf8000, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:92 +0xb5 fp=0xc000d43b00 sp=0xc000d43ab0 pc=0x13b0c55
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf80f0, 0xc000bf8000, 0xc000027820, 0xc0000304d0, 0xc000bf80f0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d fp=0xc000d43b48 sp=0xc000d43b00 pc=0x13b0cfd
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf80f0, 0xc000bf8000)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2 fp=0xc000d43f98 sp=0xc000d43b48 pc=0x14fae92
runtime.goexit()
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000d43fa0 sp=0xc000d43f98 pc=0x1058911
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 1 [chan send]:
main.auditGitReference.func1(0xc000bf83c0, 0xc000bf83c0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:520 +0x32b
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*commitPreIterator).ForEach(0xc00012e720, 0xc000a1ad20, 0x16f1600, 0xc00012e720)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit_walker.go:103 +0x65
main.auditGitReference(0xc0000af1d0, 0xc00015a540, 0x0, 0x0, 0xc00013c1c0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:499 +0x279
main.auditGitRepo(0xc0000af1d0, 0x0, 0x0, 0xc00011ddb0, 0xd93f1f82, 0xbed1d93f1f82)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:470 +0x23c
main.run(0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:306 +0x105
main.main()
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:251 +0xea

goroutine 36 [runnable]:
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*idxfileEntryIter).Next(0xc001dd4020, 0xc00001f800, 0xc35fa1, 0xc001651ae8)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:266 +0x25a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash(0xc000252880, 0x142c196, 0xc00013c9c0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:200 +0x110
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).FindHash(0xc000252880, 0xe30e46, 0x0, 0x0, 0x0, 0x14, 0xf03e1f0aa05a57fc)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:172 +0x111
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).decodeObjectAt(0xc000148f68, 0x16f44e0, 0xc00000cae0, 0x16f3d60, 0xc000252880, 0xe30e46, 0x0, 0x0, 0xf29030d0779f9e95, 0xe30e46)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:301 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).getFromPackfile(0xc000148f68, 0x304b930673e10c87, 0x779f9e95b1674781, 0xf29030d0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:293 +0x2cc
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).EncodedObject(0xc000148f68, 0x4b930673e10c8703, 0x9f9e95b167478130, 0xf29030d077, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:174 +0x3f5
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetBlob(0x1f852a0, 0xc000148f50, 0x304b930673e10c87, 0x779f9e95b1674781, 0xf29030d0, 0x73e10c87000081a4, 0xb1674781304b9306, 0xf29030d0779f9e95)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/blob.go:23 +0x4e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*FileIter).Next(0xc000ccbd88, 0x167ecf0, 0xc000ccbd88, 0x100df28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/file.go:100 +0x136
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*FileIter).ForEach(0xc0000d1d88, 0xc0000d1d00, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/file.go:116 +0x61
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf8000, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:537 +0x9f9
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 38 [runnable]:
bufio.(*Reader).ReadBytes(0xc001e1e060, 0xc000044520, 0xc001dc9878, 0x1014975, 0xc001e1e060, 0xc001dc9968, 0x58)
	/usr/local/Cellar/go/1.11.1/libexec/src/bufio/bufio.go:442 +0x287
bufio.(*Reader).ReadString(0xc001e1e060, 0xc001e1e020, 0xc001dc9968, 0xc001e22000, 0x1000, 0x1000)
	/usr/local/Cellar/go/1.11.1/libexec/src/bufio/bufio.go:459 +0x38
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).Decode(0xc001e20000, 0x16f3d00, 0xc000cf2100, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:224 +0x22f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DecodeTree(0x1f852a0, 0xc000148f50, 0x16f3d00, 0xc000cf2100, 0x16f3d00, 0xc000cf2100, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:57 +0x73
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0x8d39453cbff2a2d5, 0x232664e5d65d8abe, 0xded131e9, 0xc000ca0230, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:50 +0xac
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Tree(0xc000bf80f0, 0xc000ca0230, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:76 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf81e0, 0x16f2380, 0xc0000260c0, 0xc000bf80f0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:87 +0x5a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf81e0, 0xc000bf80f0, 0xc000027820, 0xc0000304d0, 0xc000bf81e0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf81e0, 0xc000bf80f0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 39 [runnable]:
bytes.makeSlice.func1()
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:226 +0x69
bytes.makeSlice(0x600, 0xc001df8000, 0x600, 0x600)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:231 +0x8c
bytes.(*Buffer).grow(0xc000c707e0, 0x200, 0x200)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:144 +0x15a
bytes.(*Buffer).ReadFrom(0xc000c707e0, 0x2d65230, 0xc000c443c0, 0x1f9e698, 0xc000c707e0, 0x141d001)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:204 +0x4b
io.copyBuffer(0x16ec2c0, 0xc000c707e0, 0x2d65230, 0xc000c443c0, 0xc001de2000, 0x8000, 0x8000, 0x15b4ea0, 0xc000478500, 0x2d65230)
	/usr/local/Cellar/go/1.11.1/libexec/src/io/io.go:388 +0x303
io.CopyBuffer(0x16ec2c0, 0xc000c707e0, 0x2d65230, 0xc000c443c0, 0xc001de2000, 0x8000, 0x8000, 0xc000086900, 0xc000d64000, 0x16014a0)
	/usr/local/Cellar/go/1.11.1/libexec/src/io/io.go:375 +0x82
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).copyObject(0xc000cfe280, 0x16ec2c0, 0xc000c707e0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/scanner.go:297 +0x2a4
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).NextObject(0xc000cfe280, 0x16ec2c0, 0xc000c707e0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/scanner.go:272 +0xa2
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).fillOFSDeltaObjectContent(0xc000cfe2d0, 0x16f3d00, 0xc00015a1c0, 0x39c88, 0x7a, 0xc000d47780)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:353 +0x72
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).getNextObject(0xc000cfe2d0, 0xc000d2e180, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:303 +0x19e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).getObjectContent(0xc000cfe2d0, 0x4d66a, 0x0, 0x0, 0x16f44e0, 0xc000d30100)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:283 +0xd3
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*FSObject).Reader(0xc000dae000, 0x660, 0x9f0205c05eda1cd5, 0xc0ce43aad0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/fsobject.go:66 +0x18a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).Decode(0xc000cfe230, 0x16f3dc0, 0xc000dae000, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:216 +0x115
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DecodeTree(0x1f852a0, 0xc000148f50, 0x16f3dc0, 0xc000dae000, 0x16f3dc0, 0xc000dae000, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:57 +0x73
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0xe1fce1fc99ff517e, 0x9f0205c05eda1cd5, 0xce43aad0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:50 +0xac
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Tree(0xc000bf82d0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:76 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf82d0, 0x16f2380, 0xc0000260c0, 0xc000bf81e0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:82 +0x2f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf82d0, 0xc000bf81e0, 0xc000027820, 0xc0000304d0, 0xc000bf82d0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf82d0, 0xc000bf81e0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

Wiki: error , From # 8 To # 10 ⬇️

From

8. Running against a Github Organization

To

10. Running against a Github User

Error

1. The ❤️ two times ?
2. The Code 💛 --github-org three times ?
3. And NOTE issues ❓

Source -> Html

Running against a Github Organization ❤️

gitleaks --github-org=gitleakstestorg 💛

NOTE: you may want to use --disk if the organization you are auditing is large

Running against a Github Organization ❤️

gitleaks --github-org=gitleakstestorg 💛

NOTE1: you may want to use --disk if the organization you are auditing is large

NOTE2: you may want to use --log=debug to see which repos you will be auditing.

Running against a Github User

gitleaks --github-org=gitleakstest 💛

NOTE: you may want to use --disk if the user you are auditing is large

NOTE2: you may want to use --log=debug to see which repos you will be auditing.

Idea: port the leak checker to Hercules

Hercules is a nice extensible Git repo analysis engine on top of go-git. Should be very easy to write a plugin and effectively do not depend on GitHub API.

Add travis yaml

Need to setup travis

--github-org leaving with blank

Hello,
The flag "--github-org=some org url" not working and leaving with blank ?

Support other git url types

these are not properly supported

git://...
user@host:...

Organization/User Scan

Retrieve organization/user repos using https://api.github.com/orgs/<org name>/repos or https://api.github.com/users/<username>/repos. Iterate through repos, clone repo, generate report.

usage:
$ gitleaks -u https://github.com/<username>
$ gitleaks -o https://github.com/<org name>

checkRegex test fails

Looks like the test is failing because checkRegex is being assigned to two vars but only returns one, a string[]

https://github.com/zricethezav/gitleaks/blob/06c2141a02ce51ab8ba9726c5d03a4a3f41f0004/checks_test.go#L23

How to set GITHUB_TOKEN in Docker env?

Hey! Thanks for this great and important tool!

I'd like to scan my private repositories, and according to your wiki, I need to set the GITHUB_TOKEN. My question is, how do I do that running your app with the one-liner for docker? Where/how can I access the env file to configure it?

Thanks!

List of regexes for service secrets

Would love some help on gathering a list of regexes for services so that folks could search for a service they use. This list could be put into its own wiki page and referenced in the README

Any plans to enable scanning an existing cloned repo?

I don't need a separate copy of a repo cloned, I already have one locally, and want to scan it in place for leaks. Useful for CI stages like linting. Arguably this should be the default, since it's trivial to clone a repo yourself and cd into it or give the path to it.

Add support for GitHub Enterprise

When scanning single repositories it's possible to provide a custom URL to a GitHub Enterprise instance, however when running the Org/User scan the tool assumes the public GitHub instance.

It would be create to have an option like "Base GitHub URL" to override this to support GitHub Enterprise deployments.

Spanish Translation

Hi guys, I´m a ENG-SPA translator, I was wondering if you were interested in having a Spanish translation for your project.

Saludos.
Juan Hernández - Crowding Profile