Giter Site home page Giter Site logo

gitleaks / gitleaks Goto Github PK

View Code? Open in Web Editor NEW
15.3K 15.3K 1.3K 5.09 MB

Protect and discover secrets using Gitleaks ๐Ÿ”‘

Home Page: https://gitleaks.io

License: MIT License

Go 99.41% Dockerfile 0.15% Makefile 0.18% Python 0.26%
devsecops git gitleaks go golang hacktoberfest secret security security-tools

gitleaks's Introduction

Gitleaks

โ”Œโ”€โ—‹โ”€โ”€โ”€โ”
โ”‚ โ”‚โ•ฒ  โ”‚
โ”‚ โ”‚ โ—‹ โ”‚
โ”‚ โ—‹ โ–‘ โ”‚
โ””โ”€โ–‘โ”€โ”€โ”€โ”˜

Github Test gitleaks badge Follow @zricethezav

Join our Discord! Discord

Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

โžœ  ~/code(master) gitleaks detect --source . -v

    โ—‹
    โ”‚โ•ฒ
    โ”‚ โ—‹
    โ—‹ โ–‘
    โ–‘    gitleaks


Finding:     "export BUNDLE_ENTERPRISE__CONTRIBSYS__COM=cafebabe:deadbeef",
Secret:      cafebabe:deadbeef
RuleID:      sidekiq-secret
Entropy:     2.609850
File:        cmd/generate/config/rules/sidekiq.go
Line:        23
Commit:      cd5226711335c68be1e720b318b7bc3135a30eb2
Author:      John
Email:       [email protected]
Date:        2022-08-03T12:31:40Z
Fingerprint: cd5226711335c68be1e720b318b7bc3135a30eb2:cmd/generate/config/rules/sidekiq.go:sidekiq-secret:23

Getting Started

Gitleaks can be installed using Homebrew, Docker, or Go. Gitleaks is also available in binary form for many popular platforms and OS types on the releases page. In addition, Gitleaks can be implemented as a pre-commit hook directly in your repo or as a GitHub action using Gitleaks-Action.

Installing

# MacOS
brew install gitleaks

# Docker (DockerHub)
docker pull zricethezav/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path zricethezav/gitleaks:latest [COMMAND] --source="/path" [OPTIONS]

# Docker (ghcr.io)
docker pull ghcr.io/gitleaks/gitleaks:latest
docker run -v ${path_to_host_folder_to_scan}:/path ghcr.io/gitleaks/gitleaks:latest [COMMAND] --source="/path" [OPTIONS]

# From Source
git clone https://github.com/gitleaks/gitleaks.git
cd gitleaks
make build

GitHub Action

Check out the official Gitleaks GitHub Action

name: gitleaks
on: [pull_request, push, workflow_dispatch]
jobs:
  scan:
    name: gitleaks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE}} # Only required for Organizations, not personal accounts.

Pre-Commit

  1. Install pre-commit from https://pre-commit.com/#install

  2. Create a .pre-commit-config.yaml file at the root of your repository with the following content:

    repos:
      - repo: https://github.com/gitleaks/gitleaks
        rev: v8.16.1
        hooks:
          - id: gitleaks
    

    for a native execution of GitLeaks or use the gitleaks-docker pre-commit ID for executing GitLeaks using the official Docker images

  3. Auto-update the config to the latest repos' versions by executing pre-commit autoupdate

  4. Install with pre-commit install

  5. Now you're all set!

โžœ git commit -m "this commit contains a secret"
Detect hardcoded secrets.................................................Failed

Note: to disable the gitleaks pre-commit hook you can prepend SKIP=gitleaks to the commit command and it will skip running gitleaks

โžœ SKIP=gitleaks git commit -m "skip gitleaks check"
Detect hardcoded secrets................................................Skipped

Usage

Usage:
  gitleaks [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  detect      detect secrets in code
  help        Help about any command
  protect     protect secrets in code
  version     display gitleaks version

Flags:
  -b, --baseline-path string       path to baseline with issues that can be ignored
  -c, --config string              config file path
                                   order of precedence:
                                   1. --config/-c
                                   2. env var GITLEAKS_CONFIG
                                   3. (--source/-s)/.gitleaks.toml
                                   If none of the three options are used, then gitleaks will use the default config
      --exit-code int              exit code when leaks have been encountered (default 1)
  -h, --help                       help for gitleaks
  -l, --log-level string           log level (trace, debug, info, warn, error, fatal) (default "info")
      --max-target-megabytes int   files larger than this will be skipped
      --no-color                   turn off color for verbose output
      --no-banner                  suppress banner
      --redact                     redact secrets from logs and stdout
  -f, --report-format string       output format (json, csv, junit, sarif) (default "json")
  -r, --report-path string         report file
  -s, --source string              path to source (default ".")
  -v, --verbose                    show verbose output from scan

Use "gitleaks [command] --help" for more information about a command.

Commands

There are two commands you will use to detect secrets; detect and protect.

Detect

The detect command is used to scan repos, directories, and files. This command can be used on developer machines and in CI environments.

When running detect on a git repository, gitleaks will parse the output of a git log -p command (you can see how this executed here). git log -p generates patches which gitleaks will use to detect secrets. You can configure what commits git log will range over by using the --log-opts flag. --log-opts accepts any option for git log -p. For example, if you wanted to run gitleaks on a range of commits you could use the following command: gitleaks detect --source . --log-opts="--all commitA..commitB". See the git log documentation for more information.

You can scan files and directories by using the --no-git option.

If you want to run only specific rules you can do so by using the --enable-rule option (with a rule ID as a parameter), this flag can be used multiple times. For example: --enable-rule=atlassian-api-token will only apply that rule. You can find a list of rules here.

Protect

The protect command is used to scan uncommitted changes in a git repo. This command should be used on developer machines in accordance with shifting left on security. When running protect on a git repository, gitleaks will parse the output of a git diff command (you can see how this executed here). You can set the --staged flag to check for changes in commits that have been git added. The --staged flag should be used when running Gitleaks as a pre-commit.

NOTE: the protect command can only be used on git repos, running protect on files or directories will result in an error message.

Creating a baseline

When scanning large repositories or repositories with a long history, it can be convenient to use a baseline. When using a baseline, gitleaks will ignore any old findings that are present in the baseline. A baseline can be any gitleaks report. To create a gitleaks report, run gitleaks with the --report-path parameter.

gitleaks detect --report-path gitleaks-report.json # This will save the report in a file called gitleaks-report.json

Once as baseline is created it can be applied when running the detect command again:

gitleaks detect --baseline-path gitleaks-report.json --report-path findings.json

After running the detect command with the --baseline-path parameter, report output (findings.json) will only contain new issues.

Verify Findings

You can verify a finding found by gitleaks using a git log command. Example output:

Finding:     aws_secret="AKIAIMNOJVGFDXXXE4OA"
RuleID:      aws-access-token
Secret       AKIAIMNOJVGFDXXXE4OA
Entropy:     3.65
File:        checks_test.go
Line:        37
Commit:      ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29
Author:      Zachary Rice
Email:       [email protected]
Date:        2018-01-28T17:39:00Z
Fingerprint: ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29:checks_test.go:aws-access-token:37

We can use the following format to verify the leak:

git log -L {StartLine,EndLine}:{File} {Commit}

So in this example it would look like:

git log -L 37,37:checks_test.go ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29

Which gives us:

commit ec2fc9d6cb0954fb3b57201cf6133c48d8ca0d29
Author: zricethezav <[email protected]>
Date:   Sun Jan 28 17:39:00 2018 -0500

    [update] entropy check

diff --git a/checks_test.go b/checks_test.go
--- a/checks_test.go
+++ b/checks_test.go
@@ -28,0 +37,1 @@
+               "aws_secret= \"AKIAIMNOJVGFDXXXE4OA\"":          true,

Pre-Commit hook

You can run Gitleaks as a pre-commit hook by copying the example pre-commit.py script into your .git/hooks/ directory.

Configuration

Gitleaks offers a configuration format you can follow to write your own secret detection rules:

# Title for the gitleaks configuration file.
title = "Gitleaks title"

# Extend the base (this) configuration. When you extend a configuration
# the base rules take precedence over the extended rules. I.e., if there are
# duplicate rules in both the base configuration and the extended configuration
# the base rules will override the extended rules.
# Another thing to know with extending configurations is you can chain together
# multiple configuration files to a depth of 2. Allowlist arrays are appended
# and can contain duplicates.
# useDefault and path can NOT be used at the same time. Choose one.
[extend]
# useDefault will extend the base configuration with the default gitleaks config:
# https://github.com/zricethezav/gitleaks/blob/master/config/gitleaks.toml
useDefault = true
# or you can supply a path to a configuration. Path is relative to where gitleaks
# was invoked, not the location of the base config.
path = "common_config.toml"

# An array of tables that contain information that define instructions
# on how to detect secrets
[[rules]]

# Unique identifier for this rule
id = "awesome-rule-1"

# Short human readable description of the rule.
description = "awesome rule 1"

# Golang regular expression used to detect secrets. Note Golang's regex engine
# does not support lookaheads.
regex = '''one-go-style-regex-for-this-rule'''

# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used
# in conjunction with a valid `regex` entry.
path = '''a-file-path-regex'''

# Array of strings used for metadata and reporting purposes.
tags = ["tag","another tag"]

# Int used to extract secret from regex match and used as the group that will have
# its entropy checked if `entropy` is set.
secretGroup = 3

# Float representing the minimum shannon entropy a regex group must have to be considered a secret.
entropy = 3.5

# Keywords are used for pre-regex check filtering. Rules that contain
# keywords will perform a quick string compare check to make sure the
# keyword(s) are in the content being scanned. Ideally these values should
# either be part of the idenitifer or unique strings specific to the rule's regex
# (introduced in v8.6.0)
keywords = [
  "auth",
  "password",
  "token",
]

# You can include an allowlist table for a single rule to reduce false positives or ignore commits
# with known/rotated secrets
[rules.allowlist]
description = "ignore commit A"
commits = [ "commit-A", "commit-B"]
paths = [
  '''go\.mod''',
  '''go\.sum'''
]
# note: (rule) regexTarget defaults to check the _Secret_ in the finding.
# if regexTarget is not specified then _Secret_ will be used.
# Acceptable values for regexTarget are "match" and "line"
regexTarget = "match"
regexes = [
  '''process''',
  '''getenv''',
]
# note: stopwords targets the extracted secret, not the entire regex match
# like 'regexes' does. (stopwords introduced in 8.8.0)
stopwords = [
  '''client''',
  '''endpoint''',
]


# This is a global allowlist which has a higher order of precedence than rule-specific allowlists.
# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no
# secrets will be detected for said commit. The same logic applies for regexes and paths.
[allowlist]
description = "global allow list"
commits = [ "commit-A", "commit-B", "commit-C"]
paths = [
  '''gitleaks\.toml''',
  '''(.*?)(jpg|gif|doc)'''
]

# note: (global) regexTarget defaults to check the _Secret_ in the finding.
# if regexTarget is not specified then _Secret_ will be used.
# Acceptable values for regexTarget are "match" and "line"
regexTarget = "match"

regexes = [
  '''219-09-9999''',
  '''078-05-1120''',
  '''(9[0-9]{2}|666)-\d{2}-\d{4}''',
]
# note: stopwords targets the extracted secret, not the entire regex match
# like 'regexes' does. (stopwords introduced in 8.8.0)
stopwords = [
  '''client''',
  '''endpoint''',
]

Refer to the default gitleaks config for examples or follow the contributing guidelines if you would like to contribute to the default configuration. Additionally, you can check out this gitleaks blog post which covers advanced configuration setups.

Additional Configuration

gitleaks:allow

If you are knowingly committing a test secret that gitleaks will catch you can add a gitleaks:allow comment to that line which will instruct gitleaks to ignore that secret. Ex:

class CustomClass:
    discord_client_secret = '8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ'  #gitleaks:allow

.gitleaksignore

You can ignore specific findings by creating a .gitleaksignore file at the root of your repo. In release v8.10.0 Gitleaks added a Fingerprint value to the Gitleaks report. Each leak, or finding, has a Fingerprint that uniquely identifies a secret. Add this fingerprint to the .gitleaksignore file to ignore that specific secret. See Gitleaks' .gitleaksignore for an example. Note: this feature is experimental and is subject to change in the future.

Sponsorships

Tines Sponsorship

Exit Codes

You can always set the exit code when leaks are encountered with the --exit-code flag. Default exit codes below:

0 - no leaks present
1 - leaks or error encountered
126 - unknown flag

gitleaks's People

Contributors

adamkobi avatar alexandear avatar amith-legit avatar arjunyel avatar baruchiro avatar becojo avatar bufferoverflow avatar chenrui333 avatar craigmarksmith avatar denwwer avatar devnixs avatar electriquo avatar eripa avatar grahlmanmatthew avatar ido-dy avatar joostvoskuil avatar maltemorgenstern avatar milo-minderbinder avatar noelalgora avatar petegallagher avatar realzcong avatar rgmz avatar sadikkuzu avatar savely-krasovsky avatar sergiomarotco avatar very-doge-wow avatar w0rmr1d3r avatar wayne-snyk avatar ziyaddin avatar zricethezav avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gitleaks's Issues

Timings compared to truffleHog

I would like to add a timing chart to the README comparing times and false positives between truffleHog/similar tools

Entropy checks

Although the first version did entropy checks apart from regexes (https://github.com/zricethezav/gitleaks/blob/v0.1.0/checks.go#L34), that's no longer the case in the latest version.

I know entropy checks could create lots of false positives, but they can also catch things that regex don't so I see a lot of value in there. Also, using the whitelisting options can reduce a lot of the false positives.

Could this be added again?

Go not building/installing

Hello,

This is my first time installing a go repo. I did sudo go get -u -v -t github.com/zricethezav/gitleaks and it only came back with this response:

github.com/zricethezav/gitleaks (download)

So I'm not quite sure if I'm not building correctly or what else I could be doing wrong. My go version is 1.8.3 and I'm on linux/amd64 Ubuntu 17.10. Sorry about my incompetence!

And I did try installing the binaries but I'm not sure how to do those either.

more stopWords please

  • BEGIN CERTIFICATE
    (.crt CERTIFICATE)
  • BEGIN RSA PRIVATE KEY
    (.key PRIVATE KEY)
  • secret
    (Oauth or jwt.io or other API but passed the regx check)

and there some false alarm with

[
	{
		"content": "+[![Cocoapods](https://cocoapod-badges.herokuapp.com/v/DateTools/badge.png)](http://cocoapods.org/?q=datetools)",
		"commit": "70b07b600c79cfaf7dbcc68402dbda73d32e965e"
	},
	{
		"content": "+[![Cocoapods](https://cocoapod-badges.herokuapp.com/v/DateTools/badge.png)](http://cocoapods.org/?q=datetools)",
		"commit": "f6f9cbe91ce9564ffe65c3dede2dbab017d95eb5"
	}
]

Spanish Translation

Hi guys, Iยดm a ENG-SPA translator, I was wondering if you were interested in having a Spanish translation for your project.

Saludos.
Juan Hernรกndez - Crowding Profile

gitleaks scan bypasses RSA key

This happens for some of my repositories when i did scan today. However, the same scan has been done on October 17th, listed all leaks but now it doesn't for few repositories. The same RSA keys have been identified in some repo's and got listed in the report.

I did check the files where it actually found leaks in my last scan. The keys still exists.
No changes made to those repositories from last 4 months.

Can anyone help on this. Will provide any information if required.

Persistant/Local Repos

A feature that has been requested by a few folks is the ability to audit local repos as scheduled tasks. @JustinAzoff in #8 suggested using a data store like boltDB to cache commits so we don't run redundant checks on commits that have already been vetted.

I'm thinking we keep the default behavior but add some options like a --dir, -d flag to tell gitleaks run this for a local repo: $ gitleaks --dir {my_repo}. For persistent support we can add a --persist, -p flag that tell gitleaks to skip the rm -rf call and check the data store before running all checks.

"fatal error: concurrent map read and map write" scanning repository

Version: Built from master (87d916f as of writing)
OS: macOS 10.14
Go: go version go1.11.1 darwin/amd64

When running 'gitleaks' against a local repository I have (gitleaks --repo-path=path/to/my/repo), gitleaks panics with message listed below.

Unfortunately, the repository is private so I cannot share it, but if there's any information I can give to help narrow this down or steps I can take, I am happy to. Thank you.

fatal error: concurrent map read and map write

goroutine 37 [running]:
runtime.throw(0x166cf3b, 0x21)
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/panic.go:608 +0x72 fp=0xc000d43178 sp=0xc000d43148 pc=0x102c342
runtime.mapaccess2_fast64(0x15888c0, 0xc00001f800, 0x32ec0d, 0xc000d2e240, 0x3f)
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/map_fast64.go:61 +0x1a8 fp=0xc000d431a0 sp=0xc000d43178 pc=0x1012668
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).FindHash(0xc000252880, 0x32ec0d, 0x0, 0x0, 0x0, 0x14, 0xf03e1f0aa05a57fc)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:177 +0x6b fp=0xc000d43200 sp=0xc000d431a0 pc=0x140c47b
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).decodeObjectAt(0xc000148f68, 0x16f44e0, 0xc000d301c0, 0x16f3d60, 0xc000252880, 0x32ec0d, 0x0, 0x0, 0x662e5d6c66d49473, 0x32ec0d)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:301 +0x56 fp=0xc000d43290 sp=0xc000d43200 pc=0x1437676
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).getFromPackfile(0xc000148f68, 0x2f9db7ce84927f1f, 0x66d49473dcedf309, 0x662e5d6c, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:293 +0x2cc fp=0xc000d43398 sp=0xc000d43290 pc=0x143749c
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).EncodedObject(0xc000148f68, 0x9db7ce84927f1f02, 0xd49473dcedf3092f, 0x662e5d6c66, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:174 +0x3f5 fp=0xc000d43438 sp=0xc000d43398 pc=0x1436c05
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0x2f9db7ce84927f1f, 0x66d49473dcedf309, 0x662e5d6c, 0x203000, 0x203000, 0x1020f28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:45 +0x4e fp=0xc000d43488 sp=0xc000d43438 pc=0x13bb95e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*TreeWalker).Next(0xc000d436c0, 0xc000cfe5a0, 0x9, 0xc000d62140, 0x14, 0x84927f1f00004000, 0xdcedf3092f9db7ce, 0x662e5d6c66d49473, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:408 +0x56f fp=0xc000d435c8 sp=0xc000d43488 pc=0x13bdf7f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.transformChildren(0xc000c44230, 0x15efbc0, 0xc001dec701, 0x1b73280, 0x28, 0x38)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/treenoder.go:105 +0x21d fp=0xc000d43720 sp=0xc000d435c8 pc=0x13be9bd
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*treeNoder).Children(0xc000cfe3c0, 0x1b73280, 0x1f41b00, 0xc000001c80, 0x203000, 0x1020f28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/treenoder.go:87 +0x61 fp=0xc000d43760 sp=0xc000d43720 pc=0x13be6b1
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/internal/frame.New(0x16f3760, 0xc000cfe3c0, 0x48, 0x48, 0xc000cfe460)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/internal/frame/frame.go:29 +0x3b fp=0xc000d437e0 sp=0xc000d43760 pc=0x133fcfb
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.newIter(0x16f3760, 0xc000cfe3c0, 0x0, 0x0, 0x0, 0x15efc80, 0x1, 0xc000cfe460)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/iter.go:90 +0x87 fp=0xc000d43840 sp=0xc000d437e0 pc=0x1342db7
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.NewIter(0x16f3760, 0xc000cfe3c0, 0x1, 0x16ec5c0, 0xc00006a060)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/iter.go:71 +0x46 fp=0xc000d43890 sp=0xc000d43840 pc=0x1342c26
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.newDoubleIter(0x16f3760, 0xc000cfe3c0, 0x16f3760, 0xc000cfe410, 0x167ecf8, 0xc000001c80, 0x1000, 0x1000)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/doubleiter.go:43 +0x67 fp=0xc000d43940 sp=0xc000d43890 pc=0x1341ed7
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie.DiffTreeContext(0x16f2380, 0xc0000260c0, 0x16f3760, 0xc000cfe3c0, 0x16f3760, 0xc000cfe410, 0x167ecf8, 0x13bb9bc, 0x1f852a0, 0xc000148f50, ...)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/utils/merkletrie/difftree.go:277 +0x96 fp=0xc000d439d0 sp=0xc000d43940 pc=0x1341046
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DiffTreeContext(0x16f2380, 0xc0000260c0, 0xc000c44230, 0xc000cfe320, 0x8a0f72a6f3d94d5e, 0x3faba7ce, 0xc000cfe320, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/difftree.go:28 +0x157 fp=0xc000d43a58 sp=0xc000d439d0 pc=0x13b5b37
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).PatchContext(0xc000c44230, 0x16f2380, 0xc0000260c0, 0xc000cfe320, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:316 +0x4d fp=0xc000d43ab0 sp=0xc000d43a58 pc=0x13bd89d
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf80f0, 0x16f2380, 0xc0000260c0, 0xc000bf8000, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:92 +0xb5 fp=0xc000d43b00 sp=0xc000d43ab0 pc=0x13b0c55
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf80f0, 0xc000bf8000, 0xc000027820, 0xc0000304d0, 0xc000bf80f0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d fp=0xc000d43b48 sp=0xc000d43b00 pc=0x13b0cfd
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf80f0, 0xc000bf8000)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2 fp=0xc000d43f98 sp=0xc000d43b48 pc=0x14fae92
runtime.goexit()
	/usr/local/Cellar/go/1.11.1/libexec/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000d43fa0 sp=0xc000d43f98 pc=0x1058911
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 1 [chan send]:
main.auditGitReference.func1(0xc000bf83c0, 0xc000bf83c0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:520 +0x32b
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*commitPreIterator).ForEach(0xc00012e720, 0xc000a1ad20, 0x16f1600, 0xc00012e720)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit_walker.go:103 +0x65
main.auditGitReference(0xc0000af1d0, 0xc00015a540, 0x0, 0x0, 0xc00013c1c0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:499 +0x279
main.auditGitRepo(0xc0000af1d0, 0x0, 0x0, 0xc00011ddb0, 0xd93f1f82, 0xbed1d93f1f82)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:470 +0x23c
main.run(0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:306 +0x105
main.main()
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:251 +0xea

goroutine 36 [runnable]:
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*idxfileEntryIter).Next(0xc001dd4020, 0xc00001f800, 0xc35fa1, 0xc001651ae8)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:266 +0x25a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).genOffsetHash(0xc000252880, 0x142c196, 0xc00013c9c0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:200 +0x110
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile.(*MemoryIndex).FindHash(0xc000252880, 0xe30e46, 0x0, 0x0, 0x0, 0x14, 0xf03e1f0aa05a57fc)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/idxfile/idxfile.go:172 +0x111
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).decodeObjectAt(0xc000148f68, 0x16f44e0, 0xc00000cae0, 0x16f3d60, 0xc000252880, 0xe30e46, 0x0, 0x0, 0xf29030d0779f9e95, 0xe30e46)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:301 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).getFromPackfile(0xc000148f68, 0x304b930673e10c87, 0x779f9e95b1674781, 0xf29030d0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:293 +0x2cc
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem.(*ObjectStorage).EncodedObject(0xc000148f68, 0x4b930673e10c8703, 0x9f9e95b167478130, 0xf29030d077, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/storage/filesystem/object.go:174 +0x3f5
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetBlob(0x1f852a0, 0xc000148f50, 0x304b930673e10c87, 0x779f9e95b1674781, 0xf29030d0, 0x73e10c87000081a4, 0xb1674781304b9306, 0xf29030d0779f9e95)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/blob.go:23 +0x4e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*FileIter).Next(0xc000ccbd88, 0x167ecf0, 0xc000ccbd88, 0x100df28)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/file.go:100 +0x136
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*FileIter).ForEach(0xc0000d1d88, 0xc0000d1d00, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/file.go:116 +0x61
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf8000, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:537 +0x9f9
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 38 [runnable]:
bufio.(*Reader).ReadBytes(0xc001e1e060, 0xc000044520, 0xc001dc9878, 0x1014975, 0xc001e1e060, 0xc001dc9968, 0x58)
	/usr/local/Cellar/go/1.11.1/libexec/src/bufio/bufio.go:442 +0x287
bufio.(*Reader).ReadString(0xc001e1e060, 0xc001e1e020, 0xc001dc9968, 0xc001e22000, 0x1000, 0x1000)
	/usr/local/Cellar/go/1.11.1/libexec/src/bufio/bufio.go:459 +0x38
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).Decode(0xc001e20000, 0x16f3d00, 0xc000cf2100, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:224 +0x22f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DecodeTree(0x1f852a0, 0xc000148f50, 0x16f3d00, 0xc000cf2100, 0x16f3d00, 0xc000cf2100, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:57 +0x73
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0x8d39453cbff2a2d5, 0x232664e5d65d8abe, 0xded131e9, 0xc000ca0230, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:50 +0xac
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Tree(0xc000bf80f0, 0xc000ca0230, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:76 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf81e0, 0x16f2380, 0xc0000260c0, 0xc000bf80f0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:87 +0x5a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf81e0, 0xc000bf80f0, 0xc000027820, 0xc0000304d0, 0xc000bf81e0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf81e0, 0xc000bf80f0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

goroutine 39 [runnable]:
bytes.makeSlice.func1()
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:226 +0x69
bytes.makeSlice(0x600, 0xc001df8000, 0x600, 0x600)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:231 +0x8c
bytes.(*Buffer).grow(0xc000c707e0, 0x200, 0x200)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:144 +0x15a
bytes.(*Buffer).ReadFrom(0xc000c707e0, 0x2d65230, 0xc000c443c0, 0x1f9e698, 0xc000c707e0, 0x141d001)
	/usr/local/Cellar/go/1.11.1/libexec/src/bytes/buffer.go:204 +0x4b
io.copyBuffer(0x16ec2c0, 0xc000c707e0, 0x2d65230, 0xc000c443c0, 0xc001de2000, 0x8000, 0x8000, 0x15b4ea0, 0xc000478500, 0x2d65230)
	/usr/local/Cellar/go/1.11.1/libexec/src/io/io.go:388 +0x303
io.CopyBuffer(0x16ec2c0, 0xc000c707e0, 0x2d65230, 0xc000c443c0, 0xc001de2000, 0x8000, 0x8000, 0xc000086900, 0xc000d64000, 0x16014a0)
	/usr/local/Cellar/go/1.11.1/libexec/src/io/io.go:375 +0x82
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).copyObject(0xc000cfe280, 0x16ec2c0, 0xc000c707e0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/scanner.go:297 +0x2a4
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Scanner).NextObject(0xc000cfe280, 0x16ec2c0, 0xc000c707e0, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/scanner.go:272 +0xa2
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).fillOFSDeltaObjectContent(0xc000cfe2d0, 0x16f3d00, 0xc00015a1c0, 0x39c88, 0x7a, 0xc000d47780)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:353 +0x72
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).getNextObject(0xc000cfe2d0, 0xc000d2e180, 0x0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:303 +0x19e
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*Packfile).getObjectContent(0xc000cfe2d0, 0x4d66a, 0x0, 0x0, 0x16f44e0, 0xc000d30100)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/packfile.go:283 +0xd3
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile.(*FSObject).Reader(0xc000dae000, 0x660, 0x9f0205c05eda1cd5, 0xc0ce43aad0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/format/packfile/fsobject.go:66 +0x18a
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Tree).Decode(0xc000cfe230, 0x16f3dc0, 0xc000dae000, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:216 +0x115
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.DecodeTree(0x1f852a0, 0xc000148f50, 0x16f3dc0, 0xc000dae000, 0x16f3dc0, 0xc000dae000, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:57 +0x73
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.GetTree(0x1f852a0, 0xc000148f50, 0xe1fce1fc99ff517e, 0x9f0205c05eda1cd5, 0xce43aad0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/tree.go:50 +0xac
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Tree(0xc000bf82d0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:76 +0x56
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).PatchContext(0xc000bf82d0, 0x16f2380, 0xc0000260c0, 0xc000bf81e0, 0x0, 0x0, 0x0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:82 +0x2f
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Commit).Patch(0xc000bf82d0, 0xc000bf81e0, 0xc000027820, 0xc0000304d0, 0xc000bf82d0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/commit.go:97 +0x4d
main.auditGitReference.func1.1(0xc000027820, 0xc0000304d0, 0xc00015a540, 0x165d595, 0x1, 0xc000027830, 0xc00013d1e0, 0xc000bf82d0, 0xc000bf81e0)
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:564 +0xc2
created by main.auditGitReference.func1
	/Users/robert.hencke/src/github.com/zricethezav/gitleaks/main.go:521 +0x3af

Output URL to specific repo in JSON

It would be nice if the output JSON would contain the URL to the specific repo. Since different orgs can have repos of the same name, it's a bit cumbersome to have to go back and figure out which specific repo has the leaks.

Thanks

Add support for github.*

For enterprises that host their own private github (e.g. github.dhs.gov, github.asp.net, etc...), it appears that the github links aren't recognized by the code as valid github links, so the error Unknown option https://github.asp.net shows up. It would be nice if .gov and .net addresses were recognized in addition to the base github.com url.

Is there somewhere in your code that you're validating this? I could try and help implement this...

scanning github orgs' private repos

trying to use the --github-org option, but it's only targetting public repos. my API key definitely has full repo access (to public and private), what am I missing?

[feature request] define disk temp path

Thanks for such amazing tool!
I'm scanning a huge organization and not sure if the space in my disk will be enough to store everything under /tmp/ directory.
So my suggest is include an optional parameter to define the temp directory when --disk is used.
I'd like to help implement that but I don't know golang enough to contribute.

Note: Gitleaks rewrite coming

Hey all,

First of all, thanks for all the support. I'll be honest - I made gitleaks as an exercise to learn Go so it's a pretty novice code base. Since then I've learned some things and am working on a rewrite that will add some cool features such as branch and PR targeting, as well as support for any remote target (gitleaks will no longer be github specific when cloning remote). Additionally, I'm making use of src-d's go-git plumbing to make the audits quicker with an 'in-memory' option -- no more shelling out git x commands.

I'd like to open the floor for feature requests here. Shout out to @keirans who has already come up with some features:

  • externalizing regex
  • cleaner options
  • white listing commits
  • regex matching on commit messages

Please add a comment if there is a feature you would like to see in the next gitleaks version

Authentication Error

I am running a scan against a repo that we, However it is giving me authentication error message even to I can clone the repo without any issue.

NFO[2018-09-17T11:11:26-04:00] cloning https://github.*********.com/*******/suite.git
WARN[2018-09-17T11:11:26-04:00] skipping audit for repo suite.git due to cloning error: authentication required
INFO[2018-09-17T11:11:26-04:00] writing report to %s/tmp/leaks

git clone command for the above repo runs perfectly fine.

Add support for GitHub Enterprise

When scanning single repositories it's possible to provide a custom URL to a GitHub Enterprise instance, however when running the Org/User scan the tool assumes the public GitHub instance.

It would be create to have an option like "Base GitHub URL" to override this to support GitHub Enterprise deployments.

Wiki: error , From # 8 To # 10 โฌ‡๏ธ

From

To

Error

  • 1. The โค๏ธ two times ?
  • 2. The Code ๐Ÿ’› --github-org three times ?
  • 3. And NOTE issues โ“

Source -> Html

  1. Running against a Github Organization โค๏ธ
gitleaks --github-org=gitleakstestorg ๐Ÿ’› 

NOTE: you may want to use --disk if the organization you are auditing is large

  1. Running against a Github Organization โค๏ธ
gitleaks --github-org=gitleakstestorg ๐Ÿ’› 

NOTE1: you may want to use --disk if the organization you are auditing is large

NOTE2: you may want to use --log=debug to see which repos you will be auditing.

  1. Running against a Github User
gitleaks --github-org=gitleakstest ๐Ÿ’› 

NOTE: you may want to use --disk if the user you are auditing is large

NOTE2: you may want to use --log=debug to see which repos you will be auditing.

Organization/User Scan

Retrieve organization/user repos using https://api.github.com/orgs/<org name>/repos or https://api.github.com/users/<username>/repos. Iterate through repos, clone repo, generate report.

usage:
$ gitleaks -u https://github.com/<username>
$ gitleaks -o https://github.com/<org name>

Exclude organization repository

Can we somehow exclude forked repositories or skip the list of repo names?

$ gitleaks --github-org=companyltd --exclude-forks
... cloning: ...
... skipping: [fork] repo

or (combination):

$ gitleaks --github-org=companyltd --exclude=redhat,other-name
... cloning: ...
... skipping: redhat

Private Repositories - API Token

Hello!
First I'd like to express my thanks for this awesome project so thank you :)

Is there a reason why I cannot search private repos after providing gitleaks with a read-only API token? It still asks me for my github.com username and password after passing in my --token.

e.g., command:

./gitleaks --temp --token= --verbose --org https://github.com/myOrg

Group secrets in the report

I have found the report a bit noisy when one possible secret has been found multiple times and I have had to spend some time reviewing them. I'd say that it'd be better to report a possible secret only once and show an array of commits where it was found, here's an example:

[
    {
        "offender": "AKIALALEMEL33243OLIA",
        "reason": "AWS",
        "commits": [
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "AKIALALEMEL33243OLIAE",
                "message": "Add secrets\n"
            },
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "AKIALALEMEL33243OLIAE",
                "message": "Add secrets\n"
            }
        ]
    },
    {
        "offender": "62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
        "reason": "High Entropy",
        "commits": [
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "secret = 62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
                "message": "Add secrets"
            },
            {
                "authorEmail": "[email protected]",
                "authorName": "Foo Bar",
                "branch": "refs/heads/master",
                "date": "2017-04-07 10-47:27",
                "file": "README.md",
                "hash": "cd9fa7ffef0bdfe58274605ea6fc2f883fd3fb90",
                "line": "secret = 62cdb7020ff920e5aa642c3d4066950dd1f01f4d",
                "message": "Add secrets"
            }
        ]
    }
]

What do you think about this?

Scan for "secret_key"

This should be added to scan beacause we can have some recaptcha_secret_key and stuff like that in the repositories.

panic: runtime error: invalid memory address or nil pointer dereference

Error:

โžœ  Downloads ./gitleaks-linux-amd64 --repo-path=/home/projects/test 
INFO[2018-07-24T01:09:57+03:00] cloning                                      
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8c4fc5]

goroutine 1176 [running]:
github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object.(*Patch).FilePatches(...)
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/vendor/gopkg.in/src-d/go-git.v4/plumbing/object/patch.go:125
main.auditRef.func1.1(0xc4201ac040, 0x0, 0x0, 0xc4200909c0, 0xc432bfc1e0, 0xc432bfc0f0)
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/main.go:355 +0x55
created by main.auditRef.func1
	/Users/zach/Go/src/github.com/zricethezav/gitleaks/main.go:348 +0x1a0

System:

โžœ  Downloads ./gitleaks-linux-amd64 --version
1.1.2
โžœ  Downloads uname -a
Linux NB-NIX 4.17.8-1-ARCH #1 SMP PREEMPT Wed Jul 18 09:56:24 UTC 2018 x86_64 GNU/Linux

have other repos, and it works with them


what else do you need?

Output full path in JSON

A number of the results I've come across so far have been been in test type directories. If the full path to the file was available in the JSON it would make it quicker at a glance to easily dismiss results without having to track down the leak manually

Thanks

Does the docker image work for any git repository?

I'm trying to use the docker image to test a git repository that is hosted on an internal GitLab instance. And I get an error:

$ docker run --rm --name=gitleaks zricethezav/gitleaks https://gitlab.internal/mygroup/myproject

usage: gitleaks [options] <URL>/<path_to_repo>

Options:
 -u --user 		Git user mode
 -r --repo 		Git repo mode
 -o --org 		Git organization mode
 -l --local 		Local mode, gitleaks will look for local repo in <path>
 -v --verbose 		Verbose mode, will output leaks as gitleaks finds them
 --report-path=<STR> 	Report output, default $GITLEAKS_HOME/report
 --clone-path=<STR>	Gitleaks will clone repos here, default $GITLEAKS_HOME/clones
 -t --temp 		Clone to temporary directory
 --concurrency=<INT> 	Upper bound on concurrent "git diff"
 --since=<STR> 		Commit to stop at
 --b64Entropy=<INT> 	Base64 entropy cutoff (default is 70)
 --hexEntropy=<INT>  	Hex entropy cutoff (default is 40)
 -e --entropy		Enable entropy		
 -h --help 		Display this message
 --token=<STR>    	Github API token
 --stopwords  		Enables stopwords

Unknown option https://gitlab.internal/mygroup/myproject

usage: gitleaks [options] <URL>/<path_to_repo>

Options:
 -u --user 		Git user mode
 -r --repo 		Git repo mode
 -o --org 		Git organization mode
 -l --local 		Local mode, gitleaks will look for local repo in <path>
 -v --verbose 		Verbose mode, will output leaks as gitleaks finds them
 --report-path=<STR> 	Report output, default $GITLEAKS_HOME/report
 --clone-path=<STR>	Gitleaks will clone repos here, default $GITLEAKS_HOME/clones
 -t --temp 		Clone to temporary directory
 --concurrency=<INT> 	Upper bound on concurrent "git diff"
 --since=<STR> 		Commit to stop at
 --b64Entropy=<INT> 	Base64 entropy cutoff (default is 70)
 --hexEntropy=<INT>  	Hex entropy cutoff (default is 40)
 -e --entropy		Enable entropy		
 -h --help 		Display this message
 --token=<STR>    	Github API token
 --stopwords  		Enables stopwords

What is this Unknown option https://gitlab.internal/mygroup/myproject error? Where does it come from?

Could it be a proxy related issue? My company is using proxies.

Any plans to enable scanning an existing cloned repo?

I don't need a separate copy of a repo cloned, I already have one locally, and want to scan it in place for leaks. Useful for CI stages like linting. Arguably this should be the default, since it's trivial to clone a repo yourself and cd into it or give the path to it.

How to set GITHUB_TOKEN in Docker env?

Hey! Thanks for this great and important tool!

I'd like to scan my private repositories, and according to your wiki, I need to set the GITHUB_TOKEN. My question is, how do I do that running your app with the one-liner for docker? Where/how can I access the env file to configure it?

Thanks!

List of regexes for service secrets

Would love some help on gathering a list of regexes for services so that folks could search for a service they use. This list could be put into its own wiki page and referenced in the README

gitleaks --verbose

gitleaks --verbose option not working even not listing in all options using help.

Add support for scanning an enterprise Github organization

Tried scanning an organization in our enterprise github and I get an error:

gitleaks -o https://github.enterprise.com/enterprisegithub
2018/02/23 09:32:43 Get https://api.github.com/orgs/enterprisegithub/repos: read tcp 10.152.32.109:52205->192.30.253.116:443: read: connection reset by peer

It's clear now it doesn't use the full url provided so when I attempt to provide the full url according the usage from the help message, nothing happens:

gitleaks -o enterprisegithub https://github.enterprise.com/

Uknown option https://github.enterprise.com/

usage: gitleaks [options] <url>

Scanning a single repo does work on the other hand, but has to be done to every repo in the org.

Git leaks - Usability feedback

Hey There,
Thanks again for a really great tool. I'm providing this feedback here, as per our discussions on reddit DM's.

I've got a bit of feedback that i think would help greatly with general usability. I'm slightly selfish in that these would greatly help me, however, I think it would also help other peoples adoption as well.

  • Customise the datadir / workdir directory on the CLI
    Currently all data produced by the tool goes into ~/.gitleaks/ , however, for larger audits, it would be really great to be able to specify this directory so that the user could target a different location for performance or archival reasons.

  • JSON data output enhancements
    This is a really great capability, possibly one of the killer features of your implementation.
    Right now, for repositories that contain no results, you create a JSON file with a null value.
    Ideally this should either not be created, or it should contain an empty array. This would ease parsing capabilities for those using your tool to generate large batches of results then post process them using something like Ruby (Which is what I am doing). My general feeling is that no report should be created if there are no matches, however, I think consistency is key however.

  • --quiet mode / --logfile
    A quiet mode or logfile capability would help with automating your tool further, alongside more detailed exit codes as detailed below.

  • Return code definitions / Extended return code capability ?
    Right now, from what I understand, the tool returns 0 for successful execution, and non-zero for unsuccessful execution. It would be great to have a bit more detail exposed as exit codes for different circumstances, such as if a scan executed, and returned matches, etc.

This would help greatly with automating your tool further. However, I'd love to hear what you and other users think about this.
A good example of this implemented well is in Puppet's --detailed-exit-codes functionality.

  • Externalising the regex matches ?
    I'm not sure if this is something that will greatly affect performance, but it would be great to externalise the REGEX matches into a configuration file so that you don't have to rebuild the binary when updating rules. Putting them into a runtime configuration file (Maybe YAML) would really help usability. Being able to specify the datadir / workdir in this same file would also be great.

A very basic example could look like this:

---
config:
  workdir: /somewhere/else
regex:
  AWS:
    pattern: <REGEX>

Once again, thanks again for a great tool :)

K

How to find a file

How to find the file inside the repo from the log below?
OBS.

{
   "line": "\txptoname = `-----BEGIN PRIVATE KEY-----",
   "commit": "abc123",
   "offender": "-----BEGIN PRIVATE KEY-----",
   "reason": "PKCS8",
   "commitMsg": "tralala",
   "author": "that guy",
   "file": "test/xpto/file.go",
   "branch": "aaaa/heads/master",
   "repo": "xptorepo",
   "date": "2017-11-14T00:09:57-08:00"
}

Filter repositories of user/organisation

Hello,

Something I noticed is, if I fork a big project (like node or better, the linux kernel ) and then I scan myself, it will take a while !

From here I can see two areas that can get an enhancement:

  • Options to scan or not forked project
  • Options to filter repos by size ( number of commits, project size or number of files ?)
  • (Or scan repositories ordered by size, so at some point I could kill the process)

Scanning private git repositories

Would be very useful to be able to provide http credentials as an option to authenticate with private git repositories. Currently does not appear to support anything but public projects, unless I have misunderstood the options.

Consumes a lot of memory when running against an org

Interesting project. I was testing it against uber's github org and it effectively run out of memory twice, even when the --disk option was specified. I did not investigate the cause but it is likely to be a mem leak.

Stop words option to prevent false positives

If we come across a line: conn = client.s3(aws_key=settings.AWS_KEY there will be a regex match on aws_key= which will then check settings.AWS_KEY for entropy. If the entropy is past a certain value, gitleaks determines this is a valid leak which it is not. A simple way to reduce false positives is to add a stop word checker in checks.go.

There could be a default set of stop words, something like: settings, client, connection, setting, ... that are checked if a strict option is set: gitleaks -s <repo url>.

more filters and options

Here are some filters and options that would be nice to have:

  • ignore filetypes
  • commit depth
  • commit range (HASH1 - HASH2)
  • scan PR #
  • Redact offending string from report/output
  • include commit url in leaks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.