nyancrimew/goop

Yet another tool to dump a git repository from a website, focused on as-complete-as-possible dumps and handling weird edge-cases.

goop's Issues

skip png, html .... files

Is there a way to skip all non essential files ?

[Crash] interface conversion

Hi !

I'm got a crash when i'm running the tool.
I'm not familiar with interfaces in golang and I cant debug it.

Can you take a look please

panic: interface conversion: jobtracker.Context is *workers.RecursiveDownloadContext, not workers.RecursiveDownloadContext

goroutine 25 [running]:
github.com/deletescape/goop/internal/workers.RecursiveDownloadWorker(0xc0000c81c0, 0xc0000b41b0, 0xa, 0x14e3700, 0xc0000af5f0)
/Users/segfault42/Documents/goop/internal/workers/recursivedownload.go:22 +0xf49
github.com/deletescape/jobtracker.workRoutine(0xc0000c81c0, 0x15c5470, 0x14e3700, 0xc0000af5f0)
/Users/segfault42/go/pkg/mod/github.com/deletescape/[email protected]/jobtracker.go:132 +0x92
created by github.com/deletescape/jobtracker.(*JobTracker).StartAndWait
/Users/segfault42/go/pkg/mod/github.com/deletescape/[email protected]/jobtracker.go:74 +0x85

request add tor

skip comments in .gitignore download

No command goop found,

Cant run the program i get this error after running goop
No command goop found,

Look into go2git

We could probably further boost speed and reliabilty by switching to a libgit2 binding like go2git, especially since go-git obviously struggles with some things the original git implementation doesn't. This would also allow us to get rid of all our calls out to the git commandline.

can't install on debian 10 ( buster ) docker

hello
i'm not a go expert, please be warn

i just run a brand new debian 10 via docker:

$ apt update; apt install golang
$ go get -u github.com/deletescape/goop

the directory is full of things: /root/go/src/github.com/deletescape/goop
but how are we supposed to lunch the software ?
"goop" is not reconized as a command

so i tried:
$ go run /root/go/src/github.com/deletescape/goop/main.go http://test.com
same with
$ go run /root/go/src/github.com/deletescape/goop/cmd/goop.go http://test.com

it fails with:
/root/go/src/github.com/go-git/go-git/storage/filesystem/dotgit/dotgit.go:17:2: code in directory /root/go/src/github.com/go-git/go-billy/osfs expects import "github.com/go-git/go-billy/v5/osfs"

thank you

Use git porcelain

in all places where we call out to the git cli we should be using --porcelain to parse data from a stable format.

Create objects using go-git

Instead of calling out to the git command twice to hash the missing files and then check them out, do the same thing natively using go-git.

I am not sure if this is possible, but it probably should be.

Add header to http requests?

Is it possible to provide a header value pair that is used for every http request sent to the server? If not could you please tell me where to edit the code to add such an option, I'm not really well versed with go :D

Fix the issue where everything locks up

Probably related to the absolutly horrendous way i am doing concurrency. Should probably be replaced with some channels feeding workers with jobs, instead of whatever this is.

go get does nothing

No idea what's wrong here, but go get -u github.com/deletescape/goop hangs... Other projects like https://github.com/deletescape/sloot work just fine.

Oh lol, while writing this, I got an output, but still no binary in go/bin:

go get -u github.com/deletescape/goop

code in directory /home/c0d3/go/src/github.com/go-git/go-billy/osfs expects import "github.com/go-git/go-billy/v5/osfs"

Disclaimer: non-go guy here ;)

Guess pack file hashes without .git/objects/info/packs

find out if its possible to guess the pack file hashes without .git/objects/info/packs, which is for example the case with non-pushed repositories.

Use git hash-object

After downloading missing files via HTTP, add them to git objects via git hash-object.

Please add binary release

Thank You!