malfunkt / hyperfox Goto Github PK

View Code? Open in Web Editor NEW

1.6K 1.6K 183.0 2.08 MB

HTTP/HTTPS MITM proxy and recorder.

Home Page: https://hyperfox.org

License: MIT License

Go 89.37% Makefile 6.59% Shell 2.40% Dockerfile 1.64%

hyperfox's People

Contributors

Stargazers

Watchers

Forkers

funkygao gnomix zined cazacugmihai acidventure archs loveshell lunixbochs gopster85 pirvu djbdjb00djb nydw changliwei prodigeni rmoorman yisi23 cartertsai securityigi bradparks williamren zhuweijava cmdev8 neuroradiology masdude international zerodark max-liu sergiodl sxhao achlys4rev hubert-he andradeandrey sneakyz gcbinsp roger2000hk jinglingshu hotelzululima flyawaytang jamesliandroid r00tb0x nierox zofuthan no2key miaojoy winxxp octotopos jameslinus coldfire-x sigma-random ghubgenius drptbl znanl jackylhs56 bobbyzhu thisissoon securextools ver007 r00tak mdabbagh88 chinpeng houcy mrphishxxx mstat gvsurenderreddy jdwang001 xieqiwen1993 dipsec jackhuyh joel-duqiong ellipsys naylamp6 sector009 jinhan1414 reigster devopsmi ketutgunawan jqk6 olivierh59500 secsecsec laudarch doduytrung jannson t3377 gophersgang furusiyya xygwfxu itsme72002 rootsystem2010 jkloop45 vasujain00 anubhav722 jitendersrivastava stevenchen0x01 h0k5 tcz001 lhwhit jamesrealweb3 cubedro hstart akpotter

hyperfox's Issues

hi,may i ask a question about flow control (single client connect to server )

I setup a https proxy server ,and provide the proxy service for many people,and now i want to control the flow each person ..can hyperfox make it ? thanks!

undefined: tls.ClientHelloInfo

Hello,

When I launch "go get github.com/xiam/hyperfox" I obtain the following result:

github.com/xiam/hyperfox/proxy

/usr/lib/go/src/pkg/github.com/xiam/hyperfox/proxy/main.go:301: undefined: tls.ClientHelloInfo
/usr/lib/go/src/pkg/github.com/xiam/hyperfox/proxy/main.go:325: unknown tls.Config field 'GetCertificate' in struct literal

Have you an idea ?

I work on Debian in multi release mode and the go version is "go1.3.2 linux/amd64"

Thanks in advance ;)

Static port for captures

I want to run hyperfox inside a docker container. But I cannot configure docker to forward a port without knowing it beforehand and for that I would need to be able to configure the port to view the captures.

Problem with https proxy

Hello, hyperfox is a great project. I am having some problems with https proxy. I encountered an error when I ran this program.
http: TLS handshake error from 127.0.0.1:62777: tls: oversized record received with length 20037
Then I tried to upgrade my go version. Currently, the go version is 1.12.5.
When I run again, it becomes another error.
http: TLS : first record does not look like a TLS handshake
My system is win10 amd64.
Where am I configured wrong?
I will be very grateful for your answer.

can't start if i use -k -c

sudo /Users/xuxiang/Downloads/hyperfox_darwin_amd64 -c /Users/xuxiang/go/src/hyperfox/lib/gencert/certs/xx.com/cert.pem -k /Users/xuxiang/go/src/hyperfox/lib/gencert/certs/xx.com/key.pem master [7157638] untracked
2018/01/26 11:07:56 Hyperfox v1.9.8 // https://hyperfox.org
2018/01/26 11:07:56 By José Carlos Nieto.

2018/01/26 11:07:56 Initializing database hyperfox-00007.db...
2018/01/26 11:07:56 Starting (local) API server...
2018/01/26 11:07:56 Watch live capture at http://live.hyperfox.org/#/?source=127.0.0.1:65234

2018/01/26 11:07:56 Listening for incoming HTTPs client requests on 0.0.0.0:10443.
2018/01/26 11:07:56 Listening for incoming HTTP client requests on 0.0.0.0:1080.
2018/01/26 11:07:56 Failed to bind on the given interface (HTTP): %!(EXTRA *net.OpError=listen tcp 0.0.0.0:1080: bind: address already in use)

xuxiang@promote:/git/github/sqlmap (*)

sudo /Users/xuxiang/Downloads/hyperfox_darwin_amd64 -c "/Users/xuxiang/go/src/hyperfox/lib/gencert/certs/xx.com/cert.pem" -k "/Users/xuxiang/go/src/hyperfox/lib/gencert/certs/xx.com/key.pem" master [7157638] untracked
2018/01/26 11:08:39 Hyperfox v1.9.8 // https://hyperfox.org
2018/01/26 11:08:39 By José Carlos Nieto.

2018/01/26 11:08:39 Initializing database hyperfox-00008.db...
2018/01/26 11:08:39 Starting (local) API server...
2018/01/26 11:08:39 Watch live capture at http://live.hyperfox.org/#/?source=127.0.0.1:65419

2018/01/26 11:08:39 Listening for incoming HTTPs client requests on 0.0.0.0:10443.
panic: sync: unlock of unlocked mutex

goroutine 19 [running]:
sync.(*Mutex).Unlock(0xc420012570)
/usr/local/go/src/sync/mutex.go:184 +0xc1
sync.(*Once).Do(0xc420012570, 0xc420043c58)
/usr/local/go/src/sync/once.go:46 +0x77
net/http.(*Server).setupHTTP2_ServeTLS(0xc420012500, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2950 +0x4c
net/http.(*Server).ServeTLS(0xc420012500, 0x478df40, 0xc42000e1a0, 0xc42007c3d2, 0x40, 0xc420014151, 0x3f, 0x0, 0x1)
/usr/local/go/src/net/http/server.go:2744 +0x43
net/http.(*Server).ListenAndServeTLS(0xc420012500, 0xc42007c3d2, 0x40, 0xc420014151, 0x3f, 0x406f1a2, 0x0)
/usr/local/go/src/net/http/server.go:2943 +0xd1
github.com/malfunkt/hyperfox/lib/proxy.(*Proxy).StartTLS(0xc420012500, 0xc420017ab0, 0xd, 0x2, 0x2)
/Users/rev/go/src/github.com/malfunkt/hyperfox/lib/proxy/proxy.go:332 +0x31f
main.main.func3(0xc420017aa0, 0xc420012500)
/Users/rev/go/src/github.com/malfunkt/hyperfox/main.go:152 +0x12d
created by main.main
/Users/rev/go/src/github.com/malfunkt/hyperfox/main.go:150 +0x63c

Read overflow error by HTTPS server

I have started hyperfox with default settings and http work well but https prompt this error:
2017/06/19 17:06:43 http: TLS handshake error from 127.0.0.1:51308: tls: oversized record received with length 20037
Browser is showing connection refused from proxy. And no https request is getting passed through hyperfox. Please help me how I can make this work. Thanks!

Capture HTTP and HTTPS

How do you capture both HTTP and HTTPS on the same port? Seems there are two different ports in use.

`dsniff` link on https://hyperfox.org/capturing-https-traffic points to the sample CA certificate

The link to dsniff near the bottom of the "Capturing HTTPS Traffic" page points to the sample CA certificate on the GitHub repo rather than the dsniff website.

How to work it with shadowsocks?

i want to catch https flow in shadowsocks ， what should i do?

TLS oversized record error

Hi,

I'm trying to setup HTTPs capturing, but I'm receiving multiple errors like this when initializing HTTPs requests:

2015/03/08 12:48:15 http: TLS handshake error from 192.168.0.102:52759: tls: oversized record received with length 20037

I'm running the latest OS X and have followed the tutorials on https://hyperfox.org/capturing-https-traffic. Hyperfox version is 0.9. HTTP capturing works fine.

Any idea what might be wrong?

Cannot get latest version: module contains a go.mod file, so module path should be github.com/malfunkt/hyperfox/v2

Background

The github.com/malfunkt/hyperfox uses Go modules and the current release version is v2. And it’s module path is "github.com/malfunkt/hyperfox", instead of "github.com/malfunkt/hyperfox/v2". It must comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation. Quoting the specification:

A package that has opted in to modules must include the major version in the import path to import any v2+ modules
To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2.
https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Steps to Reproduce

GO111MODULE=on, run go get targeting any version >= v2.0.0 of the malfunkt/hyperfox:

$ go get github.com/malfunkt/[email protected]
go: finding github.com/malfunkt/hyperfox v2.0.0
go: finding github.com/malfunkt/hyperfox v2.0.0
go get github.com/malfunkt/[email protected]: github.com/malfunkt/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

run go get github.com/malfunkt/hyperfox, the version will stuck in v1.9.8:

$go get github.com/malfunkt/hyperfox
go: downloading github.com/malfunkt/hyperfox v1.9.8
go: github.com/malfunkt/hyperfox upgrade => v1.9.8

SO anyone using Go modules will not be able to easily use any newer version of malfunkt/hyperfox.

Solution

1. Kill the go.mod files, rolling back to GOPATH.

This would push them back to not being managed by Go modules (instead of incorrectly using Go modules).
Ensure compatibility for downstream module-aware projects and module-unaware projects projects

I see these dependencies in your go.mod file, which need modle awareness. So you'd better not use third-party tools(such as: Dep, glide, govendor…).

github.com/mdp/qrterminal/v3 v3.0.0

You also need to update the import path to:

import github.com/mdp/qrterminal/…

2. Fix module path to strictly follow SIV rules.

Patch the go.mod file to declare the module path as github.com/malfunkt/hyperfox/v2 as per the specs. And adjust all internal imports.
The downstream projects might be negatively affected in their building if they are module-unaware (Go versions older than 1.9.7 and 1.10.3; Or use third-party dependency management tools, such as: Dep, glide,govendor…).

[*] You can see who will be affected here: [1 module-unaware user, i.e., Labs22/BlackServerOS]
https://github.com/search?l=Go&p=2&q=malfunkt%2Fhyperfox&type=Code

If you don't want to break the above repos. This method can provides better backwards-compatibility.
Release a v2 or higher module through the major subdirectory strategy: Create a new v2 subdirectory (github.com/malfunkt/hyperfox/v2) and place a new go.mod file in that subdirectory. The module path must end with /v2. Copy or move the code into the v2 subdirectory. Update import statements within the module to also use /v2 (import "github.com/malfunkt/hyperfox/v2/…"). Tag the release with v2.x.y.

3. Suggest your downstream module users use hash instead of a version tag.

If the standard rule of go modules conflicts with your development mode. Or not intended to be used as a library and does not make any guarantees about the API. So you can’t comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation.
Regardless, since it's against one of the design choices of Go, it'll be a bit of a hack. Instead of go get github.com/malfunkt/hyperfox@version-tag, module users need to use this following way to get the malfunkt/hyperfox:
(1) Search for the tag you want (in browser)
(2) Get the commit hash for the tag you want
(3) Run go get github.com/malfunkt/hyperfox@commit-hash
(4) Edit the go.mod file to put a comment about which version you actually used
This will make it difficult for module users to get and upgrade malfunkt/hyperfox.

[*] You can see who will be affected here: [1 module user, e.g., malfunkt/hyperfox]
https://github.com/search?l=Go&p=2&q=malfunkt%2Fhyperfox&type=Code

Summary

You can make a choice to fix DM issues by balancing your own development schedules/mode against the affects on the downstream projects.

For this issue, Solution 2 can maximize your benefits and with minimal impacts to your downstream projects the ecosystem.

References

Getting TLS Failure

I am trying your "proxy.StartTLS" call and when I send a request from the browser, the console shows:

tls: first record does not look like a TLS handshake

Ideas on what that could be? Thank you.

panic serving 127.0.0.1:XXX: Unknown condition type uint

Full stack trace:

toby @ ~ () => hyperfox 
2016/12/10 14:41:23 Hyperfox v1.9.7 // https://hyperfox.org
2016/12/10 14:41:23 By José Carlos Nieto.

2016/12/10 14:41:23 Initializing database hyperfox-00001.db...
2016/12/10 14:41:23 Starting (local) API server...
2016/12/10 14:41:23 Watch live capture at http://live.hyperfox.org/#/?source=127.0.0.1:44673

2016/12/10 14:41:23 Listening for incoming HTTP client requests on 0.0.0.0:1080.
127.0.0.1:35848 - - [10/Dec/2016:14:41:57 +0000] "GET http://google.com/ HTTP/1.1" 302 261
127.0.0.1:35848 - - [10/Dec/2016:14:41:57 +0000] "GET http://www.google.co.uk/?gfe_rd=cr&ei=NRRMWIDQD5HS8AeAzraQAQ HTTP/1.1" 302 277
2016/12/10 14:42:16 http: panic serving 127.0.0.1:35802: Unknown condition type uint
goroutine 44 [running]:
net/http.(*conn).serve.func1(0xc420144500)
	/usr/local/go/src/net/http/server.go:1491 +0x12a
panic(0x8095e0, 0xc42020c820)
	/usr/local/go/src/runtime/panic.go:458 +0x243
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*templateWithUtils).ToWhereWithArguments(0xc420032080, 0x809760, 0xc42020c610, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8bfff8, 0x1, ...)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/convert.go:146 +0x11f1
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*templateWithUtils).ToWhereWithArguments(0xc420032080, 0x8027a0, 0xc4201fbae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x81ebe0, 0x0, ...)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/convert.go:88 +0x1640
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*selector).Where(0xc4200d5200, 0xc42020c600, 0x1, 0x1, 0xc694a0, 0xc4200d5200)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/select.go:77 +0x13a
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter.(*Result).buildSelect(0xc4200c6e10, 0x41ac4b, 0x8095e0)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter/result.go:212 +0x136
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter.(*Result).One(0xc4200c6e10, 0x7f6f80, 0xc42000b900, 0xe, 0xc68720)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter/result.go:135 +0x2f
main.getHandler(0xc651c0, 0xc4200dd790, 0xc4200c6d20)
	/app/src/github.com/malfunkt/hyperfox/service.go:205 +0x863
net/http.HandlerFunc.ServeHTTP(0x8c0310, 0xc651c0, 0xc4200dd790, 0xc4200c6d20)
	/usr/local/go/src/net/http/server.go:1726 +0x44
github.com/malfunkt/hyperfox/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc4200134f0, 0xc651c0, 0xc4200dd790, 0xc4200c6d20)
	/app/src/github.com/malfunkt/hyperfox/vendor/github.com/gorilla/mux/mux.go:107 +0x10d
net/http.serverHandler.ServeHTTP(0xc42001ef00, 0xc651c0, 0xc4200dd790, 0xc4200c6b40)
	/usr/local/go/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc420144500, 0xc65900, 0xc4201546c0)
	/usr/local/go/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2293 +0x44d
2016/12/10 14:42:16 http: panic serving 127.0.0.1:35806: Unknown condition type uint
goroutine 54 [running]:
net/http.(*conn).serve.func1(0xc42001f780)
	/usr/local/go/src/net/http/server.go:1491 +0x12a
panic(0x8095e0, 0xc4201da6c0)
	/usr/local/go/src/runtime/panic.go:458 +0x243
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*templateWithUtils).ToWhereWithArguments(0xc420032080, 0x809760, 0xc4201da4b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x10, 0x8095e0, ...)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/convert.go:146 +0x11f1
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*templateWithUtils).ToWhereWithArguments(0xc420032080, 0x8027a0, 0xc4201d6940, 0x0, 0x0, 0x0, 0x0, 0x0, 0x81ebe0, 0x0, ...)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/convert.go:88 +0x1640
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder.(*selector).Where(0xc42016e540, 0xc4201da4a0, 0x1, 0x1, 0xc694a0, 0xc42016e540)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/lib/sqlbuilder/select.go:77 +0x13a
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter.(*Result).buildSelect(0xc420189e00, 0x41ac4b, 0x8095e0)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter/result.go:212 +0x136
github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter.(*Result).One(0xc420189e00, 0x7f6f80, 0xc420143300, 0xe, 0xc68720)
	/app/src/github.com/malfunkt/hyperfox/vendor/upper.io/db.v2/internal/sqladapter/result.go:135 +0x2f
main.getHandler(0xc651c0, 0xc4201869c0, 0xc420189d10)
	/app/src/github.com/malfunkt/hyperfox/service.go:205 +0x863
net/http.HandlerFunc.ServeHTTP(0x8c0310, 0xc651c0, 0xc4201869c0, 0xc420189d10)
	/usr/local/go/src/net/http/server.go:1726 +0x44
github.com/malfunkt/hyperfox/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc4200134f0, 0xc651c0, 0xc4201869c0, 0xc420189d10)
	/app/src/github.com/malfunkt/hyperfox/vendor/github.com/gorilla/mux/mux.go:107 +0x10d
net/http.serverHandler.ServeHTTP(0xc42001ef00, 0xc651c0, 0xc4201869c0, 0xc4200c6f00)
	/usr/local/go/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc42001f780, 0xc65900, 0xc420017e00)
	/usr/local/go/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2293 +0x44d

Was triggered clicking the i icon next to a request.

snags

Some custom middlewares

Hi, Could Hyperfox provide some custom middlewares, such as dump the response manually to output to the db which could be customed by our developers

Choose an storage backend that does not depend on CGO

Currently Hyperfox saves data to a SQLite database.

This offers some benefits:

Data is not locked to Hyperfox.
Users can use a variety of tools to examine records.

But also carries some drawbacks:

Easy cross compilation is not possible.
Simple compilation is not that straightforward.

It would be nice to:

Cross-compile Hyperfox easily to provide binaries for different OSs with no effort.
Offer a readable database format (or exporting options), so the user is not locked to Hyperfox.

The database we choose:

Must be embedded so it does not require any additional servers or programs to run.
Must offer search and indexing capabilities.
Must work fine with big blobs of binary data.
Should accept date and time range queries.
Should provide command line or visual tools for searching database files.

Some options to consider are boltdb, goleveldb and ql. I don't have a lot of experience with boltdb and goleveldb so any informed opinions on the matter would be very much appreciated.

Database fails to load in the browser

After capture, I am trying to load the database in the browser by pointing to 0.0.0.0:1080.

Here's what I get

lost of lines like this

2017/09/08 16:01:06 Failed to save to database: unable to open database file
2017/09/08 16:01:06 Failed to save to database: unable to open database file

followed by lots of lines like

2017/09/08 16:01:07 http: Accept error: accept tcp [::]:1080: accept4: too many open files; retrying in 640ms
2017/09/08 16:01:07 http: Accept error: accept tcp [::]:1080: accept4: too many open files; retrying in 1s

I am running hyperfox on ubuntu linux

io.Copy "context canceled"

I have been looking at potentially using hyperfox as the base for writing a GitHub API mocking tool, but I cannot even get it to proxy. Here is my command line:

sudo bin/hyperfox -addr 127.0.0.1 -https 9999 \
   --ca-cert /Users/mikeschinkel/Projects/hyperfox/ca/rootCA.crt \
   --ca-key /Users/mikeschinkel/Projects/hyperfox/ca/rootCA.key

When I run a Go program that uses github.com/google/go-github to call Client.Repositories.GetContents() which in turn calls the API I get this output:

2021/02/18 16:44:19 io.Copy: "context canceled"
127.0.0.1:52515 - - [18/Feb/2021:16:44:19 -0500] "CONNECT https://api.github.com:443 HTTP/1.1" 400 161256```

I debugged through it for over an hour but still cannot figure out what I am doing wrong.

BTW, I implemented a proxy in Go based on this code and it worked fine — though I was not able to inspect the information returned which is why I looked at hyperfox — so I know that the GET request by github.com/google/go-github works.

hyperfox.org is offline

Just checked hyperfox.org few minutes ago and it’s down

Windows support.

I think Hyperfox would compile fine on a Windows machine but I can't test it since I don't have a license so some help would be appreciated.

Some concerns:

Hyperfox depends on mattn's SQLite driver which requires CGO.
We need instructions for building and testing from scratch on a VirtualBox's virtual machine and not a binary build.
Instructions for enabling IP forwarding are also needed.