useflyent / fhttp Goto Github PK

The f stands for fly

Go 99.90% Dockerfile 0.04% Makefile 0.05%

fhttp's Introduction

fhttp

Features

Ordered Headers

The package allows for both pseudo header order and normal header order. Most of the code is from this this Pull Request.

Note on HTTP/1.1 header order Although the header key is capitalized, the header order slice must be in lowercase.

	req.Header = http.Header{
		"X-NewRelic-ID":         {"12345"},
		"x-api-key":             {"ABCDE12345"},
		"MESH-Commerce-Channel": {"android-app-phone"},
		"mesh-version":          {"cart=4"},
		"X-Request-Auth":        {"hawkHeader"},
		"X-acf-sensor-data":     {"3456"},
		"Content-Type":          {"application/json; charset=UTF-8"},
		"Accept":                {"application/json"},
		"Transfer-Encoding":     {"chunked"},
		"Host":                  {"example.com"},
		"Connection":            {"Keep-Alive"},
		"Accept-Encoding":       {"gzip"},
		HeaderOrderKey: {
			"x-newrelic-id",
			"x-api-key",
			"mesh-commerce-channel",
			"mesh-version",
			"user-agent",
			"x-request-auth",
			"x-acf-sensor-data",
			"transfer-encoding",
			"content-type",
			"accept",
			"host",
			"connection",
			"accept-encoding",
		},
		PHeaderOrderKey: {
			":method",
			":path",
			":authority",
			":scheme",
		},
	}

Connection settings

fhhtp has Chrome-like connection settings, as shown below:

SETTINGS_HEADER_TABLE_SIZE = 65536 (2^16)
SETTINGS_ENABLE_PUSH = 1
SETTINGS_MAX_CONCURRENT_STREAMS = 1000
SETTINGS_INITIAL_WINDOW_SIZE = 6291456
SETTINGS_MAX_FRAME_SIZE = 16384 (2^14)
SETTINGS_MAX_HEADER_LIST_SIZE = 262144 (2^18)

The default net/http settings, on the other hand, are the following:

SETTINGS_HEADER_TABLE_SIZE = 4096
SETTINGS_ENABLE_PUSH = 0
SETTINGS_MAX_CONCURRENT_STREAMS = unlimited
SETTINGS_INITIAL_WINDOW_SIZE = 4194304
SETTINGS_MAX_FRAME_SIZE = 16384
SETTINGS_MAX_HEADER_LIST_SIZE = 10485760

The ENABLE_PUSH implementation was merged from this Pull Request.

gzip, deflate, and br encoding

gzip, deflate, and br encoding are all supported by the package.

Pseudo header order

fhttp supports pseudo header order for http2, helping mitigate fingerprinting. You can read more about how it works here.

Backward compatible with net/http

Although this library is an extension of net/http, it is also meant to be backward compatible. Replacing

import (
   "net/http"
)

with

import (
    http "github.com/useflyent/fhttp"
)

SHOULD not break anything.

Versatile Content-Length and Transfer-Encoding headers

fhttp user to set custom Content-Length and Transfer-Encoding headers of all types.

To set an empty Content-Length header

req.Header = http.Header{
	"Content-Length": {http.ContentLengthEmpty},
}

To ignore setting the Content-Length header

req.Header = http.Header{
    "Content-Length": {http.ContentLengthDelete},
}

Any Content-Length or Transfer-Encoding headers set will be prioritized and fhttp will not set proper Content-length or Transfer-Encoding headers

Credits

Special thanks to the following people for helping me with this project.

cc for guiding me when I first started this project and inspiring me with cclient
umasi for being good rubber ducky and giving me tips for http2 headers

fhttp's People

Contributors

Stargazers

Watchers

fhttp's Issues

Which version of golang is the modification based on?

Hello,
Thank you for good package.

May I ask which version of golang is the modification based on?

check for codes before writing header

in server.go, check if code < 100 or > 999, before checkWriteHeaderCode(code)

there are times it passes code 0, which will hangup socket on client end, and crash server end

License

Hello there @zMrKrabz, I'm interested on using and modifying fhttp, especially from the fork that @Danny-Dasilva has(https://github.com/Danny-Dasilva/fhttp), but it doesn't have a LICENSE and it gets dice, so if you might add a general license there with your wishes for the use of this project, then I would talk to Danny to pull it there[or PR the update myself], then play with it further.

Request timeout not working

Hello,
Thank you for good package.

I want to use utls with proxy and tried this package with context and http.client timeout. But it's not working for me.

requestMaxTimeout := 10 //seconds
ctxTimeout, cancelRequest := context.WithTimeout(context.Background(), time.Second*time.Duration(requestMaxTimeout))
defer cancelRequest()
httpClient = http.Client{
			Timeout: time.Second*time.Duration(requestMaxTimeout),
			Transport: &http.Transport{Proxy: http.ProxyURL(proxyDetails.ProxyURL),
				TLSClientConfig: &tls.Config{InsecureSkipVerify: true}},
			CheckRedirect: func(req *http.Request, via []*http.Request) error {
				return http.ErrUseLastResponse
			},

response, err := httpClient.Do(request.WithContext(ctxTimeout))

Request hangs with deflate-encoded data

Example:

package main

import (
	"fmt"
	fhttp "github.com/useflyent/fhttp"
	"io/ioutil"
	"log"
	"net/http"
	"time"
)

func main() {
		t := time.Second * 30
		// net/http
		client := &http.Client{Timeout: t}
		req, err := http.NewRequest("GET", "https://httpbin.org/deflate", nil)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println("Initiating request: net/http")
		resp, err := client.Do(req)
		if err != nil {
			log.Fatal(err)
		}
		defer resp.Body.Close()
		bodyText, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%s\n", bodyText)

		// fhttp
		fclient := &fhttp.Client{Timeout: t}
		freq, err := fhttp.NewRequest("GET", "https://httpbin.org/deflate", nil)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println("Initiating request: fhttp")
		fresp, err := fclient.Do(freq)
		if err != nil {
			log.Fatal(err)
		}
		defer fresp.Body.Close()
		text, err := ioutil.ReadAll(fresp.Body)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%s\n", text)
	}

This outputs:

Initiating request: net/http
gJ�J�L�����w^��F��M1&�Ć�o���w��=����F��X%h�hQR^��}��MD�
Initiating request: fhttp
reset err CANCEL StreamID: 1
2021/08/06 18:37:35 Get "https://httpbin.org/deflate": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Go version:
go version go1.16.3 windows/amd64

It appears to work with net/http but not with fhttp

Possible detection via http2 settings frame order

I overlooked the fact that the http2 settings will technically be ordered with InitialWindowSize and HeaderTableSize always being the last two header frames sent. No site is detecting via http2 setting order at the moment, so it is not problem, but the fix is also pretty easy. In order to fix this error, there needs to be an underlying API change detailed below. What are everyone's opinions on this?

Code that creates error:

	setMaxHeader := false
	if t.Settings != nil {
		for _, setting := range t.Settings {
			if setting.ID == SettingMaxHeaderListSize {
				setMaxHeader = true
			}
			if setting.ID == SettingHeaderTableSize || setting.ID == SettingInitialWindowSize {
				return nil, errSettingsIncludeIllegalSettings
			}
			initialSettings = append(initialSettings, setting)
		}
	}
	if t.InitialWindowSize != 0 {
		initialSettings = append(initialSettings, Setting{ID: SettingInitialWindowSize, Val: t.InitialWindowSize})
	} else {
		initialSettings = append(initialSettings, Setting{ID: SettingInitialWindowSize, Val: transportDefaultStreamFlow})
	}
	if t.HeaderTableSize != 0 {
		initialSettings = append(initialSettings, Setting{ID: SettingHeaderTableSize, Val: t.HeaderTableSize})
	} else {
		initialSettings = append(initialSettings, Setting{ID: SettingHeaderTableSize, Val: initialHeaderTableSize})
	}
	if max := t.maxHeaderListSize(); max != 0 && !setMaxHeader {
		initialSettings = append(initialSettings, Setting{ID: SettingMaxHeaderListSize, Val: max})
	}

Proposed Fix:

Have the http2 Transport Settings field changed from

	// Settings should not include InitialWindowSize or HeaderTableSize, set that in Transport
	Settings          []Setting
	InitialWindowSize uint32 // if nil, will use global initialWindowSize
	HeaderTableSize   uint32 // if nil, will use global initialHeaderTableSize

    Settings map[SettingID]uint32

This would require everyone using the library to change their code, but since this isn't a pressing issue that seems like unnecessary work.

Not backwards compatible with net/http?

I'm working on adapting another project that makes use of this library to emulate the TLS fingerprint of chrome. In particular, I'm trying to establish a WebSocket connection, which the library doesn't support currently. I'm running into an issue (seemingly) stemming from this library- during the course of establishing my WS connection, I need to call functions that expect a value of type net/http. When I try to use fhttp in its place, the code doesn't seem to type check.

As an example, here's some code that produces an issue:

connBackend, resp, err := dialer.Dial(pageURL, req.Header)

I encounter a similar problem anywhere that a vanilla net/http property is expected, but an fhttp property is supplied.

What I have tried:

Casting the req.Header as a net/http.Header - this works for this case, but later on when I need to pass along the ResponseWriter or something more complex, it doesn't:

connPub, err := upgrader.Upgrade(w, req, upgradeHeader)

^ Here I can convert upgradeHeader to the desired type but not the first two. I feel like this isn't the right approach- fhttp should be a drop in, no? Is there any workaround?