projectdiscovery/rawhttp

Raw HTTP client in Go for complete request control and customization.

rawhttp

rawhttp is a Go package for making HTTP requests in a raw way.

• Forked and adapted from https://github.com/gorilla/http and https://github.com/valyala/fasthttp
• The original idea is inspired by @tomnomnom/rawhttp work

Library Usage

A simple example to get started with rawhttp is available at examples. For documentation, please refer godoc

Note

rawhttp internally uses fastdialer to dial connections and fastdialer has a disk cache for DNS lookups. While using rawhttp .Close() method should be called at end of the program to remove temporary files created by fastdialer.

License

rawhttp is distributed under MIT License

rawhttp's Issues

Wrong default port with HTTPS

[Maintenance] Resolve all linter errors

$ golangci-lint run
example/server/server.go:8:5: `i` is unused (deadcode)
var i int
    ^
client/status.go:64:5: `invalidStatus` is unused (deadcode)
var invalidStatus Status
    ^
clientpipeline/client.go:23:5: `errorChPool` is unused (deadcode)
var errorChPool sync.Pool
    ^
clientpipeline/http.go:15:2: `maxSmallFileSize` is unused (deadcode)
        maxSmallFileSize       = 2 * 4096
        ^
clientpipeline/http.go:18:7: `defaultMaxInMemoryFileSize` is unused (deadcode)
const defaultMaxInMemoryFileSize = 16 * 1024 * 1024
      ^
clientpipeline/http.go:24:5: `errRequestHostRequired` is unused (deadcode)
var errRequestHostRequired = errors.New("missing required Host header in request")
    ^
clientpipeline/http.go:26:6: `acquireBufioWriter` is unused (deadcode)
func acquireBufioWriter(w io.Writer) *bufio.Writer {
     ^
clientpipeline/http.go:36:6: `releaseBufioWriter` is unused (deadcode)
func releaseBufioWriter(bw *bufio.Writer) {
     ^
clientpipeline/http.go:43:7: `minCompressLen` is unused (deadcode)
const minCompressLen = 200
      ^
clientpipeline/http.go:69:6: `copyZeroAlloc` is unused (deadcode)
func copyZeroAlloc(w io.Writer, r io.Reader) (int64, error) {
     ^
util.go:49:6: `fromResponse` is unused (deadcode)
func fromResponse(resp *client.Response) (client.Version, client.Status, map[string][]string, io.Reader) {
     ^
example/server/server.go:20:21: Error return value of `http.ListenAndServe` is not checked (errcheck)
        http.ListenAndServe(":10000", nil)
                           ^
clientpipeline/response.go:225:14: Error return value of `io.ReadFull` is not checked (errcheck)
                io.ReadFull(r, resp.body)
                           ^
client.go:161:19: Error return value of `conn.SetDeadline` is not checked (errcheck)
                conn.SetDeadline(time.Now().Add(options.Timeout))
                                ^
clientpipeline/http.go:50:6: type `flushWriter` is unused (unused)
type flushWriter struct {
     ^
clientpipeline/http.go:45:6: type `writeFlusher` is unused (unused)
type writeFlusher interface {
     ^
clientpipeline/http.go:40:5: var `bufioWriterPool` is unused (unused)
var bufioWriterPool sync.Pool
    ^
clientpipeline/http.go:55:23: func `(*flushWriter).Write` is unused (unused)
func (w *flushWriter) Write(p []byte) (int, error) {
                      ^
clientpipeline/response.go:83:4: ineffectual assignment to err (ineffassign)
                        err = errors.New("invalid header")
                        ^
pipelineclient.go:91:3: ineffectual assignment to host (ineffassign)
                host += ":80"
                ^
pipelineclient.go:94:3: ineffectual assignment to host (ineffassign)
                host = headers["Host"][0]
                ^
conn.go:47:4: SA4006: this value of `c` is never used (staticcheck)
                        c[0], c = c[len(c)-1], c[:len(c)-1]
                        ^

advised to add a feature

in python,we can set host in headers to the value that different from the real host of http url,however,by my testrawhttp) may not support this feature,so i advise rawhttpto support this feature

Bug in tls handling allows servers to control readTimeout

rawhttp version:

v.0.1.0 (latest)

Current Behavior:

RawHTTP clients waits forever, if the server does not send any data and the protocol was https

In the current implementation the tlsConn does not respect the timeout settings once the connection is established.
The function tlsConn.Handshake() will wait until the server sends data or closes the connection.

This means a single tarpit server, which sends TCP-Alive packets but no data, is able to halt rawhttp clients.

Related Ticket for nuclei: projectdiscovery/nuclei#2432

Expected Behavior:

cancel the connection and apply the specified timeout for connection and reading

Steps To Reproduce:

See test

Anything else:

Related: #54

The following PR fixes a bug when requesting "https" targets.

Simple test

The following snippet creates a server which holds the connection open and a client with a timeout connecting to the server.
The server component can also be a netcat listener nc -v -l -p 1337.

2022/08/15 18:09:08 [c] request to https://127.0.0.1:1337
2022/08/15 18:09:08 [s] Starting srv on 127.0.0.1:1337
2022/08/15 18:09:08 [s] tarpitting 127.0.0.1:41670 for 1m30s
2022/08/15 18:10:39 [s] release 127.0.0.1:41670 from tarpit
2022/08/15 18:10:39 [c] finished
2022/08/15 18:10:39 took 90.087851282

package main

import(
	"log"
	"fmt"
	"time"
	"net"

	"github.com/projectdiscovery/rawhttp"
)

func main() {

	addr := "127.0.0.1:1337"
	sleep := 90 * time.Second
	timeout := 2 * time.Second 

	ln, err  := ListenAndServe(addr, sleep)
	if err != nil {
		log.Fatal(err)
	}
	defer ln.Close()

	// perform check 
	start := time.Now()
	Check(addr, timeout)
	took := time.Now().Sub(start)

	log.Println("took", took.Seconds())
}


func Check(addr string, timeout time.Duration)(err error){

	options := rawhttp.DefaultOptions
	options.Timeout = timeout
	client := rawhttp.NewClient(options)
	uri := fmt.Sprintf("https://%s",addr)

	log.Printf("[c] request to %s", uri)
	_, err = client.Get(uri)
	log.Printf("[c] finished")
	return 
}

// --[Server Components]--
func ListenAndServe(addr string, sleep time.Duration)(listener net.Listener, err error){
	listener, err  = net.Listen("tcp", addr)
	if err != nil {
		return 
	}
	go StartServer(listener, sleep)
	return 
}
func StartServer(ln net.Listener, sleep time.Duration){
	log.Println("[s] Starting srv on", ln.Addr())
	for {
		conn, err := ln.Accept()
		if err != nil {
			continue
		}
		go handleClientConn(conn, sleep)
	}
}
// tarpit function
func handleClientConn(conn net.Conn, sleepDur time.Duration) {
	defer conn.Close()
	log.Printf("[s] tarpitting %s for %s\n",conn.RemoteAddr(), sleepDur.String())

	time.Sleep(sleepDur)
	log.Printf("[s] release %s from tarpit\n", conn.RemoteAddr())
}

Adding support for no RFC-compliant HTTP status codes

Please describe your feature request:

As the library's purpose is to fill the gap between standard RFC-compliant HTTP and unconventional one, supporting non-standard http status codes would be helpful. For the time being, any numeric value with an arbitrary number of digits should be considered valid.

Describe the use case of this feature:

Covering edge cases like:

• projectdiscovery/nuclei#2626

Requirements:

• The function ReadStatusCode in client/reader.go should accept arbitrary numbers of any size, including those not listed in rfc7231
• Add test cases both for positive/negative scenarios
• Update the comments/docs to include info about the function accepting non-rfc status codes

add ztls fallback support

Please describe your feature request:

• with addition of ztls fallback support in utils projectdiscovery/retryablehttp-go#100 the same should be done in rawhttp

Describe the use case of this feature:

use `urlutils` for url parsing

rawhttp version:

master

Current Behavior:

• currently rawhttp uses url.Parse() for parsing urls and it has encoding and other issues

Expected Behavior:

• use urlutils.ParseURL() in place of url.Parse() which handles all url related parsing edgecases

dial tcp "no such host" error on a specific case

Hello,

I'm currently facing a weird issue with one specific case when using the lib to send "malformed" HTTP request. My idea is to send a request to a specific HTTP(S) server, but with a different "Host" header in the HTTP request, to test for Host header attacks. Most of the time, this works well, but I sometimes receive this issue when running the code.

go run test.go

Target server: https://foo.com
Hostname: bar.com
Error while requesting https://foo.com with bar.com
2021/10/15 11:03:52 Error: dial tcp: lookup bar.com on 192.168.1.1:53: no such host
exit status 1

In this case, the hostname I want to use is not known by the DNS, but this should not be a problem. And it woks perfectly in other cases. I attached the code I use as txt (it's crappy, but it's the first time I write something in Go :)).

test.go.txt

HTTP/3 Support

add 00 support?

Please describe your feature request:

i wish rawhttp can send 0x00 data to server

Describe the use case of this feature:

as we know ,burp support send 00(in hex format) data to server ，and server can read it normally. but when i send a body with 00,the server return a 500 error

Add support for http proxy

Please describe your feature request:

It would be helpful to add support for HTTP(s) proxy.

Describe the use case of this feature:

Proxies or reverse proxies exploitation as non-RFC compliant requests would be dropped.

The feasibility of the task should be investigated, as the unsafe nature of the requests makes it difficult to respect standard proxy communication protocols.

Add automatic release gh action

• Add action for automatic release (see https://github.com/projectdiscovery/roundrobin/blob/main/.github/workflows/release-tag.yml)
• Ensure new releases are created only if there are new commits (see https://github.com/projectdiscovery/team-backlogs/issues/228)

Allowing multiple host headers

As for now, 2nd host header gets ignored.

rawhttp 0.0.8 version tlshandshake error

as the title, there are sometimes will occurs a panic about tlshandshake

Add SNI support

Description

SNI support should be added for TLS connections

HTTP/2 Support

Lint issues

https://github.com/projectdiscovery/rawhttp/runs/5334366334?check_suite_focus=true

Maintenance Tasks

• Bump go to 1.18
• Execute go mod tidy
• Fix any linting errors
• Align main/master <=> dev
• Delete dev branch
• Update relevant GH actions to use master/main only
• Build various examples as part of GH build workflows

HTTPS errors out on some targets

HTTP Pipelining

Remove "push" from GH actions

The check is already covered by the on "pull_request" event

Update docs

• Update README.md with more info about the library
• Add a minimal usage example snippet

advised to support set tls finger

Please describe your feature request:

i hope can set a tls fingerprint with https://github.com/refraction-networking/utls

Describe the use case of this feature:

some sites may block my request if i'm not visited it by browsers

fix breaking changes in utils/url

rawhttp version:

main

Current Behavior:

• update param related breaking changes
• depends on projectdiscovery/utils#190

Expected Behavior:

• required for projectdiscovery/nuclei#3887

Steps To Reproduce:

Anything else:

302 redirect failed

rawhttp version:

0.1.11

Current Behavior:

POST To a url with some data and with a 302 response,and with a followredirect option

Expected Behavior:

follow redirect and get the response after redirect

Steps To Reproduce:

1. call 'rawhttp ' with
   response, err := rawhttpClient.DoRawWithOptions(request.Method, request.URL, request.Path,
   reqHeaderMap, strings.NewReader(request.Body), options, &reqChains)
2. See error at do method of client.go

Anything else:

i run it at windows

Raw Request Bug [304 Response = i/o timeout]

Sorry, but the example with nuclei is easier for me

Nuclei version:

[INF] Current Version: 2.8.9

Current Behavior:

When sending an unprocessed request with a 'If-Modified-sinCe' header, no response is received and the request goes into a long timeout.

Expected Behavior:

An raw request must receive a valid response.

Steps To Reproduce:

Template:

id: C
info:
  name: C
  author: C
  description: C
 
requests:
  - raw:
      - |+
        GET /service-worker.js HTTP/1.1
        Host: rutube.ru
        If-Modified-Since: Tue, 21 Feb 2023 14:51:35 GMT

    unsafe: true

Command:

./nuclei -debug -t temple.yaml -u https://rutube.ru

2. Notice that instead of response

HTTP/1.1 304 Not Modified
Connection: close
Cache-Control: max-age=0
Date: Sat, 25 Feb 2023 17:42:57 GMT
Last-Modified: Tue, 21 Feb 2023 14:51:35 GMT
Server: QRATOR
Set-Cookie: uuid=94823265-14f0-d067-34b0-180e;

we get exception could not read http body: read tcp 192.168.1.1:8080->198.248.233.148:443: i/o timeout

You can remove unsafe: true from the yaml file to see the correct response.
OS: Windows 10, Ubuntu 18.04

Adding method with options to default client (custom timeout support

advise to add a .string() function

now,rawhttp only has .body to get response body,however, this will occur "i/o timeout" problem in some cases such as in proxy environt,so i advise rawhttp to add a .string() to get response body directly

Memory leak

aeaed82
https://github.com/projectdiscovery/rawhttp/blame/main/proxy/http.go#L36
The commit above will cause a coroutine/memory leak. After calling the fastdialer library, the dialer is not closed, which will result in the coroutines related to the leveldb opened by it not being able to exit normally.
This issue will lead to memory leaks in nuclei when proxies are used.

when to merge http proxy and socks5 proxy from dev to master?

when to merge http proxy and socks5 proxy from dev to master? thank you

rawhttp 0

rawhttp version:

Current Behavior:

Expected Behavior:

Steps To Reproduce:

Anything else:

Remove all validations and automatic RFC formatting

advise to support decompress gzip response body

because some server will send compressed data back, so i very strongly to advise rawhttp to support decompress data automatically

Add support for fastdialer

Please describe your feature request:

It would be useful to add support for fastdialer within rawhttp (ref: projectdiscovery/nuclei#2283)

Implementation

Nuclei fails to obtain the source ip in case of unsafe requests as the rawhttp library use go native net dial. The implementation can proceed as follows:

• Add fastdialer as option:

type Options struct {
	....
	FastDialer *fastdialer.Dialer
	...
}

• In the dial function if options.FastDialer client is defined use it instead of net.DialTimeout and net.Dial. For example:

func clientDial(protocol, addr string, timeout time.Duration, options *Options) (net.Conn, error) {
	// http
	if protocol == "http" {
		if timeout > 0 {
			if options.FastDialer != nil {
				ctxTimeout, _ := context.WithTimeout(context.Background(), timeout)
				return options.FastDialer.Dial(ctxTimeout, "tcp", addr)
			} else {
				return net.DialTimeout("tcp", addr, timeout)
			}
		}
		if options.FastDialer != nil {
			return options.FastDialer.Dial(context.Background(), "tcp", addr)
		} else {
			return net.Dial("tcp", addr)
		}
	}
...

Note

FastDialer.Dial populates an internal key-value store with the dialed ip, so that when FastDialer.GetDialedIP is called the correct ip is retrieved (indirectly fixing projectdiscovery/nuclei#2283)