eddieivan01/nic

🌀 Nic is a HTTP request client with elegant and easy-to-use API

Nic

English | 中文

Nic is a HTTP request client which has elegant, easy-to-use API

---

Features

• wrapper of HTTP std lib, provids elegant and easy-to-use API

• keep session via nic.Session structure, nic.Session is go-routine safe

---

Installation

To install nic, enter the following command

$ go get -v -u github.com/eddieivan01/nic

---

Quick start

Do a HTTP request like this

resp, err := nic.Get("http://example.com", nil)
if err != nil {
    log.Fatal(err.Error())
}
fmt.Println(resp.Text)

---

Documentation

do a basic request

nic could do these methods' request

"HEAD", "GET", "POST", "DELETE", "OPTIONS", "PUT", "PATCH"

import (
	"fmt"
    "github.com/eddieivan01/nic"
)

func main() {
    url := "http://example.com"
    resp, err := nic.Get(url, nil)
    if err != nil {
        log.Fatal(err.Error())
    }
    fmt.Println(resp.Text)
}

post request with some form data

as you see, all requests' parameters are passed by nic.H, and the inner is saved in nic.KV, it's actually map[string]interface{}

resp, err := nic.Post(url, nic.H{
    Data : nic.KV{
        "nic" : "nic",
    },
    Headers : nic.KV{
        "X-Forwarded-For" : "127.0.0.1",
    },
})

request with cookies

of course, you can also set it in Headers

resp, err := nic.Get(url, nic.H{
    Cookies : nic.KV{
        "cookie" : "nic",
    },
})

request with files

you can upload files with files' name + files' content which is []byte type, and can also upload via local file path

while uploading a file, you can set multipart form's field name, filename and MIME type

for more convenient setting files parameters, you can invoke in a chain to set filename and MIME type

resp, err := nic.Post(url, nic.H{
    Files : nic.KV{
        "file1": nic.File(
                    "nic.go", 
                    []byte("package nic")),
        "file2": nic.FileFromPath("./nic.go").
                    MIME("text/plain").
                    FName("nic"),
    },
})

request with JSON

resp, err := nic.Post(url, nic.H{
    JSON : nic.KV{
        "nic" : "nic",
    }
})

request with unencoded raw message

resp, err := nic.Post(url, nic.H{
    Raw : "post body which is unencoded",
})

using chunked transfer

The default is not to use chunked transfer

enable the transfer-encoding: chunked

resp, _ := nic.Get(url, nic.H{
    Chunked: true,
})

set query params

resp, err := nic.Get(url, nic.H {
    Params: nic.KV {
        "a": "1",
    },
})

all the parameters you could set

H struct {
    Params  KV
    Data    KV
    Raw     string
    Headers KV
    Cookies KV
    Auth    KV
    Proxy   string
    JSON    KV
    Files   KV

    AllowRedirect      bool
    Timeout            int64
    Chunked            bool
    DisableKeepAlives  bool
    DisableCompression bool
    SkipVerifyTLS      bool
}

NOTICE

nic.H can only have one of the following four parameters

H.Raw, H.Data, H.Files, H.JSON

request with session, which could handle server's set-cookie header

session := nic.NewSession()
resp, err := session.Post("http://example.com/login", nic.H{
    Data : nic.KV{
        "uname" : "nic",
        "passwd" : "nic",
    },
})

// ......

resp, err = session.Get("http://example.com/userinfo", nil)

handle response

resp, _ := nic.Get(url, nil)
fmt.Println(resp.Text)
fmt.Println(resp.Bytes)

handle JSON response

resp, _ := nic.Get(url, nil)

type S struct {
    P1 string `json:"p1"`
    P2 string `json:"p2"`
}

s := &S{}
err := resp.JSON(&s)

if err == nil {
    fmt.Println(s.P1, s.P2)
}

change response's encoding

SetEncode will convert resp.Bytes to resp.Text if encoding is changed every time be called

resp, _ := nic.Get(url, nil)
err := resp.SetEncode("gbk")

if err == nil {
    fmt.Println(resp.Text)
}

save response's content as a file

resp, _ := nic.Get("http://example.com/1.jpg", nil)
err := resp.SaveFile("1.jpg")

---

register a request/response hook

session := nic.NewSession()
session.RegisterBeforeReqHook(func(r *http.Request) error {
    r.URL.RawQuery = "nic=nic"
    return nil
})
session.RegisterAfterRespHook(func(r *http.Response) error {
    r.Header.Set("nic", "nic")
    return nil
})

session.Get(url, nil)

---

QA

• Q:

  How to get origin *http.Request from nic.Session?

  A:

  by nic.Session.GetRequest method

• Q:

  How to pass origin *http.Response to goquery-like DOM-parsing-libs from nic.Response?

  A:

  use resp, _ := nic.Get(...); resp.Response to access origin anonymous structure *http.Response; and (*http.Response).Body's IO.Reader has been saved, you can use *http.Response as if it were the original structure

• Q:

  Redirection is allowed 10 times by default, how could I increase the number?

  A:

  by access nic.Session.Client then change its CheckRedirect property

• Q:

  How to use the chunked transfer-encoding?

  A:

  by nic.H{Chunked: true}

nic's Issues

请求增加状态码支持

handle `Set-Cookie` while redirecting

不能正常使用POST请求

code：

package main

import (
	"github.com/eddieivan01/nic"
	"log"
	"fmt"
)

func post_test(num string){
	url := "http://xxx.action?depth=2"
	session := &nic.Session{}
	resp, err := session.Post(url, nic.H{
		Headers: nic.KV{
			"Accept": "application/json, text/javascript, */*; q=0.01",
			"X-Requested-With": "XMLHttpRequest",
			"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
			"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
		},
		Data: nic.KV{
			"$containerNo_like": num,
			"page": "1",
			"rows": "30",
		},
		AllowRedirect: true,
	})
	if err != nil{
		log.Println(err.Error())
	}
	fmt.Println(resp.Text)
}

func main(){
	post_test("1234567")
}

结果一直没反应，已经排除服务器问题。

file上传缺少form相应设置

fiddler 抓包信息:

------WebKitFormBoundaryBUZFWBgRBSOaC51N
Content-Disposition: form-data; name="entrustNos"

20190038713
------WebKitFormBoundaryBUZFWBgRBSOaC51N
Content-Disposition: form-data; name="fileList[0].fileName"

request paper
------WebKitFormBoundaryBUZFWBgRBSOaC51N
Content-Disposition: form-data; name="fileList[0].fileRemark"


------WebKitFormBoundaryBUZFWBgRBSOaC51N
Content-Disposition: form-data; name="uploadArr"; filename="38713.pdf"
Content-Type: application/pdf

%PDF-1.3
%    
1 0 obj
<< 
/Creator (Canon SC1011)
......

用python requests可以这样写：

......
files = {
	'entrustNos': [None, '20190038713'],
	'fileList[0].fileName': [None, 'request paper'],
	'fileList[0].fileRemark': [None, None],
	'uploadArr': ['38713.pdf', open('38713.pdf', 'rb'), 'application/pdf']
}

response = rq.post('http://xxx.com', headers=headers, cookies=cookies, files = files)

翻看nic的源码没发现相应的处理方式。

大量Get请求 会报错connection reset by peer

handle duplicate headers correctly

如何传递slice类型data？

首先感谢作者写的这个库，这是我用过的最舒服的golang http库。
如下Python requests代码：

import requests

cookies = {
    'JSESSIONID': '...............',
    'nt_pl_cmp_ucode': '..........',
}

headers = {
    'Connection': 'keep-alive',
    'User-Agent': 'Mozilla/5.0',
	.......
}

data = [
   ...............
  ('dealPeopleCodes', '01'),
  ('dealPeopleCodes', '02'),
  ('dealPeopleCodes', '03'),
  ('personProCodes', '06'),
  ('personProCodes', '01'),
  ('personProCodes', '07'),
  ('personProCodes', '02'),
  ('personProCodes', '04'),
  ('personProCodes', '05'),
  .................
]

response = requests.post('https://www.xxx.com/action', headers=headers, cookies=cookies, data=data)

在nic中该如何传递类似Python list类型的数据？

能否增加连接超时设置和读取超时设置

请求增加url params支持