fwhezfwhez/tcpx

a cross-language tcp framwork in golang

A very convenient tcp framework in golang.

• Have fun
• Heartbeat
• Auth
• User pool
• Graceful
• Middleware
• Message
• Marshaller
• TLS

Start

go get github.com/fwhezfwhez/tcpx

Dependency

if you want to run program in this repo,you should prepare protoc,proto-gen-go environment. It's good to compile yourself from these repos,but there is already release versions referring to their doc. Make sure run protoc --version available.

protoc: https://github.com/golang/protobuf

proto-gen-go:https://github.com/golang/protobuf/tree/master/protoc-gen-go

Benchmark

https://github.com/fwhezfwhez/tcpx/blob/master/benchmark_test.go

cases	exec times	cost time per loop	cost mem per loop	cost object num per loop	url
OnMessage	2000000	643 ns/op	1368 B/op	5 allocs/op	click to location
Mux without middleware	2000000	761 ns/op	1368 B/op	5 allocs/op	click to location
Mux with middleware	2000000	768 ns/op	1368 B/op	5 allocs/op	click to location

Pack

Tcpx has its well-designed pack. To focus on detail, you can refer to: https://github.com/fwhezfwhez/tcpx/tree/master/examples/modules/pack-detail

[4]byte -- length             fixed_size,binary big endian encode
[4]byte -- messageID          fixed_size,binary big endian encode
[4]byte -- headerLength       fixed_size,binary big endian encode
[4]byte -- bodyLength         fixed_size,binary big endian encode
[]byte -- header              marshal by json
[]byte -- body                marshal by marshaller

According to this pack rule, tcpx has 2 well-designed routing ways and their pack structure:

messageID type pack

header:
{
    "Router-Type": "MESSAGE_ID"
}

urlPattern pack

header:
{
    "Router-Type": "URL_PATTERN"
    "Router-Pattern-Value": "/login/"
}

Chat

https://github.com/fwhezfwhez/tcpx/tree/master/examples/modules/chat

It examples a chat using tcpx.

Raw

https://github.com/fwhezfwhez/tcpx/tree/master/examples/modules/raw

It examples how to send stream without rule, nothing to do with messageID/urlPattern system. You can send all stream you want. Global middleware and anchor middleware are still working as the example said.

IM

Here is an example of IM system using tcpx.

https://github.com/q1n9-jair/tcpx-demo

Product practice

tcpx's Issues

需要新添加可自动检测指定时间内没发送心跳就下线

需要新添加可自动检测指定时间内没发送心跳就下线以及可获取到没发心跳的userid
能及时获取到 客户端断开的userid

[feature-asking] Can add read/write timeout?

For the reliability of TCP services, timeout options need to be set for read and write.

[enhancement-candidate] Graceful stop, graceFul restart practice bad

Graceful stop/restart just stops listener without considering whether there exists unconsuming request in server. It's not graceful as expected.

Now graceful stop/restart only make effect that tcp stops/restart will not break process but make no effect to tcp requests itselves.

[feature-asking] Can add connections,throughput,response delay benchmarks ?

tcp client的例子跑一下就死了

[0 0 0 31 0 0 0 6 0 0 0 2 0 0 0 17 123 125 34 109 121 32 110 97 109 101 32 105 115 32 116 99 112 120 34]
收到服务端消息块: {
    "message_id": 6,
    "header": {},
    "body": "my name is tcpx"
  }
服务端消息: my name is tcpx
panic: 2021-03-05 14:49:01 | /root/go/src/github.com/fwhezfwhez/tcpx/examples/sayHello/client/main.go:29 | EOF


goroutine 7 [running]:
main.main.func1(0x813a20, 0xc00000e080)
        /root/go/src/github.com/fwhezfwhez/tcpx/examples/sayHello/client/main.go:29 +0x42b
created by main.main
        /root/go/src/github.com/fwhezfwhez/tcpx/examples/sayHello/client/main.go:22 +0xbe
exit status 2

是bug还是我用的不对呢，我按照readme来的。

go get github.com/fwhezfwhez/tcpx go get github.com/fwhezfwhez/tcpx: no matching versions for query "upgrade"

hello
i try get get tcpx many times, always report error. But i try other repositories, like github.com/fwhezfwhez/model_convert, it is successful

connection pool high-availeble supporting

[usage-asking] How to graceful shutdown tcpx server ?

[Promotion] Provide url style routing way, help tcpx practice better!

Hello!
Tcpx now provide a better practice routing style for developer. In the past, routers are like:

srv.AddHandler(1, login)
srv.AddHandler(3, auth)

This kind of routing is based on messageID. It seems easy but require developers to manage all messageID themselves, it's really a tiring job.

But now ,tcpx provides better routing style, it will gradually replace the old messageID routing.(But will not abandon usage of messageID and still keep compatible of old stf). It works like:

srv.Any("/login/", login)
srv.Any("/auth/", auth)

Also, when server side wants to reply:

func login(c *tcpx.Context) {
    c.JSONURLPattern(map[string]interface{}{
        "message": "login success, welcome!"
    })
}

More usages of url-pattern routing, you may refer to

[feature-asking]How tcpx works in distribution system?

It's easy to build a single server spot using tcpx. But once user number too much, one server must expand to several or clouds.

How to use tcpx to build multi services to deal with increasing number of requests.

心跳具体要怎么配置？

官方例子之后还要设置什么？客户端发不发送心跳效果都一样，服务端并不会断开客户端的连接

有做分布式的计划吗？

请问有做分布式的计划吗？

[feature-asking] Can you normalize the name of the network package read interface function?

The following functions don't seem to clearly define what to do:
tcpx.FirstBlockOf()
tcpx.FirstBlockOfBytes()

I used this:
tcpx.UnpackToBlockFromReader()

picture repo

title

标题pic

有时候解析包出错

不知道为什么如果接受10次,有8次发过来的数据,包的长度不对,只有两次的包长是对的
for {
//buff := make([]byte , 1024)
//conn.Read(buff)
//fmt.Println(string(buff)) // 每次直接这里读是有东西

	buf, e := tcpx.FirstBlockOf(conn)   
	len,e := tcpx.PackJSON.LengthOf(buf)    //长度不对 ,比如说错误的包长是 6 
	fmt.Println(len)

}
服务端发送的数据结构
type Data struct {
Table string json:"table"
Sql string json:"sql"
Count string json:"count"
Args []interface{} json:"args"
Ids []string json:"ids"
}

发送代码
ctx.JSON(6 , data)

需要新增一个 可以获取客户端(非极端环境下)断开连接的userid

需要新增一个 可以获取客户端(非极端环境下)断开连接的userid，知道哪个用户id下线

需要新增一个 HeartBeatModeDetail 可重写的

需要新增一个 HeartBeatModeDetail 可重写的，因为我们公司做多台 需要互相各个用户 公共心跳池

Have a complete protocol specification for writing client programs in other languages.

Have some complete protocol specification for writing client programs in other languages.

/root/go/src/github.com/fwhezfwhez/tcpx/log.go:34: can't read 784 length content block from reader, but read 196

server ran for a while and stopped with the error:
/root/go/src/github.com/fwhezfwhez/tcpx/log.go:34: can't read 784 length content block from reader, but read 196

can't finger out what happened.

当收到错误报文时，会产生fatal error: runtime: out of memory致命错误

func LengthOf(stream []byte) (int32, error) {
	if len(stream) < 4 {
		return 0, errors.New(fmt.Sprintf("stream lenth should be bigger than 4"))
	}
	length := binary.BigEndian.Uint32(stream[0:4])
	return int32(length), nil
}

当收到任意报文时，没有对length上限进行判断，可能会开辟过大的内存。
测试代码

func main() {
	var tcpAddr *net.TCPAddr
	tcpAddr, _ = net.ResolveTCPAddr("tcp", "127.0.0.1:7170")

	conn, _ := net.DialTCP("tcp", nil, tcpAddr)
	b := []byte("time\n")
	conn.Write(b)

	time.Sleep(time.Second * 1)
}

测试机内存为2G，当收到错误的报文时，会产生fatal error: runtime: out of memory致命错误

[Promotion]Profile usage of anchor middleware

Hello, TCPX now provides well practice on anchor middleware, which now can get called without times limit.
In the old version, operation below is not allowed:

srv.Use("middle1", middle1)
srv.AddHandler()
srv.UnUse("middle1")

srv.Use("middle1")
...
srv.UnUse("middle1")

Because the same middleware cant't use twice.

Now it's promoted and you can use it well like:

srv.Use("middle1", middle1)
srv.AddHandler()
srv.UnUse("middle1")

srv.Use("middle1")
srv.AddHandler()
srv.AddHandler()
srv.UnUse("middle1")

This promotion is linked below

[Promotion] Enable tls

Hello, tcpx now support tls for your server.Know that:

• If you want to suport tls using ISP commercial solution, then you don't actually need this functional promotion.
• If you got correct pem pair of key.pem and cert.pem, or you want a self-built,unauthorized certification and generate key.pem and cert.pem by command openssl genrsa > key.pem openssl req -new -x509 -key key.pem > cert.pem, then this promotion is just for it

Example:

package main

import (
	"fmt"
	"github.com/fwhezfwhez/tcpx"
)

func main() {
	r := tcpx.NewTcpX(nil)
	r.Any("/login/", func(c *tcpx.Context) {
		fmt.Printf("recv a login")
	})

	// TODO You might modify here to locate your pem files' real path
	var certPath = "G:\\go_workspace\\GOPATH\\src\\tcpx\\examples\\modules\\tls\\pem\\cert.pem"
	var keyPath = "G:\\go_workspace\\GOPATH\\src\\tcpx\\examples\\modules\\tls\\pem\\key.pem"

	//var pemPath = "G:\\go_workspace\\GOPATH\\src\\github.com\\fwhezfwhez\\tcpx\\examples\\modules\\tls\\pem"
	if e := r.LoadTLSFile(certPath, keyPath); e != nil {
		panic(e)
	}

	if e := r.ListenAndServe("tcp", ":8080"); e != nil {
		panic(e)
	}
}

Client example:

package main

import (
	"crypto/tls"
	"crypto/x509"
	"github.com/fwhezfwhez/tcpx"
)

func main() {
	// Connecting with a custom root-certificate set.
	// All tls shares a same rootPEM
	const rootPEM = `
-----BEGIN CERTIFICATE-----
MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTUwNDA0MTUxNTU1WjBJMQswCQYDVQQG
EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
ahqyzFPdFUuLH8gZYR/Nnag+YyuENWllhMgZxUYi+FOVvuOAShDGKuy6lyARxzmZ
EASg8GF6lSWMTlJ14rbtCMoU/M4iarNOz0YDl5cDfsCx3nuvRTPPuj5xt970JSXC
DTWJnZ37DhF5iR43xa+OcmkCAwEAAaOB+zCB+DAfBgNVHSMEGDAWgBTAephojYn7
qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUSt0GFhu89mi1dvWBtrtiGrpagS8wEgYD
VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwOgYDVR0fBDMwMTAvoC2g
K4YpaHR0cDovL2NybC5nZW90cnVzdC5jb20vY3Jscy9ndGdsb2JhbC5jcmwwPQYI
KwYBBQUHAQEEMTAvMC0GCCsGAQUFBzABhiFodHRwOi8vZ3RnbG9iYWwtb2NzcC5n
ZW90cnVzdC5jb20wFwYDVR0gBBAwDjAMBgorBgEEAdZ5AgUBMA0GCSqGSIb3DQEB
BQUAA4IBAQA21waAESetKhSbOHezI6B1WLuxfoNCunLaHtiONgaX4PCVOzf9G0JY
/iLIa704XtE7JW4S615ndkZAkNoUyHgN7ZVm2o6Gb4ChulYylYbc3GrKBIxbf/a/
zG+FA1jDaFETzf3I93k9mTXwVqO94FntT0QJo544evZG0R0SnU++0ED8Vf4GXjza
HFa9llF7b1cq26KqltyMdMKVvvBulRP/F/A8rLIQjcxz++iPAsbw+zOzlTvjwsto
WHPbqCRiOwY1nQ2pM714A5AuTHhdUDqB1O6gyHA43LL5Z/qHQF1hwFGPa4NrzQU6
yuGnBXj8ytqU0CwIPX4WecigUCAkVDNx
-----END CERTIFICATE-----`

	// First, create the set of root certificates. For this example we only
	// have one. It's also possible to omit this in order to use the
	// default root set of the current operating system.
	roots := x509.NewCertPool()
	ok := roots.AppendCertsFromPEM([]byte(rootPEM))
	if !ok {
		panic("failed to parse root certificate")
	}
	conn, err := tls.Dial("tcp", "localhost:8080", &tls.Config{
		RootCAs:            roots,
		InsecureSkipVerify: true, // If you make your server side certification yourself, you should set this field true to skip verify.
	})
	if err != nil {
		panic("failed to connect: " + err.Error())
	}
	defer conn.Close()
	msg := tcpx.NewURLPatternMessage("/login/", nil)
	buf, e := msg.Pack(nil)

	if e != nil {
		panic(e)
	}
	conn.Write(buf)
}

Feature required!

Hello!
Tcpx is all the time developing.If you want any special feature, you can suggest here.

Formating:
[feature-asking] Can ... ?

[usage-aking] What/Where/When/How ... ?

[description] ...

[feature-asking] Can user from tcp protocol send message to user from kcp/udp?

[description] If two users are login via tcp,udp port, I want to send messages to each other.

[usage-asking] How to design middleware?

[description] I want to use middleware but don't know how to operate.

Specifies whether TCP can be established by proxy

I'm having a problem and I haven't found a way to link proxies using tcpx like socks5

会自动触发掉线

客户端(安卓机器 小米 华为 vovo 等)使用 srv.OnClose = func(c *tcpx.Context) 会自动触发掉线 (心跳正常发送的时候)