soypat/seqs

seqs: the hottest, most idiomatic userspace TCP/IP implementation on the internet. lwip in go basically

seqs

seqs is what is commonly referred to as a userspace IP implementation. It handles:

• Ethernet protocol
• IP packet marshalling to sub-protocols:
  • ARP requests and responses
  • UDP packet handling
  • DHCP client requests and DHCP server
  • TCP connections over IP with support for multiple listeners on same port. These implement net.Conn and net.Listener interfaces. See stacks/tcpconn.go
  • HTTP: Algorithm to reuse heap memory between requests and avoid allocations. See httpx package
  • NTP client for resolving time offset to a NTP server

Example of use

// stack works by having access to Ethernet packet sending
// and processing. NIC is our physical link to the internet.
var NIC NetworkInterfaceCard = getNIC()

stack := stacks.NewPortStack(stacks.PortStackConfig{
    MAC:             MAC,
    MaxOpenPortsTCP: 1,
    MaxOpenPortsUDP: 1,
    MTU:             2048,
})
// stack.RecvEth should be called on receiving an ethernet packet. It should NOT block.
NIC.SetRecvEthHandle(stack.RecvEth)

// Static IP setting.
ip := netip.AddrFrom4([4]byte{192, 168, 1, 45}) 
stack.SetAddr(ip)

// Or can request an address via DHCP.
dhcpClient := stacks.NewDHCPClient(stack, dhcp.DefaultClientPort)
err = dhcpClient.BeginRequest(stacks.DHCPRequestConfig{
    RequestedAddr: netip.AddrFrom4([4]byte{192, 168, 1, 69}),
    Xid:           0x12345678,
    Hostname:      "tinygo-pico",
})
if err != nil {
    panic(err)
}

fmt.Println("Start DHCP...")
for !dhcpClient.Done() {
    doNICPoll(NIC)
    time.Sleep(time.Second / 10)
}

offeredIP := dhcpClient.Offer()
fmt.Println("got offer:", offeredIP)
stack.SetAddr(offeredIP)

How to use seqs

go mod download github.com/soypat/seqs@latest

History - Precursors to seqs

Before seqs there was:

• ether-swtch - First known instance of a (barely) working TCP/IP stack in Go working on embedded systems, circa June 2021. Could blink an Arduino UNO's LED via HTTP (!). Famously bad design, code, performance, readability.
• dgrams - Library prepared for Pico W's wifi chip. Already shows similarities with seqs. Circa May 2023.

seqs's Issues

FinWait1: ACK of previous data segment may trigger transition to FinWait2 (preventing direct transition to TimeWait by the following FIN|ACK)

When requesting a web page from the cyw43439 based http-server example (using wget, curl, or a browser), I am observing a wireshark log similar to the following excerpt:

No.	Time Source	Dest   Proto Len  Info
 4 0.097 client pico_w HTTP  180  GET / HTTP/1.1 
 5 0.201 pico_w client TCP    54  http(80) → 58856   [ACK] Seq=1 Ack=127 Win=1410 Len=0
 6 0.254 pico_w client HTTP 1244  HTTP/1.1 200 OK  (text/html)
 7 0.254 client pico_w TCP    54  58856 → http(80) ❶ [ACK] Seq=127 Ack=1191 Win=63070 Len=0
 8 0.254 pico_w client TCP    54  http(80) → 58856 ➋ [FIN, ACK] Seq=1191 Ack=127 Win=1536 Len=0
 9 0.258 client pico_w TCP    54  58856 → http(80) ➌ [FIN, ACK] Seq=127 Ack=1192 Win=63070 Len=0
10 0.307                      54  Ignore >
11 0.310 pico_w client TCP    54  http(80) → 58856   [ACK] Seq=1192 Ack=128 Win=1536 Len=0

After the server pico_w has transmitted the web page of 1244 bytes, the client sends an ACK segment (❶).
At nearly the same time the server already sends its FIN+ACK (➋), which the client acknowledges
(➌, the client uses the ACK number 1192, which is 1191+1, hence the ACK refers to the server's FIN+ACK).

In the server's log a slightly different order can be observed: The ACK (1) is received (resp. processed) after the FIN+ACK (2).

Got webpage request!
0.335 DEBUG TCP:delayed-close port=80
0.337 DEBUG TCP:send plen=12440.338 level=DBG-2 tcb:snd state=FinWait1 pend=0 snd.nxt=1192 snd.una=1 snd.wnd=64240
0.338 DBG-2 tcb:snd seg.seq=1191 seg.ack=1723085760 seg.wnd=2030 seg.flags=17 seg.data=0
0.340 INFO  TCP:tx-statechange port=80 old=Established new=FinWait1 txflags=[FIN,ACK] ➋
0.341 DEBUG TCP:send plen=540.395 level=DBG-2 Stack.RecvEth:start plen=54

0.397 DBG-2 tcb:rcv state=FinWait2 rcv.nxt=1723085760 rcv.wnd=2030 challenge=false
0.398 DBG-2 recv:seg seg.seq=1723085760 seg.ack=1191 seg.wnd=63070 seg.flags=16 seg.data=0
0.399 INFO  TCP:rx-statechange port=80 old=FinWait1 new=FinWait2 rxflags=[ACK] ❶

0.400 DBG-2 TCPConn.stateCheck:hasPending
0.401 DBG-2 tcb:snd state=FinWait2 pend=0 snd.nxt=1192 snd.una=1191 snd.wnd=63070
0.402 DBG-2 tcb:snd seg.seq=1192 seg.ack=1723085760 seg.wnd=2030 seg.flags=16 seg.data=0
0.403 DBG-2 TCPConn.send:start
0.404 DEBUG TCP:send plen=54

0.406 DBG-2 Stack.RecvEth:start plen=54
0.407 DEBUG TCP:recv opt=0 ipopt=0 payload=0
0.408 DBG-2 TCPConn.recv:start
0.409 DBG-2 tcb:rcv state=TimeWait rcv.nxt=1723085761 rcv.wnd=2030 challenge=false
0.410 DBG-2 recv:seg seg.seq=1723085760 seg.ack=1192 seg.wnd=63070 seg.flags=17 seg.data=0
0.410 INFO  TCP:rx-statechange port=80 old=FinWait2 new=TimeWait rxflags=[FIN,ACK] ➌

0.411 DBG-2 TCPConn.stateCheck:hasPending
0.412 DBG-2 tcb:snd state=TimeWait pend=0 snd.nxt=1192 snd.una=1192 snd.wnd=63070
0.413 DBG-2 tcb:snd seg.seq=1192 seg.ack=1723085761 seg.wnd=2030 seg.flags=16 seg.data=0
0.414 DBG-2 TCPConn.stateCheck:closed

As client and server send ❶ and ➋ independently, it can happen that ❶ is captured in the wireshark log before ➋, but on the server side ❶ is received and processed slightly after ➋.

Since the server is in FinWait1 state, when ❶ is received, ControlBlock.rcvFinWait1 (control.go:246) is called. Because the hasAck switch case applies, FinWait2 state is entered.

Wouldn't it be more correct if ❶ would be ignored by rcvFinWait1, since it does not acknowledge the FIN, but a previous date segment? The server could wait some more time until ➌ arrives, which contains the matching FIN and ACK.

The first case expression of the switch statement in ControlBlock.rcvFinWait1 already contains the
sub condition seg.ACK == tcb.snd.NXT. I think moving this condition to the initialization of hasAck would change the behaviour so that TimeWait is entered directly from FinWait1:

• orig: hasAck := flags&FlagACK != 0
• mod: hasAck := flags&FlagACK != 0 && seg.ACK == tcb.snd.NXT

It would make sure that only an ACK that actually acknowledges the FIN previously sent by the server is accepted. With this change, I am observing the following transition in Pico W's log:

0.904 INFO TCP:tx-statechange port=80 old=Established new=FinWait1 txflags=[FIN,ACK]
0.015 INFO TCP:rx-statechange port=80 old=FinWait1 new=TimeWait rxflags=[FIN,ACK]

As an example, I have attached a patch containing the change to rcvFinWait1, and a test case. finwait1_ignore_data_ack.patch.gz

Remove debug binary from git history

The binary occupies 4MB on the git repo, which is a real waste
529526c

unexpected stateClosing during HTTP request using wget

Using wget to request a page from a version of your http-server example adapted for T1S,
I found that at the end of the wget communication a Closing state is entered,
that will result in a panic within a switch statement in ControlBlock.Recv.

level=INFO msg=TCP:tx-statechange port=80 old=Established new=FinWait1 txflags=[FIN,ACK]
level=INFO msg=TCP:rx-statechange port=80 old=FinWait1 new=Closing rxflags=[FIN,ACK]
panic: unexpected stateClosing

goroutine 1 [running]:
github.com/soypat/seqs.(*ControlBlock).Recv(0x40000cc890, {0xf99d3916, 0x4b4, 0x0, 0xf8da, 0x10})
        github.com/soypat/[email protected]/control_user.go:187 +0x744

Apparently the condition seg.ACK == tcb.snd.NXT in github.com/soypat/[email protected]/control.go:245 is not fulfilled in this case.

When using a browser instead of wget, I don't observe the Closing state: In this case, a direct
transition from FinWait1 to TimeWait can be seen:

level=INFO msg=TCP:rx-statechange port=80 old=FinWait1 new=TimeWait rxflags=[FIN,ACK]

I added the lines

	case StateClosing:
		if seg.Flags.HasAny(FlagACK) {
			tcb.state = StateTimeWait
			pending, err = FlagACK, nil
		}

-- based on a TCP state diagram -- to the switch statement in ControlBlock.Recv, which results in the following message flow:

level=INFO msg=TCP:tx-statechange port=80 old=Established new=FinWait1 txflags=[FIN,ACK]
level=INFO msg=TCP:rx-statechange port=80 old=FinWait1 new=Closing rxflags=[FIN,ACK]
level=INFO msg=TCP:rx-statechange port=80 old=Closing new=TimeWait rxflags=[ACK]
level=INFO msg=lst:freeConnForReuse lport=80 rport=49158

Not sure if the added lines are sufficient.

Zero Copy Rx packet buffering

Seqs today

1. eth package which represents packet headers as structs with fields..
2. Above implies that when packets are parsed the header bytes are copied onto the stack
3. The stack.TCPPacket and stack.UDPPacket types are more resource intensive than really necessary since they require copying IP/TCP options and payload

While point 3 can be easily fixed by just passing around the buffer, removing copying in point 2 is not as easy and would require the creation of a new package.

zc package

A Zero Copy package is proposed to reduce sopying during packet processing and alleviating stack growth.
This package would likely have a larger scope than the current eth package. Below are some thoughts/suggestions on types to start a conversation:

1. A base Buffer type which has a backing []byte type. This type offers no guarantees of what is contained inside the buffer. It offers methods such as

   func (b Buffer) Ethernet() (EthernetBuffer, error)
   func (b Buffer) IPv4() (IPv4Buffer, error)

2. These EthernetBuffer and IPv4Buffer types are guaranteed to contain valid buffer information up to where their header ends. All these really need is a single uintptr or unsafe.Pointer backing type. Their methods would be roughly equivalent to the fields of the eth types today and would use unsafe package to extract the data. Methods should check for nullity and panic if pointer is null, which means the Ethernet/IPv4 method returned an error and the error was ignored before trying to extract data. Maybe as a first approximation the backing type can be a byte slice and this be changed in the future to use thin pointers.

3. TCP/UDP: TCPBuffer and UDPBuffer types, which can be returned by IPv4Buffer (along with an error since buffer length may not be sufficient). The TCPBuffer type would need to be a semi-fat pointer since it has no information on the end of the payload. The UDPBuffer type could do with only a thin pointer.

   func (ipb IPv4Buffer) TCP() (TCPBuffer, error)
   func (ipb IPv4Buffer) UDP() (UDPBuffer, error)

Development

I think the above description is enough to start writing and testing the package out. The first step I'd imagine would be to start with a partial/complete zc package implementation and try replacing the packet parsing in PortStack.RecvEth method and benchmark speed improvements to see if Zero Copy benefits are really worth it.

tcb:rcv.reject with segment drop stops TCP

I tried both the reference implementation from the book Automate Your Home Using Go and your example/http-server code.

It works until the first request appears.
(Sometimes !dhcpclient.IsDone() although the DHCP server assigned that IP, but yeah, I can somewhat fix that setting the same RequestedIP in config)

-- snip ---
time=1970-01-01T00:00:09.011Z level=INFO msg=listening addr=http://192.168.160.203:80
time=1970-01-01T00:00:25.714Z level=INFO msg=TCP:rx-statechange port=80 old=Listen new=SynRcvd rxflags=[SYN]
time=1970-01-01T00:00:25.869Z level=ERROR msg=tcb:rcv.reject err="drop segment"

After that happened all TCP requests time out or fail as the particular IP address is no longer reachable.
No more logs are spawned, it seems dead.

With curl sometimes I can get up to three requests through and then it stops. But with Firefox Developer Edition it stops directly at the first approach.

I know that TCP/IP is kinda fragile and things like that might happen.
Yet I'd expect that system to somehow discard these errors gracefully and keep on receiving.

If you do have any idea on what makes this happen and how to cope with it, I'd happily fork and fiddle around.

DHCP not getting IP on TP-Link AC-1750

Running examples/tcpserver/main.go in soypat/cyw43439, I get a Wifi connection to my test router TP-Link AC1750, but no DHCP IP is given.

sfeldma@nuc:~/work/cyw43439$ tinygo flash -target=pico -opt=1 -stack-size=32kb -size=short -monitor  ./examples/tcpserver/
   code    data     bss |   flash     ram
 564232   15676    3916 |  579908   19592
Connected to /dev/ttyACM1. Press Ctrl-C to exit.
starting program
time=1970-01-01T00:00:02.001Z level=DEBUG msg="starting program"



MAC: 28:cd:c1:01:13:7c
dhcp ongoing...
time=1970-01-01T00:00:06.980Z level=DEBUG msg=UDP:send plen=590
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
time=1970-01-01T00:00:09.070Z level=DEBUG msg=UDP:recv plen=548
time=1970-01-01T00:00:09.071Z level=DEBUG msg=dhcp-rx msgtype=2
time=1970-01-01T00:00:09.072Z level=DEBUG msg=UDP:send plen=590
time=1970-01-01T00:00:09.121Z level=DEBUG msg=UDP:recv plen=548
time=1970-01-01T00:00:09.122Z level=DEBUG msg=dhcp-rx msgtype=6
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...
dhcp ongoing...