aschils/packetdrill_mptcp

A packetdrill version allowing to 1) use multiple sockets at the same time, 2) test multipath tcp network stack implementation.

packetdrill_mptcp's Issues

Packetdrill does not complain when there is a missing data-ack

Seen packet-trace (with tcpdump):
14:07:32.179996 IP 192.0.2.1.8080 > 192.168.0.1.49727: Flags [FP.], seq 1001:2001, ack 1, win 257, options [mptcp dss fin ack 2405626053 seq 124797060 subseq 1001 len 1000 csum 0x1f8b], length 1000
14:07:32.234807 IP 192.168.0.1.49727 > 192.0.2.1.8080: Flags [.], ack 2002, win 260, options [nop,nop,TS val 4294907838 ecr 700], length 0

Expected however with packetdrill:
// Receive a segment with a FIN. Problem: it crashes the kernel in user-context
0.300 < FP. 1001:2001(1000) ack 1 win 257
0.300 > . 1:1(0) ack 2002

As can be seen, the second packet of the tcpdump-trace does not contain a DATA_ACK, but packetdrill is expecting one. However, it does not complain about an incorrect packet-sequence.

Packetdrill complains about null-packet

Seems to me to not make sense:

$ sudo packetdrill ./dss_ssn_specified_client.pkt 
./dss_ssn_specified_client.pkt:19: error handling packet: (null)

Script:

// mptcp v0.88
// connect() function, connection initiated by the kernel

// create a socket
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0.000 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
+0.000 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0

// Establish connection and verify that there was no error.
+0.0 connect(3, ..., ...) = -1 EINPROGRESS (Operation now in progress)
+0.0 > S 0:0(0) <mss 1460,sackOK,TS val 100 ecr 0,nop,wscale 7,mp_capable a> sock(3)
+0.0 < S. 0:0(0) ack 1 win 5792 <mss 1460,sackOK,nop,nop,nop,wscale 7,mp_capable b> sock(3)
+0.0 > . 1:1(0) ack 1 <...> sock(3) 
0.200 getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
0.200 fcntl(3, F_SETFL, O_RDWR) = 0   // set back to blocking

+0 write(3, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1 <dss dack4 dsn4>
+0.1 < . 1:1(0) ack 1001 win 225 <dss dack4> sock(3)

Or is something wrong with my script?

Support new subflows when the kernel is the client

As of now, I do not have the impression that it is possible to allow the creationg of new subflows when the kernel is the client.

If it is possible, how can this be done?

The kernel decides on his own to send a SYN+MP_JOIN. Packetdrill somehow needs to capture this one.

setsockopt-so_linger

Message from syslogddebian at Apr 14 01:32:33 ...kernel:[138647.861520] EIP:
[] mptcp_send_active_reset+0x49/0x11f SS:ESP 0068:f6ca7f18

This message is shown when the test 'so_linger_onoff-1_linger-0_rst.pkt' is run on debian-7.4.0-i386 with mptcp 0.88v. (test from 'examples/tests/sock_headers' folder)

Running the same test on ubuntu (13.10) x64 there is any problem.

Packetdrill does multiple calls to the random number generator

I experience that packetdrill does multiple calls to the random number generator to generate keys or nonces (for the MP_JOIN).

It looks to me that it always calls new_subflow_inbound, even if we are retransmitting SYN+MP_JOIN's, like:

0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(3, {sa_family = AF_INET, sin_port = htons(13000), sin_addr = inet_addr("192.168.0.1")}, ...) = 0
+0  listen(3, 1) = 0

+0  socket(..., SOCK_STREAM, IPPROTO_TCP) = 5
+0  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(5, {sa_family = AF_INET, sin_port = htons(13001), sin_addr = inet_addr("192.168.0.1")}, ...) = 0
+0  listen(5,1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7,mp_capable key_a> sock(3)
+0  > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7,mp_capable key_b> sock(3)
+0  < . 1:1(0) ack 1 win 257 <mp_capable key_a key_b> sock(3)
+0  accept(3, ..., ...) = 4

//First subflow
+0  < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7,mp_join_syn address_id=0 token=sha1_32(key_b)> sock(5)
+0  > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7, mp_join_syn_ack backup=0 address_id=0 sender_hmac=trunc_l64_hmac(key_b key_a) > sock(5)
+0.1  < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7,mp_join_syn address_id=0 token=sha1_32(key_b)> sock(5)
+0  > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7, mp_join_syn_ack backup=0 address_id=0 sender_hmac=trunc_l64_hmac(key_b key_a) > sock(5)
+0  < . 1:1(0) ack 1 win 32792 <mp_join_ack sender_hmac=full_160_hmac(key_a key_b)> sock(5)
+0 mp_join_accept(5) = 6

+0 > . 1:1(0) ack 1 <...> sock(6) // reliably mp_join_ack

I think it is ok that packetdrill uses static values for the random nonce or keys. If somebody wants to use different values he can still set them manually in the packetdrill-script.

so_oobinline_client.pkt Invalid TCP flag

Running ./so_oobinline_client.pkt ...
./so_oobinline_client.pkt:26: semantic error: Invalid TCP flag: 'U'

Fix mp_capable_connect.pkt

It uses the wrong destination-port in the SYN/ACK.

tcp_quickack_off.pkt fails with "error handling packet: (null)"

mptcp@mptcp:~/examples/tests/tcp_headers$ sudo packetdrill --tolerance_usecs=50000 ./tcp_quickack_off.pkt
./tcp_quickack_off.pkt:23: error handling packet: (null)

window-scaling is not respected

If the window-scaling of the window-scaling option is not correct packetdrill will not show this error. This is a bug specific to packetdrill_mptcp. The upstream packetdrill code does not have this bug.

Resetting a subflow causes weird behavior

0   socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(3, ..., ...) = 0
+0  listen(3, 1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,mp_capable a> sock(3)
+0  > S. 0:0(0) ack 1 win 28800 <mss 1460,mp_capable b> sock(3)
+0  < . 1:1(0) ack 1 win 29200 <mp_capable a b> sock(3)

+0  accept(3, ..., ...) = 4

/* establish an additional subflow  and reset it before 3-way handshake finishes */
+0  socket(..., SOCK_STREAM, IPPROTO_TCP) = 5
+0  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(5, {sa_family = AF_INET, sin_port = htons(13001), sin_addr = inet_addr("192.168.0.1")}, ...) = 0
+0  listen(5,1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,mp_join_syn backup=0 address_id=0 token=sha1_32(b)> sock(5)
+0  > S. 0:0(0) ack 1 win 28800 <mss 1460,mp_join_syn_ack backup=0 address_id=0 sender_hmac=trunc_l64_hmac(key_b key_a) > sock(5)
+0  < R. 1:1(0) ack 1 win 32792 sock(5)

/* Client send 1000 bytes */
+1  < P. 1:1001(1000) ack 1 win 32792 <dss dack4 dsn4> sock(4)
+0  > . 1:1(0) ack 1001 win 59000 <dss dack4> sock(4)

+0  read(4, ..., 1000) = 1000

+1  < R. 1001:1001(0) ack 1 win 32792 sock(4)

With this script the data segment (1000 bytes) is sent over the wrong subflow (i.e. not the master) even if I use "sock(4)". Packetdrill exits with the following error:

join-embryonic.pkt:24: error handling packet: packet is not for expected socket

How can one specify the subflow-sequence number to be used in a DSS-opt

I can see how one specifies the data-seq and the data-ack, but I don't see how one can specify the subflow-sequence number to be used.

Packetdrill does not take dll into account when calculating the dss checksum

With the following script:

0   socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(3, ..., ...) = 0
+0  listen(3, 1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7,mp_capable a>
+0  > S. 0:0(0) ack 1 <mss 1460,sackOK,nop,nop,nop,wscale 7, mp_capable b> 
+0  < . 1:1(0) ack 1 win 257 <mp_capable a b, dss dack4> //=trunc_r64_hmac(b)>  
+0  accept(3, ..., ...) = 4 

+0.1 < PF. 1:1001(1000) ack 1 win 257 <dss dack4 dsn4 dll=1000 FIN>

I voluntarily put a dll of 1000 here, which is incorrect as it should rather be 1001 (because of the DATA_FIN). What packetdrill does is that it actually computes the checksum over the whole payload and not over the first 999 bytes (1000 -1, because of the DATA_FIN).

I have the feeling that this is because of the following lines:

buff_chk.dll = (u16)tcp_payload_length;

Looks to me that we take the payload-length to compute the checksum instead of considering the dll. Please, make also sure that the checksum is correct when specifying:

+0.1 < P. 1:1001(1000) ack 1 win 257 <dss dack4 dsn4 ssn=500 dll=500>

Here, the subflow-sequence number says that the mapping only starts from byte 500. So, packetdrill should not compute the checksum over the whole payload.

so_linger_onoff-1_linger-0_rst.pkt - wrong packet-sequence

The script does not fail, but that's because the host does a seamless fallback to regular TCP.

This is, because the ack, after receiving the data does not contain a data-ack:

// Write some data, receive an ACK
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1 <...>
+0 < . 1:1(0) ack 1001 win 257

In the end the host sends a rst, but through the wrong code-path as it assums that the connection has fallen back. So, the rst from the host will not contain the MP_FAST_CLOSE-option, which it should actually use.

Btw., the script does not check for this MP_FAST_CLOSE-option, which it actually should.

Incorrect dack after data-fin

Script:

+0   socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(3, ..., ...) = 0
+0  listen(3, 1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,nop,wscale 7,mp_capable a>
+0  > S. 0:0(0) ack 1 <mss 1460,nop,wscale 7, mp_capable b> 
+0  < . 1:1(0) ack 1 win 257 <mp_capable a b, dss dack4> 
+0  accept(3, ..., ...) = 4 

+0  write(4, ..., 1000) = 1000
+0  > P. 1:1001(1000) ack 1 <dss dack4 dsn4>
+0 < . 1:1(0) ack 1001 win 225 <dss dack4>

+0.1 close(4) = 0
+0 > . 1001:1001(0) ack 1 <dss dack4 dsn4 FIN>

+0 < . 1:1(0) ack 1001 win 225 <dss dack4>
+0 > F. 1001:1001(0) ack 1 <dss dack4>

Actual sequence:

22:06:21.136467 IP 192.0.2.1.44400 > 192.168.0.1.8080: Flags [S], seq 0, win 32792, options [mss 1460,nop,wscale 7,mptcp capable csum {0x26e8b783b8a3445}], length 0
22:06:21.136518 IP 192.168.0.1.8080 > 192.0.2.1.44400: Flags [S.], seq 2107891968, ack 1, win 28800, options [mss 1460,nop,wscale 7,mptcp capable csum {0x18f7e5a0f3d9fa4d}], length 0
22:06:21.137580 IP 192.0.2.1.44400 > 192.168.0.1.8080: Flags [.], ack 1, win 257, options [mptcp capable csum {0x26e8b783b8a3445,0x18f7e5a0f3d9fa4d},mptcp dss ack 3827016030], length 0
22:06:21.137655 IP 192.168.0.1.8080 > 192.0.2.1.44400: Flags [P.], seq 1:1001, ack 1, win 225, options [mptcp dss ack 350560040 seq 3827016030 subseq 1 len 1000 csum 0x319e], length 1000
22:06:21.137705 IP 192.0.2.1.44400 > 192.168.0.1.8080: Flags [.], ack 1001, win 225, options [mptcp dss ack 3827017030], length 0
22:06:21.239056 IP 192.168.0.1.8080 > 192.0.2.1.44400: Flags [.], ack 1, win 225, options [mptcp dss fin ack 350560040 seq 3827017030 subseq 0 len 1 csum 0x319e], length 0
22:06:21.240397 IP 192.0.2.1.44400 > 192.168.0.1.8080: Flags [.], ack 1001, win 225, options [mptcp dss ack 3827016030], length 0

This last data-ack should be 3827016031, because we have received a data-fin with seq-number 3827016030.

Strange values in the dss-checksum

When using the following script:

0   socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0  bind(3, ..., ...) = 0
+0  listen(3, 1) = 0

+0  < S 0:0(0) win 32792 <mss 1460,nop,wscale 7,mp_capable a>
+0  > S. 0:0(0) ack 1 <mss 1460,nop,wscale 7, mp_capable b>
+0  < . 1:1(0) ack 1 win 257 <mp_capable a b, dss dack4> //=trunc_r64_hmac(b)>
+0  accept(3, ..., ...) = 4

+0  write(4, ..., 1000) = 1000
+0  > P. 1:1001(1000) ack 1 <dss dack4 dsn4 >
+0 < . 1:1(0) ack 1001 win 225 <dss dack4>

+0.1 < PF. 1001:1003(2) ack 1001 win 257 <dss dack4 dsn4=1001 ssn=1000 dll=3>
+0 > . 1001:1001(0) ack 1 <dss dack4>

+0.1 < P. 1:1001(1000) ack 1001 win 257 <dss dack4 dsn4=1 ssn=1 dll=1000 FIN>
+0 > . 1001:1001(0) ack 1001 <dss dack4=1001>

I get the following trace:

21:32:14.538323 IP 192.0.2.1.48340 > 192.168.0.1.8080: Flags [S], seq 0, win 32792, options [mss 1460,nop,wscale 7,mptcp capable csum {0x2d90f01f7d89c4a}], length 0
21:32:14.549530 IP 192.168.0.1.8080 > 192.0.2.1.48340: Flags [S.], seq 3230151785, ack 1, win 28800, options [mss 1460,nop,wscale 7,mptcp capable csum {0xd60e410a937986a3}], length 0
21:32:14.550076 IP 192.0.2.1.48340 > 192.168.0.1.8080: Flags [.], ack 1, win 257, options [mptcp capable csum {0x2d90f01f7d89c4a,0xd60e410a937986a3},mptcp dss ack 2353089085], length 0
21:32:14.550158 IP 192.168.0.1.8080 > 192.0.2.1.48340: Flags [P.], seq 1:1001, ack 1, win 225, options [mptcp dss ack 2183289512 seq 2353089085 subseq 1 len 1000 csum 0xfb4d], length 1000
21:32:14.550205 IP 192.0.2.1.48340 > 192.168.0.1.8080: Flags [.], ack 1001, win 225, options [mptcp dss ack 2353090085], length 0
21:32:14.650418 IP 192.0.2.1.48340 > 192.168.0.1.8080: Flags [FP.], seq 1001:1003, ack 1001, win 257, options [mptcp dss ack 2353090085 seq 2183290512 subseq 1000 len 3 csum 0x8f5], length 2
21:32:14.650504 IP 192.168.0.1.8080 > 192.0.2.1.48340: Flags [.], ack 1, win 225, options [mptcp dss ack 2183289512], length 0
21:32:14.753212 IP 192.0.2.1.48340 > 192.168.0.1.8080: Flags [P.], seq 1:1001, ack 1001, win 257, options [mptcp dss fin ack 2353090085 seq 2183289512 subseq 1 len 1000 csum 0x8f5], length 1000

You can see that the dss-csum of the last two data-packets generated by packet-drill are the same (0x8f5). This looks incorrect to me. And actually, the kernel complains about incorrect dss-checksum.

mp_join_packd_to_kernel_ipv6.pkt produces "Bad file descriptor"

mptcp@mptcp:~/examples/mptcp/mp_join$ sudo packetdrill ./mp_join_packd_to_kernel_ipv6.pkt
close: Bad file descriptor

Incorrect data-length when adding data-fin flag

The DATA_FIN consumes one byte in the data-sequence space. So, when sending a DATA_FIN, the data-length must be the size of the payload + 1.

so_rcvbuf-sndbuf_set_before_listen.pkt does not correctly test the window

If you try out regular TCP, then after setting the recv-buffer to 40KB, the socket will behave different when sending the SYN/ACK.

With regular TCP, the SYN/ACK will have a window-scale of 0 (because the window-field of the TCP header can announce the fixed 40KB).
However, in your test you do not test this. And actually, MPTCP is failing there. It announces a window-scale of 7 although it should not.

ioctl-siocinq-server.pkt parse error

The message:

Running ./ioctl-siocinq-server.pkt ...
': syntax error-server.pkt:4: parse error at '

Make data-seq-number and data-ack relative

In packetdrill, for regular TCP, one uses relative sequence numbers:
+0.0 > . 1:1(0) ack 1

However, for MPTCP we are (for now) forced to use absolute sequence numbers:

This means, if we want to specify the data-seq we always have to write "trunc_r64_hmac(b) +". I think it should allow to specify relative sequence numbers like for regular TCP. This makes live much easier and it still allows one to specify "wrong" or out-of-window sequence numbers.

packetdrill segfault

I have the following segfault (seen in dmesg):

934.129310] packetdrill[1530]: segfault at 8 ip 080749ba sp bfb02230 error 4 in packetdrill[8048000+1e3000]

with the script:

0       `sysctl -w net.ipv4.tcp_keepalive_intvl=1`
+0      `sysctl -w net.ipv4.tcp_keepalive_probes=5`
+0      `sysctl -w net.ipv4.tcp_keepalive_time=10`
+0      `ip tcp_metrics flush`

+0      socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0      setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0      bind(3, ..., ...) = 0
+0      listen(3, 1) = 0

+0      setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0

+0      < S 0:0(0) win 32792 <mss 1460,mp_capable a> sock(3)
+0      > S. 0:0(0) ack 1 <mss 1460,mp_capable b> sock(3)
+0      < . 1:1(0) ack 1 win 29200 <mp_capable a b> sock(3)

+0      accept(3, ..., ...) = 4

/* Client send 1000 bytes */
+0      < . 1:1001(1000) ack 1 win 32792 <dss dack4 dsn4> sock(4)
+0      > . 1:1(0) ack 1001 <dss dack4> sock(4)

+0      read(4, ..., 1000) = 1000

/* establish an additional subflow */
+0      socket(..., SOCK_STREAM, IPPROTO_TCP) = 5
+0      setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0      bind(5, {sa_family = AF_INET, sin_port = htons(13001), sin_addr = inet_addr("192.168.0.1")}, ...) = 0
+0      listen(5,1) = 0

+0      < S 0:0(0) win 32792 <mss 1460,mp_join_syn backup=0 address_id=0 token=sha1_32(b)> sock(5)
+0      > S. 0:0(0) ack 1 <mss 1460,mp_join_syn_ack backup=0 address_id=0 sender_hmac=trunc_l64_hmac(key_b key_a) > sock(5)
+0.1    < . 1:1(0) ack 1 win 32792 <mp_join_ack sender_hmac=full_160_hmac(key_a key_b)> sock(5)
+0      > . 1:1(0) ack 1 <...> sock(6)

+0      mp_join_accept(5) = 6

/* Keepalives are sent on all subflows */
+9.9    > . 0:0(0) ack 1 <dss dack4> sock(6)
+0      > . 0:0(0) ack 1001 <dss dack4> sock(4)
/* And repeat the probe every 3 seconds if there are no response. */
+1      > . 0:0(0) ack 1 <dss dack4> sock(6)
+0      > . 0:0(0) ack 1001 <dss dack4> sock(4)
+1      > . 0:0(0) ack 1 <dss dack4> sock(6)
+0      > . 0:0(0) ack 1001 <dss dack4> sock(4)
+1      > . 0:0(0) ack 1 <dss dack4> sock(6)
+0      > . 0:0(0) ack 1001 <dss dack4> sock(4)
+1      > . 0:0(0) ack 1 <dss dack4> sock(6)
+0      > . 0:0(0) ack 1001 <dss dack4> sock(4)

/* should fastclose */
+1      > . 1:1(0) ack 1001 <mp_fastclose> sock(4)
+0      > R. 1:1(0) ack 1 <dss dack4> sock(6)
+0      < R. 1001:1001(0) ack 1 win 32792 sock(4)

+0.1    `sysctl -w net.ipv4.tcp_keepalive_intvl=75`
+0      `sysctl -w net.ipv4.tcp_keepalive_probes=9`
+0      `sysctl -w net.ipv4.tcp_keepalive_time=7200`

Seems to happen when it sends (or receives) the reset at the end.