juniper / warp17 Goto Github PK
View Code? Open in Web Editor NEWThe Stateful Traffic Generator for Layer 1 to Layer 7
License: BSD 3-Clause "New" or "Revised" License
The Stateful Traffic Generator for Layer 1 to Layer 7
License: BSD 3-Clause "New" or "Revised" License
Hello, I'm trying to build warp17 1.7 release and get the following error:
CC tpg_test_mgmt_api.o
/opt/warp17-1.7/src/tpg_test_mgmt_api.c: In function ‘test_mgmt_set_vlan_sockopt’:
/opt/warp17-1.7/src/tpg_test_mgmt_api.c:1556:5: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation]
if (opts->has_vlanso_id)
^~
/opt/warp17-1.7/src/tpg_test_mgmt_api.c:1558:9: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’
old_opts.vlanso_pri = opts->vlanso_pri;
^~~~~~~~
cc1: all warnings being treated as errors
/usr/local/share/dpdk/mk/internal/rte.compile-pre.mk:138: recipe for target 'tpg_test_mgmt_api.o' failed
make[2]: *** [tpg_test_mgmt_api.o] Error 1
/usr/local/share/dpdk/mk/rte.extapp.mk:42: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/opt/warp17-1.7'
Makefile:72: recipe for target 'all' failed
make: *** [all] Error 2
I found that in tpg_test_mgmt_api.c in function test_mgmt_set_vlan_sockopt missed brackets after statement if (opts->has_vlanso_id)
Due to a bug in the DPDK i40e PMD (or X710 firmware) Warp17 sometimes fails to start X710 interfaces.
Original bug report: http://dpdk.org/ml/archives/users/2017-July/002200.html
Until the bug is fixed in DPDK/firmware a workaround fix should be added to Warp17 to avoid these errors. An option could be to delay restarting ports (similar to what happens in DPDK's test-pmd
when stopping ports).
In function test_display_stats_hdr(), the links speed in bytes is calculated as below:
link_speed_bytes = (uint64_t)link_info->link_speed * 1024 * 1024 / 8;
Bit rates are represented in units of 1000 not 1024, for e.g. 1kbps = 1000 bits/sec.
So the link speed calculation must be as below:
link_speed_bytes = (uint64_t)link_info->link_speed * 1000* 1000 / 8;
The "origin" remote name is hardcoded in the release.sh script. This might not always be correct.
Also, the check which prevents running the script on a non-master branch is commented out.
Hello Team,
As discussed raising an issue for TCP delay ack feature.
The current version of WARP sends TCP acknowledgment for every segment it receives due to which we are seeing a drop in throughput.
Is there milestone to implement or enhance to support TCP delayed ack mechanism as mentioned in https://tools.ietf.org/html/rfc1122 section 4.2.3.2
Thank & Regards,
Anand
Hi,
I'm trying to compile warp17 on Debian Stretch and I'm getting following errors. Any ideas? Source code from 1.6 release.
~/warp17-1.6$ make
make[1]: Entering directory '/home/tklicki/warp17-1.6/api'
PC-C warp17-common.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax specified for the proto file: warp17-common.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
warp17-common.proto:78:1: "Constants" declares support for enum aliases but no enum values share field numbers. Please remove the unnecessary 'option allow_alias = true;' declaration.
Makefile.api:118: recipe for target 'generated/c/warp17-common.pb-c.h' failed
make[1]: *** [generated/c/warp17-common.pb-c.h] Error 1
make[1]: Leaving directory '/home/tklicki/warp17-1.6/api'
Makefile:72: recipe for target 'all' failed
make: *** [all] Error 2
In function route_init(), the variable 'default_gw_per_port' is allocated for one instance'. This cause invalid memory access when multiple ports are used for traffic generation:
For e.g. in function route_v4_nh_lookup() where default_gw_per_port is accessed like this -> default_gw_per_port[port].re_nh.ip_v4;
default_gw_per_port = rte_zmalloc("default_gw_per_port",
sizeof(*default_gw_per_port),
0);
This should be
default_gw_per_port = rte_zmalloc("default_gw_per_port",
rte_eth_dev_count() * sizeof(*default_gw_per_port),
0);
Write a more clear and structured documentation that helps users and developers also through the code
The current WARP17 is released with constant window size as 1024 but when I try to increase the to_win_size parameter in ./warp17-master/api/warp17-sockopt.proto to 8000+ and run warp application connected back to back with http payload as 10000 bytes on client and server then I see two bugs/Issues.
As I configured 10000 bytes of application data and window size of 8000 bytes, application will give 8000 bytes of data to TCP(in test_case_execute_tcp_send with APP_CALL(send)) and warp will try to transmit 5 segments(tcp_data_send_segments(TCB_SEGS_PER_SEND) without ACK) with MTU size(say 1460 bytes*5 segments = 7300 bytes) and will schedule re-transmission for all 8000 bytes of data received from http app. Even though we get ACK for all 5 segments before RTO expiry WARP will wait till RTO expiry and then it will transmit the remaining bytes of data left(700 bytes) in re-transmission buffer. Which is not expected. Expected behavior should be as soon as we receive ACK for all the segments we need to cancel RTO and should start transmitting the remaining data/new data from http app.
Even though we are transmitting the left out data 700 bytes for the first time stats are getting updated as re-transmission bytes(can be seen using "show tsm statsictics").
Please find the profile configuration, warp17-sockopt.proto changes and "show tsm statistics" out put below.
add tests l3_intf port 0 ip 10.0.0.1 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.0.253
add tests client tcp port 0 test-case-id 0 src 10.0.0.1 10.0.0.1 sport 10001 10001 dest 10.0.0.253 10.0.0.253 dport 6001 6001
set tests timeouts port 0 test-case-id 0 init 0
set tests timeouts port 0 test-case-id 0 uptime infinte
set tests timeouts port 0 test-case-id 0 downtime infinite
set tests criteria port 0 test-case-id 0 run-time 3600
set tests client http port 0 test-case-id 0 GET google.com /index.html req-size 10000
add tests l3_intf port 1 ip 10.0.0.253 mask 255.255.255.0
add tests l3_gw port 1 gw 10.0.0.1
add tests server tcp port 1 test-case-id 0 src 10.0.0.253 10.0.0.253 sport 6001 6001
set tests server http port 1 test-case-id 0 200-OK resp-size 10000
start tests port 1
start tests port 0
show tests ui
message PortOptions {
optional uint32 po_mtu = 1 [default = 1500];
}
message TcpSockopt {
optional uint32 to_win_size = 1 [default = 8000];
optional uint32 to_syn_retry_cnt = 2 [default = 24]; // actually 8bit
optional uint32 to_syn_ack_retry_cnt = 3 [default = 24]; // actually 8bit
optional uint32 to_data_retry_cnt = 4 [default = 128]; // actually 8bit
optional uint32 to_retry_cnt = 5 [default = 12]; // actually 8bit
optional uint32 to_rto = 6 [default = 250]; // in msec
optional uint32 to_fin_to = 7 [default = 100]; // in msec
optional uint32 to_twait_to = 8 [default = 500]; // in msec
optional uint32 to_orphan_to = 9 [default = 100]; // in msec
optional bool to_skip_timewait = 10 [default = false];
}
warp17> show tsm statistics
Port 0 TSM statistics:
INIT : 0
LISTEN : 0
SYN_SENT : 0
SYN_RECV : 0
ESTAB : 1
FIN_WAIT_1 : 0
FIN_WAIT_2 : 0
LAST_ACK : 0
CLOSING : 0
TIME_WAIT : 0
CLOSE_WAIT : 0
CLOSED : 0
SYN retrans TO : 0
SYN/ACK retrans TO : 0
Retrans TO : 0
Retrans bytes : 126000
Missing seq : 0
SND win full : 0
Port 1 TSM statistics:
INIT : 0
LISTEN : 1
SYN_SENT : 0
SYN_RECV : 0
ESTAB : 1
FIN_WAIT_1 : 0
FIN_WAIT_2 : 0
LAST_ACK : 0
CLOSING : 0
TIME_WAIT : 0
CLOSE_WAIT : 0
CLOSED : 0
SYN retrans TO : 0
SYN/ACK retrans TO : 0
Retrans TO : 0
Retrans bytes : 126000
Missing seq : 0
SND win full : 0
Invalid command line args are not properly detected:
sudo ./build/warp17 --lcore="2-6" -n 2 -- --qmap-default max-q --ucb-p --cmd-file ../warp17.cfg/tc50K.cfg
--ucb-p
is not a valid argument but no error is returned. Moreover the --cmd-file
argument is silently skipped.
This UDP test is used to send 6k jumbo wire rate on a 10G interface through an MX with ms-mpc, basic nat config with enough ports, etc (no drops found on the MX), expecting a 6k reply from the server on the outside nat domain.
Drops are expected, therefore test parameters are 1s uptime 0s init 0s down with enough clients to keep the line saturated. It initially works fine, with about 200kpps in both directions (the max possible) but eventually traffic coming back from the server drops down, here are the rates on the interface to the server:
Input rate : 542642616 bps (11068 pps)
Output rate : 8906039072 bps (184680 pps)
Just restarting the test brings it back to max again. Full config for warp test and relevant config for MX attached.
Hi, WARP17 team,
Is there mileston to support PPPoE in WARP? If not, is it easy to be plugged in by myself ? I am quite new to WARP17, any advice would be appreciated.
I am sorry that I can only put questions here instead of WARP google group.
ARP packets/bytes received are getting added into port link RX counters, but ARP packets/bytes sent are not getting added into port link TX counters.
Hi,WARP Team,
I am a new programmer,recently, I would like to learn something about the WARP17 and how does the DPDK handle the message quickly,but when I run the test script,it will always print an error “(tpg_pktloop.c)Failed to poll for messages,(-22)”,I don't know where is the wrong process and how to solve it ,can you help me?Thanks very much!
Hi Dumitru,
Raising this issue for tracking .
or
Is it possible to have a mechanism of calculating the number of cores (approximate) required for a particular test based on sessions and rates. (~10 % of tolerance should be fine).
Thanks,
Rakesh mishra
Hi Dumitru,
When a request and response size is configured less than 17k on WARP17, We are achieving the configured send rate with maximum 1% tolerance. But when we increase the request and response size beyond 17k bytes the send rate fluctuates. (Only ~70% of the send rate is achieved)
*This may likely be the issue #20.
Please find the log below
WARP17 CLI :
/root/WARP17/warp17-dev-common/build/warp17 --lcores 0@0,1@0,2@2,3@4,4@6,5@8,6@10 -m 120000 -n 16 -- --qmap-default max-q --ucb-pool-sz 1 --tcb-pool-sz 90000
config :
add tests l3_intf port 0 ip 30.0.0.2 mask 255.0.0.0
add tests l3_gw port 0 gw 30.0.0.1
add tests l3_intf port 1 ip 40.0.0.2 mask 255.0.0.0
add tests l3_gw port 1 gw 40.0.0.1
add tests client tcp port 0 test-case-id 0 src 1.0.0.1 1.0.0.200 sport 1024 1073 dest 40.0.0.2 40.0.0.2 dport 80 80
set tests criteria port 0 test-case-id 0 run-time 6000
set tests async port 0 test-case-id 0
set tests rate port 0 test-case-id 0 open 1000
set tests rate port 0 test-case-id 0 close 0
set tests rate port 0 test-case-id 0 send 30000
set tests timeouts port 0 test-case-id 0 init 0
set tests timeouts port 0 test-case-id 0 uptime infinite
set tests client http port 0 test-case-id 0 GET 40.0.0.2 /index.html req-size 20000
set tests tcp-options port 0 test-case-id 0 win-size 65535
add tests server tcp port 1 test-case-id 0 src 40.0.0.2 40.0.0.2 sport 80 80
set tests criteria port 1 test-case-id 0 run-time 6000
set tests server http port 1 test-case-id 0 200-OK resp-size 20000
set tests tcp-options port 1 test-case-id 0 win-size 65535
start tests port 1
start tests port 0
Logs :
root@ubuntu-snp:~# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 3
cpu cores : 6
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 3
cpu cores : 6
apicid : 38
initial apicid : 38
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 4
cpu cores : 6
apicid : 8
initial apicid : 8
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 4
cpu cores : 6
apicid : 40
initial apicid : 40
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 5
cpu cores : 6
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 5
cpu cores : 6
apicid : 42
initial apicid : 42
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 6
cpu cores : 6
apicid : 12
initial apicid : 12
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 6
cpu cores : 6
apicid : 44
initial apicid : 44
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 10
cpu cores : 6
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 10
cpu cores : 6
apicid : 52
initial apicid : 52
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 0
siblings : 6
core id : 11
cpu cores : 6
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E7-8893 v2 @ 3.40GHz
stepping : 7
microcode : 0x70d
cpu MHz : 3400.157
cache size : 38400 KB
physical id : 1
siblings : 6
core id : 11
cpu cores : 6
apicid : 54
initial apicid : 54
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
bugs :
bogomips : 6801.71
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
root@ubuntu-snp:~#
warp17> show tests config port 0
Port 0 config:
L3 Interface: 30.0.0.2/255.0.0.0
GW: 30.0.0.1
Test Case Id : 0
Test type : TCP CL
Async : true
Local IPs : [1.0.0.1 -> 1.0.0.200]
Local Ports : [1024 -> 1073]
Remote IPs : [40.0.0.2 -> 40.0.0.2]
Remote Ports : [80 -> 80]
Rate Open : 1000s/s
Rate Close : 0s/s
Rate Send : 30000s/s
Delay Init : 0s
Delay Uptime : INFINITE
Delay Downtime : INFINITE
HTTP CLIENT:
Request Method : GET
Request Object : /index.html
Request Host : 40.0.0.2
Request Size : 20000
warp17> show tests config port 1
Port 1 config:
L3 Interface: 40.0.0.2/255.0.0.0
GW: 40.0.0.1
Test Case Id : 0
Test type : TCP SRV
Async : false
Local IPs : [40.0.0.2 -> 40.0.0.2]
Local Ports : [80 -> 80]
HTTP SERVER:
Response Status : 200 OK
Response Size : 20000
warp17> show tests
ui [Fixed STRING]: show tests ui
config [Fixed STRING]: show tests config port <eth_port>
state [Fixed STRING]: show tests state port <eth_port>
stats [Fixed STRING]: show tests stats port <eth_port> test-case-id <tcid>
tcp-options [Fixed STRING]: show tests tcp-options port <eth_port> test-case-id <tcid>
warp17> show tests state port 0
Port 0
TcId Type Criteria State Runtime Quick stats
0 TCP CL Run Time: 6000s RUNNING 62.13s Run Time: 62s
warp17> show tests state port 1
Port 1
TcId Type Criteria State Runtime Quick stats
0 TCP SRV Run Time: 6000s RUNNING 67.09s Run Time: 67s
warp17> show tests stats port 0
[UINT32]: show tests stats port <eth_port> test-case-id <tcid>
warp17> show tests stats port 0 test-case-id 0
Port 0, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20810
Requests Responses Invalid Msg No Len Trans-Enc
1666749 1656749 0 0 0
warp17> show tests stats port 1 test-case-id 0
Port 1, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20814
Requests Responses Invalid Msg No Len Trans-Enc
1776896 1776896 0 0 0
warp17> show tests stats port 0 test-case-id 0
Port 0, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20762
Requests Responses Invalid Msg No Len Trans-Enc
1864373 1854373 0 0 0
warp17> show tests stats port 1 test-case-id 0
Port 1, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20823
Requests Responses Invalid Msg No Len Trans-Enc
1901740 1901740 0 0 0
warp17> show tests stats port 0 test-case-id 0
Port 0, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20870
Requests Responses Invalid Msg No Len Trans-Enc
1958021 1948021 0 0 0
warp17> show tests stats port 1 test-case-id 0
Port 1, Test Case 0 Statistics:
Estab/s Closed/s Data Send/s
0 0 20812
Requests Responses Invalid Msg No Len Trans-Enc
2005808 2005808 0 0 0
warp17> show tcp statistics
Port 0 TCP statistics:
Received Packets : 60523873
Received Bytes : 46103129860
Sent Ctrl Packets : 29492706
Sent Ctrl Bytes : 1592606124
Sent Data Packets : 32426231
Sent Data Bytes : 48294706858
Malloced TCBs : 0
Freed TCBs : 0
Not found TCBs : 0
TCB alloc errors : 0
Invalid checksum : 0
Small mbuf fragment : 0
TCP hdr to small : 0
Ctrl Failed Packets : 0
DATA Failed Packets : 0
DATA Clone Failed : 0
Reserved bit set : 0
Port 1 TCP statistics:
Received Packets : 60538758
Received Bytes : 46271066076
Sent Ctrl Packets : 29550528
Sent Ctrl Bytes : 1595728512
Sent Data Packets : 32354153
Sent Data Bytes : 48121062218
Malloced TCBs : 10005
Freed TCBs : 2923
Not found TCBs : 0
TCB alloc errors : 0
Invalid checksum : 0
Small mbuf fragment : 0
TCP hdr to small : 0
Ctrl Failed Packets : 0
DATA Failed Packets : 0
DATA Clone Failed : 0
Reserved bit set : 0
warp17> show tsm statistics
Port 0 TSM statistics:
INIT : 0
LISTEN : 0
SYN_SENT : 0
SYN_RECV : 0
ESTAB : 7090
FIN_WAIT_1 : 0
FIN_WAIT_2 : 0
LAST_ACK : 0
CLOSING : 0
TIME_WAIT : 0
CLOSE_WAIT : 0
CLOSED : 0
SYN retrans TO : 0
SYN/ACK retrans TO : 0
Retrans TO : 2910
Retrans bytes : 22595851111
Missing seq : 2923
SND win full : 0
Port 1 TSM statistics:
INIT : 0
LISTEN : 5
SYN_SENT : 0
SYN_RECV : 0
ESTAB : 7077
FIN_WAIT_1 : 0
FIN_WAIT_2 : 0
LAST_ACK : 0
CLOSING : 0
TIME_WAIT : 0
CLOSE_WAIT : 0
CLOSED : 0
SYN retrans TO : 0
SYN/ACK retrans TO : 0
Retrans TO : 2923
Retrans bytes : 19383127848
Missing seq : 2910
SND win full : 0
warp17> show memory statistics
Mempool MBUF RX:
Total Allocated: 47957
Total Free : 738475
Mempool MBUF TX HDR:
Total Allocated: 5528
Total Free : 518752
Mempool MBUF CLONE:
Total Allocated: 26596
Total Free : 759836
Mempool TCB:
Total Allocated: 17082
Total Free : 92142918
Mempool UCB:
Total Allocated: 0
Total Free : 1020
warp17> show link st
Bad arguments
warp17> show link
rate [Fixed STRING]: show link rate
warp17> show link rate
Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 3.96% RX: 3.91%
Port 1: link UP, speed 10Gbps, duplex full(manual), TX: 3.91% RX: 3.96%
warp17> show m
memory [Fixed STRING]: show memory statistics
memory [Fixed STRING]: show memory statistics details
msg [Fixed STRING]: show msg statistics
msg [Fixed STRING]: show msg statistics details
warp17> show msg statistics
MSG statistics:
Messages rcvd : 7482442
Messages sent : 7482435
Messages polled : 47588579674
Messages errors : 0
Messages proc err : 0
Messages allocated : 20
Messages alloc err : 0
Messages freed : 20
warp17> show m
memory [Fixed STRING]: show memory statistics
memory [Fixed STRING]: show memory statistics details
msg [Fixed STRING]: show msg statistics
msg [Fixed STRING]: show msg statistics details
warp17> show memory statistics
Mempool MBUF RX:
Total Allocated: 47955
Total Free : 738476
Mempool MBUF TX HDR:
Total Allocated: 5503
Total Free : 518782
Mempool MBUF CLONE:
Total Allocated: 26618
Total Free : 759812
Mempool TCB:
Total Allocated: 17082
Total Free : 92142918
Mempool UCB:
Total Allocated: 0
Total Free : 1020
warp17> show http statistics
Port 0 HTTP statistics:
HTTP Req Build Err : 0
HTTP Resp Build Err : 0
Port 1 HTTP statistics:
HTTP Req Build Err : 0
HTTP Resp Build Err : 0
warp17> show tests tcp-options port 0 test-case-id 0
WIN SYN SYN/ACK DATA RETRY RTO(ms) FIN(ms) TW(ms) ORP(ms) TW-SKIP ACK-DEL
----- --- ------- ---- ----- ------- ------- ------- ------- ------- -------
65535 24 24 128 12 50 100 500 100 0 0
warp17> show tests tcp-options port 1 test-case-id 0
WIN SYN SYN/ACK DATA RETRY RTO(ms) FIN(ms) TW(ms) ORP(ms) TW-SKIP ACK-DEL
----- --- ------- ---- ----- ------- ------- ------- ------- ------- -------
65535 24 24 128 12 50 100 500 100 0 0
warp17>
warp17>
warp17> show version
warp17 v0.13 @19:42:02, Sep 7 2017
warp17>
Thanks,
Rakesh Mishra
There is a bug where in the timers are not cancelled, when the acknowledgement arrives for the data segment, which causes the timer queues to keep growing (and also re-transmissions).
Setup : establish 1 connection with a test duration of say 2 seconds, in a back to back setup with port 0 as part of client profile and port 1 as part of server profile.
There after issue
warp17> show timer statistics
Port 0 TCP Timer statistics:
RTO Timer Set : 356
RTO Timer Cancelled : 1
RTO Timer Fired : 354
Port 1 TCP Timer statistics:
RTO Timer Set : 356
RTO Timer Cancelled : 1
RTO Timer Fired : 354
I have made a corresponding fix for the same,
refer : amjaingzb@ed5f281
src/tpg_tcp_sm.c
@@ -1262,6 +1262,8 @@ static int tsm_SF_estab(tcp_control_block_t tcb, tcpEvent_t event,
SEG_LE(seg_ack, tcb->tcb_snd.nxt)) {
/ Will update tcb_snd.una inside! */
tsm_cleanup_retrans_queu(tcb, seg_ack);
/* Cancel the retransmission timer */
tcp_timer_rto_cancel(&tcb->tcb_l4);
}
/*
The stats should be available for querying through:
Latest code failing with Mellanox Driver.
DPDK Version: dpdk-stable-16.11.1
EAL: Detected 56 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:08:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1013 net_mlx5
PMD: mlx5.c:421: mlx5_pci_probe(): PCI information matches, using device "mlx5_0" (SR-IOV: false, MPS: false)
PMD: mlx5.c:444: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:592: mlx5_pci_probe(): port 1 MAC address is 24:8a:07:a4:ff:c8
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 15b3:1013 net_mlx5
PMD: mlx5.c:421: mlx5_pci_probe(): PCI information matches, using device "mlx5_1" (SR-IOV: false, MPS: false)
PMD: mlx5.c:444: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:592: mlx5_pci_probe(): port 1 MAC address is 24:8a:07:a4:ff:98
warp17> USER1: Creating mempool global_cfg_mbuf-1 on core 1(size: 1024, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf-3 on core 3(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-3 on core 3(size: 262144, obj_size: 326, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-3 on core 3(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-3 on core 3(size: 5242879, obj_size: 312, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-3 on core 3(size: 5242879, obj_size: 200, priv: 0 cache: 0)
USER1: Creating mempool global_pkt_mbuf-4 on core 4(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-4 on core 4(size: 262144, obj_size: 326, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-4 on core 4(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-4 on core 4(size: 5242879, obj_size: 312, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-4 on core 4(size: 5242879, obj_size: 200, priv: 0 cache: 0)
USER1: Found 2 Ethernet ports to start.
USER1: [port_setup_port()] Initializing Ethernet port 0.
PMD: mlx5_ethdev.c:547: dev_configure(): 0x960300: TX queues number update: 0 -> 1
PMD: mlx5_ethdev.c:557: dev_configure(): 0x960300: RX queues number update: 0 -> 1
PMD: mlx5_rxq.c:976: rxq_ctrl_setup(): 0x960300: too many SGEs (32) needed to handle requested maximum packet size 65536
USER1: ERROR: Failed rte_eth_rx_queue_setup(0, 0, ...), returned Value too large for defined data type(75)!
EAL: Error - exiting with code: 1
Cause: ERROR: Failed initializing the Ethernets ports!
Hi Dumitru,
There is a bug in calculating TCP bytes sent where l2 and l3 header length is also being added into tcp bytes sent(control and data).
Regards,
Chandrasheker M.
In tsm_SF_fin_wait_I() you send a FIN packet and increment tcb_snd.nxt.
So when FIN is re-transmitted later (by RTO) all new FINs have seq number greater by one
then initial FIN.
So for the peer system, if first FIN was lost it looks like there is still a 1B gap in the receive sequence,
and session might not be closed properly.
Below is a patch I used to fix the issue.
It is not a proper fix, just a quick and dirty one to cure situation.
Thanks
Konstantin
diff --git a/src/tpg_tcp_sm.c b/src/tpg_tcp_sm.c
index ee069eb..d968a1c 100644
--- a/src/tpg_tcp_sm.c
+++ b/src/tpg_tcp_sm.c
@@ -1606,7 +1606,9 @@ static int tsm_SF_fin_wait_I(tcp_control_block_t tcb, tcpEvent_t event,
/ TODO: actually resend data here. */
/* Resend the FIN. */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
tsm_schedule_retransmission(tcb);
break;
case TE_ORPHAN_TIMEOUT:
@@ -1923,7 +1925,9 @@ static int tsm_SF_last_ack(tcp_control_block_t *tcb, tcpEvent_t event,
/* Resend the packet that got us here.. */
/* Send <SEQ=SND.NXT><ACK=RCV.NXT><CTL=FIN,ACK> */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_ACK_FLAG | TCP_FIN_FLAG);
tcb->tcb_snd.nxt++;
/* TODO: anything else to resend here?. */
tsm_schedule_retransmission(tcb);
break;
@@ -2084,7 +2088,9 @@ static int tsm_SF_closing(tcp_control_block_t *tcb, tcpEvent_t event,
}
/* TODO: actually resend data here (resend FIN|ACK??). */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
tsm_schedule_retransmission(tcb);
break;
case TE_ORPHAN_TIMEOUT:
@@ -2352,7 +2358,9 @@ static int tsm_SF_close_wait(tcp_control_block_t *tcb, tcpEvent_t event,
tsm_retrans_data(tcb);
/* Resend <SEQ=SND.NXT><ACK=RCV.NXT><CTL=FIN,ACK> */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
break;
case TE_ORPHAN_TIMEOUT:
case TE_FIN_TIMEOUT:
$ cat patches/p1
diff --git a/src/tpg_tcp_sm.c b/src/tpg_tcp_sm.c
index ee069eb..d968a1c 100644
--- a/src/tpg_tcp_sm.c
+++ b/src/tpg_tcp_sm.c
@@ -1606,7 +1606,9 @@ static int tsm_SF_fin_wait_I(tcp_control_block_t tcb, tcpEvent_t event,
/ TODO: actually resend data here. */
/* Resend the FIN. */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
tsm_schedule_retransmission(tcb);
break;
case TE_ORPHAN_TIMEOUT:
@@ -1923,7 +1925,9 @@ static int tsm_SF_last_ack(tcp_control_block_t *tcb, tcpEvent_t event,
/* Resend the packet that got us here.. */
/* Send <SEQ=SND.NXT><ACK=RCV.NXT><CTL=FIN,ACK> */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_ACK_FLAG | TCP_FIN_FLAG);
tcb->tcb_snd.nxt++;
/* TODO: anything else to resend here?. */
tsm_schedule_retransmission(tcb);
break;
@@ -2084,7 +2088,9 @@ static int tsm_SF_closing(tcp_control_block_t *tcb, tcpEvent_t event,
}
/* TODO: actually resend data here (resend FIN|ACK??). */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
tsm_schedule_retransmission(tcb);
break;
case TE_ORPHAN_TIMEOUT:
@@ -2352,7 +2358,9 @@ static int tsm_SF_close_wait(tcp_control_block_t *tcb, tcpEvent_t event,
tsm_retrans_data(tcb);
/* Resend <SEQ=SND.NXT><ACK=RCV.NXT><CTL=FIN,ACK> */
tcb->tcb_snd.nxt--;
tcp_send_ctrl_pkt(tcb, TCP_FIN_FLAG | TCP_ACK_FLAG);
tcb->tcb_snd.nxt++;
break;
case TE_ORPHAN_TIMEOUT:
case TE_FIN_TIMEOUT:
Hi Dumitru,
We need latency stats support on WARP17, it can help us often identifying the problem when application/network performance is slow and low. Thanks in advance.
Thanks,
Rakesh mishra
Please check the infinite spelling in test_6_http_40M_sessions.cfg file line number 80.
set tests timeouts port 0 test-case-id 0 uptime infinte
set tests timeouts port 0 test-case-id 0 downtime infinite
Hi Dumitru,
We are currently installing WARP17 on our VM machine.
The machine has 4 cores and 16 GB RAM.
We have installed the DPDK and WARP17.
But when I try to run warp17 application , we hit following DPDK error.
We have already merged the patch http://dpdk.org/dev/patchwork/patch/945/
root@mxscb480d-*****1:~/WARP17/dpdk-16.11# /root/WARP17/warp17-1.4/build/warp17 -c f -n 64 -m 8192 -- --tcb-pool-sz 5000 —qmap-default max-c
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: probe driver: 15ad:7b0 net_vmxnet3
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL: probe driver: 15ad:7b0 net_vmxnet3
warp17> USER1: Creating mempool global_pkt_mbuf-2 on core 2(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-2 on core 2(size: 262144, obj_size: 310, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-2 on core 2(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-2 on core 2(size: 2560000, obj_size: 280, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-2 on core 2(size: 5242879, obj_size: 168, priv: 0 cache: 0)
USER1: Creating mempool global_pkt_mbuf-3 on core 3(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-3 on core 3(size: 262144, obj_size: 310, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-3 on core 3(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-3 on core 3(size: 2560000, obj_size: 280, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-3 on core 3(size: 5242879, obj_size: 168, priv: 0 cache: 0)
USER1: Found 2 Ethernet ports to start.
USER1: [port_setup_port()] Initializing Ethernet port 0.
USER1: Ethernet port 0 initialized with MAC address 00:50:56:84:32:86
USER1: [port_setup_port()] Initializing Ethernet port 1.
USER1: ERROR: Failed rte_eth_tx_queue_setup(1, 0, ...), returned Invalid argument(22)!
EAL: Error - exiting with code: 1
Cause: ERROR: Failed initializing the Ethernets ports!
root@mxscb480d-*****1:~/WARP17/dpdk-16.11#
DPDK :
root@mxscb480d-*****1:~/WARP17/dpdk-16.11# /usr/local/share/dpdk/tools/dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' drv=igb_uio unused=
0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=
Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3 unused=igb_uio
Other network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other crypto devices
====================
<none>
root@mxscb480d-*****1:~/WARP17/dpdk-16.11#
Please help when you find time.
Thanks a lot for what you are doing to the community :)
Thanks,
Rakesh Mishra
A new option should be added to UDP to support sending to multicast destinations. Right now there's no check on destination IP so a multicast IP is acceptable but if the IP is multicast we don't map the lower IP bits in the Ethernet Multicast address but actually try to ARP it.
When configuring HTTP as an application for a test-case the user should be allowed to specify different content types.
A new configuration option content-type
should be available to the user.
Hello WARP Team,
Setup Details:
I have setup warp17 (with DPDK 16.04) on one 40G port with 11 cores for client and 11 core server on different machines.I have 1 HTTP Client connection towards 1 server, and my request size is 13000 Bytes
Working Scenario:
If I set send-rate in the client configuration file as 500( say any value less than 1000 or any value greater than 1000, but should be multiple of 1000). Data Sends/sec calculated perfectly fine i.e 500 showing on Web-UI at client side in this case(as expected).
Problematic Scenario:
If I set send-rate in the client configuration as 1200. I observed that Data Send/sec is showing as 2000 on Warp-UI at client side. Although the expected Data Send/sec should be 1200 itself. The existing calculation for Data Send/sec looks wrong. The Calculated Data Send/sec is showing wrong for all values of send-rate configured as greater than 1000 but not multiple of 1000.( say 1200, 1700, 2200 etc).
Transaction per second ( One HTTP request and response) calculated to ~200. Transaction per second at (HTTP application level) calculated as:
( HTTP response count at t2 – HTTP response at t1) /( t2 –t1) where t2 and t1 are time in seconds with single client connection.
I would request Team to provide fix for this bug in coming git commits(or release).
I have pasted the client and server configuration file and command issued to start WARP.
Configuration Files Used:
test_client.cfg
add tests ip_version 4 l3_intf port 0 ip 10.0.0.1 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.0.253
add tests client ip_version 4 tcp port 0 test-case-id 0 src 10.0.0.1 10.0.0.1 sport 10001 10001 dest 10.0.0.253 10.0.0.253 dport 6001 6001
set tests timeouts port 0 test-case-id 0 init 0
set tests timeouts port 0 test-case-id 0 uptime 30
set tests timeouts port 0 test-case-id 0 downtime 15
set tests criteria port 0 test-case-id 0 run-time 3600
set tests rate port 0 test-case-id 0 send 1200
set tests client http port 0 test-case-id 0 GET google.com /index.html req-size 13000
start tests port 0
show tests ui
Command triggered to run Warp at client side:
./build/warp17 -c 0x103 -n 4 -m 10240 -w 0000:05:00.0 -- --qmap 0.0x100 --tcb-pool-sz 1024 --ucb-pool-sz 1024 --cmd-file ./test_client.cfg
test_server.cfg:
add tests ip_version 4 l3_intf port 0 ip 10.0.0.253 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.0.253
add tests server ip_version 4 tcp port 0 test-case-id 0 src 10.0.0.253 10.0.0.253 sport 6001 6001
set tests server http port 0 test-case-id 0 200-OK resp-size 8000
start tests port 0
show tests ui
Command to run Warp at Server side:
./build/warp17 -c 0x7FF3 -n 4 -m 10240 -w 0000:05:00.0 -- --qmap 0.0x7FF0 --tcb-pool-sz 1024 --ucb-pool-sz 1 --cmd-file ./test_server.cfg
The link for queries posted on Google Group can be found at link:
[https://groups.google.com/d/msg/warp17/9FMbgS3L5Cg/9A2fS43lBAAJ]
Best Regards,
Yogesh Kumar
We tried running WARP17 with cmd-file tests_3_http_multiple.cfg as input to start two test cases in parallel but when we see port stats using "show tests ui" it is showing that TcId 0 State as running and TcId 1 state as IDLE.
Using the following command line:
./build/warp17 --lcores '0,1,(14-27)@(2-15),(42-55)@(16-29)' -m 65536 -n 4 -w 0000:81:00.0 -w 0000:81:00.1 -- --qmap-default max-q --tcb-pool-sz 30000
And the following config that should create 15M TCP sessions on a back to back setup:
add tests l3_intf port 0 ip 30.3.0.2 mask 255.255.0.0
add tests l3_gw port 0 gw 30.3.0.1
add tests l3_intf port 1 ip 30.3.0.1 mask 255.255.0.0
add tests l3_gw port 1 gw 30.3.0.2
add tests client tcp port 0 test-case-id 0 src 60.0.0.1 60.4.147.224 sport 1024 1073 dest 30.3.0.1 30.3.0.1 dport 80 80
set tests async port 0 test-case-id 0
set tests noasync port 0 test-case-id 0
set tests rate port 0 test-case-id 0 open 250000
set tests rate port 0 test-case-id 0 close 0
set tests rate port 0 test-case-id 0 send 250000
set tests timeouts port 0 test-case-id 0 init 0
set tests timeouts port 0 test-case-id 0 uptime infinite
set tests client http port 0 test-case-id 0 GET google.com /index.html req-size 1
add tests server tcp port 1 test-case-id 0 src 30.3.0.1 30.3.0.1 sport 80 80
set tests criteria port 1 test-case-id 0 run-time 120
set tests server http port 1 test-case-id 0 200-OK resp-size 1
start tests port 1
start tests port 0
We can notice Warp17 tries to create too many TCP sessions:
warp17> show tcp statistics
Port 0 TCP statistics:
Received Packets : 62398082
Received Bytes : 2744785948
Sent Ctrl Packets : 56671470
Sent Ctrl Bytes : 3060259380
Sent Data Packets : 23759577
Sent Data Bytes : 3516417396
Malloced TCBs : 0
Freed TCBs : 0
Not found TCBs : 0
TCB alloc errors : 0
Invalid checksum : 0
Small mbuf fragment : 0
TCP hdr to small : 0
Ctrl Failed Packets : 0
DATA Failed Packets : 0
DATA Clone Failed : 0
Reserved bit set : 0
Port 1 TCP statistics:
Received Packets : 80367332
Received Bytes : 3840725822
Sent Ctrl Packets : 38639063
Sent Ctrl Bytes : 2086509402
Sent Data Packets : 23759213
Sent Data Bytes : 2779827921
Malloced TCBs : 14870912
Freed TCBs : 0
Not found TCBs : 0
TCB alloc errors : 3098784
Invalid checksum : 0
Small mbuf fragment : 0
TCP hdr to small : 0
Ctrl Failed Packets : 0
DATA Failed Packets : 0
DATA Clone Failed : 0
Reserved bit set : 0
Using max-c
works fine:
./build/warp17 --lcores '0,1,(14-27)@(2-15),(42-55)@(16-29)' -m 65536 -n 4 -w 0000:81:00.0 -w 0000:81:00.1 -- --qmap-default max-c --tcb-pool-sz 30000
On start, warp17 version 1.4 takes all available hugepages.
On exit or kill, it does not free them. At least when stoped via 'exit' command on console or with 'pkill warp17'.
This gluttony makes it hard to use warp17 in server/multiuser environment.
For now there's no counter for different types of HTTP packets (requests or responses). However, on the client side, all the sent requests are of the type configured and on the server side all the sent responses are of the type configured.
It would be nice to have specific counters for reqs and resp of application traffic (http).
There is no utility function or CLI command to clear statistics manually if we needed after test case end.
warp17 v1.4
dpdk 16.11
hugepage 1GB x 32
Command:
./build/warp17 -c 0000fff0 -n 8 -- --cmd-file cmd_http.cfg
The cmd_http.cfg is copy from examples/test_6_http_40M_sessions.cfg.
All going well. Then I stopped tests and changed some params.
stop tests port 0
set tests rate port 0 test-case-id 0 send infinite
start tests port 0
USER1 [6:test_case_init_tcp_clients()] Failed to allocate TCB.
USER1 [6:test_case_init_tcp_clients()] Failed to allocate TCB.
USER1 [6:test_case_init_tcp_clients()] Failed to allocate TCB.
USER1 [6:test_case_init_tcp_clients()] Failed to allocate TCB.
USER1 [6:test_case_init_tcp_clients()] Failed to allocate TCB.
Warp17 version: v1.5
DPDK version: 16.11.2
NIC:
$RTE_SDK/tools/dpdk-devbind.py -s | grep 0000:83:00.0
0000:83:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
lspci -vvv | grep -i ether | grep 83:00.0
83:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
In the Warp17 CLI:
warp17> show port info
Queues max rx tx rss offloads
Port Driver MTU PCI address Soc rx tx offload offload IPv4 IPv6 ex 2
---- ---------------- ----- ------------ --- ----- ----- ------- ---------- ------+---------+--
0 net_ixgbe 1500 0000:81:00.0 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
1 net_ixgbe 1500 0000:81:00.1 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
2 net_ixgbe 1500 0000:82:00.1 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
3 net_ixgbe 1500 0000:83:00.0 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
warp17> set tests mtu port 3 2022
Port 3 MTU updated!
warp17> show port info
Queues max rx tx rss offloads
Port Driver MTU PCI address Soc rx tx offload offload IPv4 IPv6 ex 2
---- ---------------- ----- ------------ --- ----- ----- ------- ---------- ------+---------+--
0 net_ixgbe 1500 0000:81:00.0 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
1 net_ixgbe 1500 0000:81:00.1 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
2 net_ixgbe 1500 0000:82:00.1 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
3 net_ixgbe 2022 0000:83:00.0 1 128 64 v4utl- v4utst--- 4-tu-- 6-tu--ETU -
warp17> set tests mtu port 3 2023
USER1: WARNING: Port 3 Failed to set MTU in HW: Invalid argument(22)
Port 3 MTU updated!
Allow users to configure multiple application types on the same test case.
Distribute the application types between sessions in that given test case based on user provided weights (or predefined default profiles).
Use the dev_info
values returned by rte_eth_dev_info_get
when configuring Ethernet ports (port_setup_port
). We currently check only for max rx/tx queues but there are others that we should also check and adjust (e.g., max_rx_pktlen, hw_offloads).
./warp17-master/src/tpg_test_mgmt_api.c line number 594. Re check the logic of FOR loop for detecting duplicate IP.
Please find the log:
warp17> add tests l3_intf port 0 ip 10.0.0.253 mask 255.255.255.0
L3 interface successfully added.
warp17> add tests l3_intf port 0 ip 10.0.0.253 mask 255.255.255.0
L3 interface successfully added.
warp17> add tests l3_intf port 0 ip 10.0.0.253 mask 255.255.255.0
L3 interface successfully added.
warp17> add tests l3_intf port 0 ip 10.0.0.253 mask 255.255.255.0
L3 interface successfully added.
In http stats it will be good if we can change http_req_cnt and http_resp_cnt from uint32 to uint64 to avoid quick integer overflow in longer runs.
Hi Dumitru and Matteo,
Raising this issue for tracking..
We are trying to send the WARP17 UDP traffic from AWS (Amazon Web Service) VM machines to VSRX, But we don’t see any state full traffic established in both UDP and TCP case.
We have conducted the test for both TCP and UDP for the below topology.
Topo 1 :
abdulh-ubuntu1(0000:00:03.0 = ens3) ----------- abdulh-vsrx------------- abdulh-ubuntu1(0000:00:05.0 = ens5)
Topo 2 :
abdulh-ubuntu1(0000:00:03.0 = ens3) ----------- abdulh-vsrx------------- abdulh-ubuntu2(0000:00:03.0 = ens3)
WARP17 release info :
warp17> show version
warp17 v0.13 @11:43:12, Aug 23 2017
warp17>
Ethtool info and OS
root@10:/home/regress# ethtool -i ens6
driver: ixgbevf
version: 2.12.1-k
firmware-version:
expansion-rom-version:
bus-info: 0000:00:06.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
root@10:/home/regress#
root@10:/home/regress# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@10:/home/regress#
root@10:/home/regress# dmidecode | grep -i "Product"
Product Name: HVM domU
root@10:/home/regress#
/home/ubuntu/warp17-dev-common/build/warp17 --lcores 0@0,1@0,2@1,3@2 -m 30000 -n 1 -- --ucb-pool-sz 20000
add tests l3_intf port 0 ip 10.0.1.128 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.1.250
add tests l3_intf port 1 ip 10.0.3.124 mask 255.255.255.0
add tests l3_gw port 1 gw 10.0.3.212
add tests client udp port 0 test-case-id 0 src 10.0.1.128 10.0.1.128 sport 1024 1043 dest 10.0.3.124 10.0.3.124 dport 80 89
set tests rate port 0 test-case-id 0 open 5000
set tests rate port 0 test-case-id 0 close 0
set tests rate port 0 test-case-id 0 send 10000
set tests timeouts port 0 test-case-id 0 init 0
set tests timeouts port 0 test-case-id 0 uptime infinite
set tests client raw port 0 test-case-id 0 data-req-plen 300 data-resp-plen 300
set tests criteria port 0 test-case-id 0 run-time 1200
add tests server udp port 1 test-case-id 0 src 10.0.3.124 10.0.3.124 sport 80 89
set tests criteria port 1 test-case-id 0 run-time 1200
set tests server raw port 1 test-case-id 0 data-req-plen 300 data-resp-plen 300
start tests port 1
start tests port 0
warp17> show tests ui
warp17>
warp17> show udp statistics
Port 0 UDP statistics:
Received Packets : 200
Received Bytes : 61600
Sent Packets : 200
Sent Bytes : 60000
Malloced UCBs : 0
Freed UCBs : 0
Not found UCBs : 200
UCB alloc errors : 0
Invalid checksum : 0
Small mbuf fragment : 0
Failed Packets : 0
Port 1 UDP statistics:
Received Packets : 200
Received Bytes : 61600
Sent Packets : 200
Sent Bytes : 60000
Malloced UCBs : 210
Freed UCBs : 0
Not found UCBs : 0
UCB alloc errors : 0
Invalid checksum : 0
Small mbuf fragment : 0
Failed Packets : 0
warp17> show link
rate [Fixed STRING]: show link rate
warp17> show link rate
Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 0.00% RX: 0.00%
Port 1: link UP, speed 10Gbps, duplex full(manual), TX: 0.00% RX: 0.00%
warp17> show memory statistics
Mempool MBUF RX:
Total Allocated: 2032
Total Free : 784400
Mempool MBUF TX HDR:
Total Allocated: 400
Total Free : 523888
Mempool MBUF CLONE:
Total Allocated: 400
Total Free : 786032
Mempool TCB:
Total Allocated: 0
Total Free : 10485758
Mempool UCB:
Total Allocated: 410
Total Free : 20479590
warp17> show port map
Port 0[socket: 0]:
Core 2[socket:0] (Tx: 0, Rx: 0)
Port 1[socket: 0]:
Core 3[socket:0] (Tx: 0, Rx: 0)
warp17> show msg statistics
MSG statistics:
Messages rcvd : 593225
Messages sent : 593221
Messages polled : 44296662870
Messages errors : 0
Messages proc err : 0
Messages allocated : 4
Messages alloc err : 0
Messages freed : 4
warp17> show port statistics
Port 0 software statistics:
Received packets : 211
Received bytes : 69860
Sent packets : 209
Sent bytes : 68778
Sent failures : 0
RX Ring If failures : 0
Simulated failures : 0
Port 1 software statistics:
Received packets : 210
Received bytes : 69800
Sent packets : 210
Sent bytes : 68820
Sent failures : 0
RX Ring If failures : 0
Simulated failures : 0
warp17>
regress@vsrx1# run show security flow session
Session ID: 145622, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1025 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1025;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145623, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1027 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1027;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145624, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1024 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1024;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145625, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1026 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1026;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145626, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1028 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1028;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145627, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1030 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1030;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145628, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1032 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1032;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145629, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1029 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
Out: 10.0.3.124/80 --> 10.0.1.128/1029;udp, Conn Tag: 0x0, If: ge-0/0/2.0, Pkts: 1, Bytes: 328,
Session ID: 145630, Policy name: default-permit/4, Timeout: 58, Valid
In: 10.0.1.128/1031 --> 10.0.3.124/80;udp, Conn Tag: 0x0, If: ge-0/0/0.0, Pkts: 1, Bytes: 328,
---(more)---[abort]
[edit]
regress@vsrx1# run show security flow statistics
Current sessions: 200
Packets forwarded: 236101
Packets dropped: 514
Fragment packets: 0
[edit]
regress@vsrx1# run show security flow statistics
Current sessions: 200
Packets forwarded: 236105
Packets dropped: 518
Fragment packets: 0
[edit]
Thanks,
Rakesh Mishra
Currently there's no way to configure a test case that would stop after the application received a given number of responses (or after a given number of requests). E.g., I am trying to form a testcase of sending only one GET request and after the response is response is received the test will stop immediately.
It would definitely be a nice feature to have.
Should be useful to have an arg command and a cli/rpc command that return version in order to better support scripts
The current behavior of stop tests port X
is to forcefully stop any running test cases (and sessions) on port X
. It would be nice if WARP17 would try to stop sessions gracefully (especially stateful sessions like TCP) in order to allow any stateful devices in the network to flush their session states.
A possibility would be to extend the current command to:
stop tests port X timeout T
The timeout
argument would be optional and the behavior would be:
If timeout
is not specified then try to stop sessions gracefully and forcefully stop them after a fixed default timeout (e.g., 2 seconds).
If timeout
is specified:
timeout
is 0 forcefully stop all sessionstimeout
is non zero then try to stop sessions gracefully. After timeout
expires, fall back to forcefully stopping sessions.AVX is present but not AVX2? I deployed the ova from the website
wrong forum, will post into google group.
With simple .cmd
add tests l3_intf port 0 ip <myIP> mask <myMASK>
add tests client tcp port 0 test-case-id 0 src 192.168.5.21 192.168.5.21 sport 10000 10000 dest 10.0.0.253 10.0.0.253 dport 6000 6000
set tests criteria port 0 test-case-id 0 clients-up 100
set tests client raw port 0 test-case-id 0 data-req-plen 100 data-resp-plen 200
start tests port 0
warp17 says USER1: Port 0, Test Case 0 "PASSED"!
. I need to run automated nightly tests and therefore want warp17 to exit at this point. Unfortunately I don't know a good way of doing this.
How do I tell warp17 to terminate after all tests are done?
A new configuration should be available for IPv4 socket options which should allow the user to set the values used for DSCP/TOS/ECN in outgoing packets.
In the raw application implementation if ra_resp_size is 0 we stay in the same state and never notify the test engine to move the session at the end of the queue.
Command cli hang, no response for key press.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.