facebookincubator / katran Goto Github PK
View Code? Open in Web Editor NEWA high performance layer 4 load balancer
License: GNU General Public License v2.0
A high performance layer 4 load balancer
License: GNU General Public License v2.0
Since connection tracking is implemented by per cpu lru maps, is there affinity of incoming packets to individual cpus? In other words, if a source ip and port pair is assigned a real on a cpu, will another cpu process a packet from the same source ip and port later and assign a different real?
lru_maps_mapping is BPF_MAP_TYPE_ARRAY_OF_MAPS containing a BPF_MAP_TYPE_LRU_HASH map per cpu. Will BPF_MAP_TYPE_LRU_HASH or BPF_MAP_TYPE_LRU_PERCPU_HASH with the BPF_F_NO_COMMON_LRU flag achieve the same goal?
is 18.04 still supported? what the official position of FB folks about it?
I0303 07:28:53.732266 28170 katran_tester.cpp:378] Testing counter's sanity. Printing on errors only
I0303 07:28:53.732388 28170 katran_tester.cpp:418] incorrect stats about katran library internals: number of failed bpf syscalls is non zero
E0303 07:28:53.732395 28170 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0303 07:28:53.732400 28170 KatranLb.cpp:1350] Error gathering map stats for vip_map
I0303 07:28:53.732403 28170 katran_tester.cpp:356] vip_map: current size is incorrect: 0
E0303 07:28:53.732409 28170 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0303 07:28:53.732411 28170 KatranLb.cpp:1350] Error gathering map stats for reals
I0303 07:28:53.732414 28170 katran_tester.cpp:356] reals: current size is incorrect: 0
[ 95%] Building CXX object example/CMakeFiles/simple_katran_client.dir/client/Main.cpp.o
collect2: error: ld returned 1 exit status
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:134: recipe for target 'example_grpc/katran_server_grpc' failed
make[2]: *** [example_grpc/katran_server_grpc] Error 1
CMakeFiles/Makefile2:1242: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 97%] Linking CXX executable simple_katran_client
collect2: error: ld returned 1 exit status
example/CMakeFiles/simple_katran_client.dir/build.make:147: recipe for target 'example/simple_katran_client' failed
make[2]: *** [example/simple_katran_client] Error 1
CMakeFiles/Makefile2:858: recipe for target 'example/CMakeFiles/simple_katran_client.dir/all' failed
make[1]: *** [example/CMakeFiles/simple_katran_client.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
Good day, I was wondering if you could add some benchmarks that includes throughput and requests a second (or packets per second) at various level of concurrency with a single node. Bonus if the benchmark compares it's results to other layer 4 load balancers to get relative speed differences
I've been playing around with the katran library for a few days now and I was curious about the following things:
The repo (once built) includes fully functioning example servers. Is there a reason these haven't been moved from "examples" to just top level apps provided as a full L4 load balancing solutions within this repo? (By that I mean something like moving example_grpc to just be the standard provided katran server.)
Since these are only examples right now, is it advised to only take these and use them in a library form / for testing and users should always be building their own full server using the katran lib?
I made a simple prometheus metrics exporter for the example_grpc server, is this something that you would want added to this repo?
Do you have any examples of how to create a pcap after you have built with KATRAN_INTROSPECTION enabled?
I am following the example of https://github.com/facebookincubator/katran/blob/master/EXAMPLE.md and under Configuration of forwarding plane section step 4:
VIP must be configured on the real
$ sudo ip a a 10.200.200.1/32 dev lo
Should the VIP is on the server instead of the real?
I tested the performance of katran, there is no difference between driver mode and generic mode, and they are all lower than lvs.
The configuration of lb machine:
$ nproc
56
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
$ cpu_layout.py
======================================================================
Core and Socket Information (as reported by '/sys/devices/system/cpu')
======================================================================
cores = [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14]
sockets = [0, 1]
Socket 0 Socket 1
-------- --------
Core 0 [0, 28] [14, 42]
Core 1 [1, 29] [15, 43]
Core 2 [2, 30] [16, 44]
Core 3 [3, 31] [17, 45]
Core 4 [4, 32] [18, 46]
Core 5 [5, 33] [19, 47]
Core 6 [6, 34] [20, 48]
Core 8 [7, 35] [21, 49]
Core 9 [8, 36] [22, 50]
Core 10 [9, 37] [23, 51]
Core 11 [10, 38] [24, 52]
Core 12 [11, 39] [25, 53]
Core 13 [12, 40] [26, 54]
Core 14 [13, 41] [27, 55]
$ lspci
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: Huawei Technologies Co., Ltd. Device d110
Physical Slot: 0-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 26
NUMA node: 0
Region 0: Memory at 92200000 (64-bit, prefetchable) [size=2M]
Region 2: I/O ports at 2020 [size=32]
Region 4: Memory at 92404000 (64-bit, prefetchable) [size=16K]
Expansion ROM at 92480000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: ixgbe
Kernel modules: ixgbe
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
Subsystem: Huawei Technologies Co., Ltd. Device d110
Physical Slot: 0-2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin B routed to IRQ 152
NUMA node: 0
Region 0: Memory at 92000000 (64-bit, prefetchable) [size=2M]
Region 2: I/O ports at 2000 [size=32]
Region 4: Memory at 92400000 (64-bit, prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: ixgbe
Kernel modules: ixgbe
$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 8
$ ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 8
# irqbalance is disabled.
# smp affinity of eth0: (0-7)
95 000000,00000001
96 000000,00000002
97 000000,00000004
98 000000,00000008
99 000000,00000010
100 000000,00000020
101 000000,00000040
109 000000,00000080
# smp affinity of eth1: (8-15)
102 000000,00000100
103 000000,00000200
104 000000,00000400
105 000000,00000800
106 000000,00001000
107 000000,00002000
108 000000,00004000
160 000000,00008000
configured service:
$ ./katran_goclient -l -rpc_port 50052
2018/12/29 20:37:37 vips len 1
VIP: 10.210.9.103 Port: 80 Protocol: tcp
Vip's flags:
->172.30.4.8 weight: 1
->172.30.4.7 weight: 1
->172.30.4.4 weight: 1
->10.210.9.100 weight: 1
->10.210.9.102 weight: 1
$ ./katran_goclient -l -rpc_port 50051
2018/12/29 20:37:38 vips len 1
VIP: 10.210.9.103 Port: 80 Protocol: tcp
Vip's flags:
->172.30.4.8 weight: 1
->172.30.4.7 weight: 1
->172.30.4.4 weight: 1
->10.210.9.100 weight: 1
->10.210.9.102 weight: 1
two clients ran wrk
.
summary stats of lb:
$ ./katran_goclient -s -sum -rpc_port 50051
summary: 1281055 pkts/sec 98705851 bytes/sec
summary: 1176094 pkts/sec 90136858 bytes/sec
summary: 1278174 pkts/sec 97861917 bytes/sec
summary: 1279117 pkts/sec 97803088 bytes/sec
$ ./katran_goclient -s -sum -rpc_port 50052
summary: 856653 pkts/sec 65913453 bytes/sec
summary: 842773 pkts/sec 64840720 bytes/sec
summary: 847563 pkts/sec 65222415 bytes/sec
summary: 880912 pkts/sec 67892596 bytes/sec
cpu usage:
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi,100.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni, 6.4 id, 0.0 wa, 0.0 hi, 93.6 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 9.2 id, 0.0 wa, 0.0 hi, 90.8 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 9.8 id, 0.0 wa, 0.0 hi, 90.2 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 9.2 id, 0.0 wa, 0.0 hi, 90.8 si, 0.0 st
%Cpu12 : 0.0 us, 0.0 sy, 0.0 ni, 19.0 id, 0.0 wa, 0.0 hi, 81.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni, 27.3 id, 0.0 wa, 0.0 hi, 72.7 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni, 40.5 id, 0.0 wa, 0.0 hi, 59.5 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 38.5 id, 0.0 wa, 0.0 hi, 61.5 si, 0.0 st
client results:
$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
32 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 112.87ms 261.00ms 1.86s 89.40%
Req/Sec 29.17k 3.76k 65.05k 77.06%
61118879 requests in 1.10m, 9.84GB read
Socket errors: connect 0, read 3, write 380, timeout 38989
Requests/sec: 928741.35
Transfer/sec: 153.08MB
$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
32 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 106.98ms 261.37ms 1.99s 90.09%
Req/Sec 38.68k 9.50k 114.19k 86.95%
90094314 requests in 1.22m, 14.50GB read
Socket errors: connect 0, read 3, write 112, timeout 37468
Requests/sec: 1231554.03
Transfer/sec: 202.99MB
Same machine and same configuration with lvs:
$ sudo ipvsadm -l -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.210.9.103:80 wrr
-> 172.30.4.4:80 Tunnel 1 0 0
-> 172.30.4.7:80 Tunnel 1 0 0
-> 172.30.4.8:80 Tunnel 1 0 0
-> 10.210.9.100:80 Tunnel 1 0 0
-> 10.210.9.102:80 Tunnel 1 0 0
cpu usage:
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 5.2 id, 0.0 wa, 0.0 hi, 94.8 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 93.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 5.2 id, 0.0 wa, 0.0 hi, 94.8 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 5.7 id, 0.0 wa, 0.0 hi, 94.3 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 94.7 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 6.3 id, 0.0 wa, 0.0 hi, 93.7 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 7.8 id, 0.0 wa, 0.0 hi, 92.2 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 3.1 id, 0.0 wa, 0.0 hi, 96.9 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni, 36.8 id, 0.0 wa, 0.0 hi, 63.2 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 50.4 id, 0.0 wa, 0.0 hi, 49.6 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 50.7 id, 0.0 wa, 0.0 hi, 49.3 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 61.4 id, 0.0 wa, 0.0 hi, 38.6 si, 0.0 st
%Cpu12 : 0.0 us, 0.0 sy, 0.0 ni, 57.8 id, 0.0 wa, 0.0 hi, 42.2 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni, 63.8 id, 0.0 wa, 0.0 hi, 36.2 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni, 63.5 id, 0.0 wa, 0.0 hi, 36.5 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 35.6 id, 0.0 wa, 0.0 hi, 64.4 si, 0.0 st
client results:
$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
32 threads and 10000 connections
^C Thread Stats Avg Stdev Max +/- Stdev
Latency 103.93ms 241.20ms 1.99s 89.54%
Req/Sec 63.61k 12.08k 159.62k 82.01%
113727680 requests in 0.94m, 18.31GB read
Socket errors: connect 0, read 0, write 30, timeout 11326
Requests/sec: 2021869.63
Transfer/sec: 333.26MB
$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
32 threads and 10000 connections
^C Thread Stats Avg Stdev Max +/- Stdev
Latency 103.67ms 241.55ms 1.98s 89.59%
Req/Sec 65.79k 10.29k 143.66k 82.61%
119779234 requests in 0.95m, 19.28GB read
Socket errors: connect 0, read 1, write 44, timeout 12137
Requests/sec: 2093134.23
Transfer/sec: 344.92MB
Is this normal? Is there any way to increase the performance?
Thanks.
HI
I got some ERRORs.
any suggestion?
~/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/bpfprog/bpf/balancer_kern.o -default_mac 52:54:00:12:35:02 -forwarding_cores=0 -healthchecker_prog ./deps/bpfprog/bpf/healthchecking_ipip.o -intf=eth0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000
I0423 05:13:18.267051 8437 KatranGrpcService.cpp:67] Starting Katran
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error finalizing and loading .BTF into kernel: -22. Ignored and continue.
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error finalizing and loading .BTF into kernel: -22. Ignored and continue.
E0423 05:13:18.308995 8437 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0423 05:13:18.310480 8437 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
Server listening on 0.0.0.0:50051
^CI0423 05:13:20.733433 8438 GrpcSignalHandler.cpp:37] Signal: 2, stopping service in 10000 milliseconds.
I0423 05:13:30.743686 8438 GrpcSignalHandler.cpp:42] Stopping Katran!
Hi,
I am trying to build Katran with the latest commit 23add38cf9839b2bacafbf7330767853f69c3810
and Ubuntu 18.04, but it is failed due to the linkage error below. This is also the same issue Travis CI build has encountered.
Scanning dependencies of target katran_server_grpc
[ 98%] Building CXX object example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o
[100%] Linking CXX executable katran_server_grpc
../../deps/grpc/build/libgpr.a(host_port.cc.o): In functiongrpc_core::(anonymous namespace)::DoSplitHostPort(absl::string_view, absl::string_view*, absl::string_view*, bool*)': /home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:77: undefined reference to
absl::string_view::find(char, unsigned long) const'
/home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:79: undefined reference toabsl::string_view::find(char, unsigned long) const' /home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:53: undefined reference to
absl::string_view::find(char, unsigned long) const'
/home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:70: undefined reference toabsl::string_view::find(char, unsigned long) const' ../../deps/grpc/build/libgpr.a(host_port.cc.o): In function
grpc_core::(anonymous namespace)::DoSplitHostPort(absl::string_view, absl::string_view*, absl::string_view*, bool*)':
/home/ubuntu/workspace/katran/_build/deps/grpc/third_party/abseil-cpp/absl/strings/string_view.h:382: undefined reference to `absl::base_internal::ThrowStdOutOfRange(char const*)'
collect2: error: ld returned 1 exit status
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:154: recipe for target 'example_grpc/katran_server_grpc' failed
make[2]: *** [example_grpc/katran_server_grpc] Error 1
CMakeFiles/Makefile2:2101: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Hi developers. I have some experience with traditional LVS technology but just started to learn about this library, so please bear with me if my questions are not on the right track.
From the documentation I learned katran's forwarding mode is a bit like LVS' TUN mode, which needs L4 machines to wrap packets to IPIP protocol and requires the L7 have public network reachability. Will tunneling be a huge computational overhead for the L4 machines?
I think in most of deployment scenarios, L7 and L4 are just in the same local network and doesn't need IPIP to work. Why not support something like LVS's
mode?
What is the design considerations of using tunnel? Thanks.
uname -a
Linux kruztw 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
gcc --version
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE
In bpf files, lru seems the cache of get_packet_dst result. How can lru handle packets when dst real server already down, but old dst still in lru map? New connection with same 5-tuple will still find the old lru dst.
Other question, why lru only handle udp timeout in connection_table_lookup, how about tcp connections?
Hi,
I am trying to reproduce the example here using virtual machines (i.e., client, server, router, katran). And the below diagram illustrates my environment:
I am using ubuntu 18.04 with the kernel 4.15.0-55-generic for the virtual machines.
client --(net1)-- router --(net2)-- katran
|
(net3)
|
server
net1: 192.168.150.0/24
net2: 10.10.10.0/24
net3: 50.10.10.0/24
client ip: 192.168.150.10
katran ip: 10.10.10.10
server ip: 50.10.10.10
router ip:
net1-if: 192.168.150.1
net2-if: 10.10.10.1
net3-if: 50.10.10.1
The routing table of the router is as below:
10.10.10.0/24 dev ens5 proto kernel scope link src 10.10.10.1
20.200.200.1 via 10.10.10.1 dev ens5
30.10.10.0/24 dev ens7 proto kernel scope link src 30.10.10.12
50.10.10.0/24 dev ens6 proto kernel scope link src 50.10.10.1
192.168.150.0/24 dev ens2 proto kernel scope link src 192.168.150.1
It seems the katran sends ipip packet correctly to the router after resolving VIP (i.e., 20.200.200.1). Here is a tcpdump from the router
15:51:55.996433 52:54:00:23:97:ef > 52:54:00:12:e8:4c, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 80)
172.16.242.206 > 50.10.10.10: (tos 0x0, ttl 63, id 21090, offset 0, flags [DF], proto TCP (6), length 60)
192.168.150.10.50276 > 20.200.200.1.22: Flags [S], cksum 0x7ddd (correct), seq 1289943810, win 29200, options [mss 1460,sackOK,TS val 2434696983 ecr 0,nop,wscale 7], length 0
15:51:57.017960 52:54:00:23:97:ef > 52:54:00:12:e8:4c, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 80)
172.16.242.206 > 50.10.10.10: (tos 0x0, ttl 63, id 21091, offset 0, flags [DF], proto TCP (6), length 60)
192.168.150.10.50276 > 20.200.200.1.22: Flags [S], cksum 0x79df (correct), seq 1289943810, win 29200, options [mss 1460,sackOK,TS val 2434698005 ecr 0,nop,wscale 7], length 0
But the problem is the router can't forward ipip packets to the server. It drops the packets before it calls the ip_recv() as shown the below kernel trace log. I expect the networking stack of the router calls the ip_recv() function to handle ipip packet.
This is the log of kernel call trace from the router when it receives ipip packets:
swapper 0 [003] 2339.377884: skb:kfree_skb: skbaddr=0xffff913f34574600 protocol=4 location=0xffffffffa6e5371f
7fffa6e40696 kfree_skb ([kernel.kallsyms])
7fffa6e5571f __netif_receive_skb_core ([kernel.kallsyms])
7fffa6e55c08 __netif_receive_skb ([kernel.kallsyms])
7fffa6e57f55 netif_receive_skb_internal ([kernel.kallsyms])
7fffa6e58c55 napi_gro_receive ([kernel.kallsyms])
7fffc004145a cp_rx_poll ([kernel.kallsyms])
7fffa6e58380 net_rx_action ([kernel.kallsyms])
7fffa72020e4 __softirqentry_text_start ([kernel.kallsyms])
7fffa66969b8 irq_exit ([kernel.kallsyms])
7fffa700462a do_IRQ ([kernel.kallsyms])
7fffa7002904 ret_from_intr ([kernel.kallsyms])
7fffa6fa3672 native_safe_halt ([kernel.kallsyms])
7fffa6fa3330 default_idle ([kernel.kallsyms])
7fffa663b035 arch_cpu_idle ([kernel.kallsyms])
7fffa6fa3803 default_idle_call ([kernel.kallsyms])
7fffa66d6db2 do_idle ([kernel.kallsyms])
7fffa66d7013 cpu_startup_entry ([kernel.kallsyms])
7fffa665b37b start_secondary ([kernel.kallsyms])
7fffa66020d5 secondary_startup_64 ([kernel.kallsyms])
Am I missing something to config the router?
thx~
Junho
Are you folks set on using GNU 2.0 as a license? Are you willing to consider using MIT or Apache?
Hi,
I am following the example setup as described in - https://github.com/facebookincubator/katran/blob/master/EXAMPLE.md. There are two ubuntu servers.
Server-1
Ubuntu 18.04.1 kernel version - 4.18.0.
This is to host katran server. Interface to bind ens4f0.201. IP assigned to this interface is IP-1.
Server-2
Ubuntu 16.04.4 LTS. Kernel version 4.10.0-27-generic.
This is to host back end real servers. Interface to bind is ens4f0.116. IP assigned to this interface is IP-2.
Ping is working fine between these two servers over these interfaces.
Server-1 ( hosting Katran server)
sudo ip tunnel add tun0 mode ipip local IP-1 remote IP-2 dev ens4f0.201
sudo ip link set up dev tun0
Server-2 ( hosting real backend servers)
sudo ip tunnel add tun0 mode ipip local IP-2 remote IP-1 dev ens4f0.116
sudo ip link set up dev tun0
Katan Server started -
sudo ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac -forwarding_cores=0 -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=ens4f0.201 -ipip_intf=tun0 -ipip6_intf=ipip60 -lru_size=10000
Note: i don't have IPv6 and hence here ipip60 is dummy. Without -ipip6_intf , it crashes.
VIP and real server IP added.
katran-master/example_grpc/goclient/bin$ ./main -A -t :22
2018/08/30 00:41:20 Adding service: 10.38.42.103:22 6
2018/08/30 00:41:20 Vip modified
exiting
katran-master/example_grpc/goclient/bin$ ./main -a -t :22 -r IP-2
Note: VIP is another IP from the same subnet of IP-1.
When ssh is attempted to the VIP 10.38.42.103, it's not working. Can you please tell me what is wrong in the setup?"ip a" doesn't show that VIP is bound to the interface where Katran server is running.
Hello, I'd like to make it possible to redirect packets to the port other than vip's port.
eg:
user --> katran(192.168.0.42:80) --> ... --> real(10.0.0.2:8080)
Could you please provide me a high level guide what needs to be changed in order to make it work? I will try to implement it.
HI, VIP gets announced using OSPF(quagga) and is set at the interface lo. ipip tunnels are present and up.
Because VIP was set at lo interface, healthcheck packets were not transmitted outside from enp0s3.
but, healthchecking_ipip.o work in enp0s3. What are the ways to solve this problem? Thanks.
is it a requirements for .travis.yml to be the way it is right now for ship-it/import-it to work? right now, because of this, travis make no sense and all PR/scheduled build are red. could we (at least temporary, till fbbuild would be able to work w/ katran) return to previous version of it?
Hi, and sorry in advance for being naive. I have an interesting application for Katran and was wondering if you could let me know whether you think Katran would be a good fit.
I need to collect and encapsulate packets relating to TCP sessions which may be received at any of a number of distinct geographically distributed egress points from a large IP network. Packets returning from the internet relating to a particular TCP session may appear at any of a number of different edge routers.
I'm wondering if I could install Katran in each egress data center and configure it to encapsulate each TCP session and route it to a consistent server within a centralized cluster of processing servers somewhere inside the network. In reality, the centralized cluster is itself a TCP-to-TCP proxy that understands the relevant L7 protocol; however, we only need Katran to help with marshalling the packets from the edge. Once we're inside the network, the network's fabric can route packets properly to their destination from the filtering servers.
It seems like this should be possible because from Katran's point of view, whether Katran is literally at the "top of the rack" or in a completely different rack somewhere else in the world should make no difference. Note that latency here is not a real concern given the underlying protocol we are proxying.
Hi,
I'm pretty much a novice in XDP and low level network protocol but reading rfc791 I understand that checksum should be calculate using the whole ip header.
E.g. :
Checksum
The internet header checksum is recomputed if the internet header is
changed. For example, a reduction of the time to live, additions or
changes to internet options, or due to fragmentation. This checksum
at the internet level is intended to protect the internet header
fields from transmission errors.
So would it be more correct to calculate checksum using iph->ihl
instead of sizeof(struct iphdr)
in balancer_helpers.h ?
(bonus question why using >> 1
instead of / 2
? 2nd solution seems semantically more appropriate, is there any reason or this is just a programming preference ?)
we test katran in our environment。and find some outer ip header checksum error about (1/100000)rate 。
I check the code in katran/lib/bpf/pckt_encap.h。
in some Small probability 。(csum & 0xffff) + (csum >> 16) may be overflow in 16bit。 it‘s may cause the problem I had mention
after fix code under groud。 this problem disappear。
- iph->check = ~((csum & 0xffff) + (csum >> 16));
+ csum = (csum & 0xffff) + (csum >> 16);
+ csum += (csum >>16);
+ iph->check = ~(csum);
thanks。
I gots some ERRORs when I executed os_run_tester.sh.
bule@bule:~/katran$ ./os_run_tester.sh
+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/bule/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/bule/katran/_build/deps
+ sudo sh -c '/home/bule/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error loading .BTF into kernel: -22.
E0304 14:05:02.500411 5329 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0304 14:05:02.500869 5329 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E0304 14:05:02.501143 5329 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E0304 14:05:02.501462 5329 BpfLoader.cpp:87] Can't find map w/ name: pckt_srcs
E0304 14:05:02.501798 5329 BpfLoader.cpp:87] Can't find map w/ name: hc_pckt_srcs_map
E0304 14:05:02.502049 5329 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0304 14:05:02.502434 5329 KatranLb.cpp:1350] Error gathering map stats for vip_map
E0304 14:05:02.502846 5329 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0304 14:05:02.503111 5329 KatranLb.cpp:1350] Error gathering map stats for reals
I0304 14:05:02.503408 5329 katran_tester.cpp:356] reals: current size is incorrect: 0
I0304 14:05:02.503669 5329 katran_tester.cpp:368] Initial testing of counters is complete
I0304 14:05:02.503984 5329 KatranLb.cpp:587] adding new vip: 10.200.1.1:80:17
I0304 14:05:02.664216 5329 KatranLb.cpp:587] adding new vip: 10.200.1.1:80:6
I0304 14:05:02.827368 5329 KatranLb.cpp:587] adding new vip: 10.200.1.2:0:6
I0304 14:05:03.000072 5329 KatranLb.cpp:587] adding new vip: 10.200.1.4:0:6
I0304 14:05:03.156711 5329 KatranLb.cpp:653] modyfing vip: 10.200.1.4:0:6
I0304 14:05:03.157140 5329 KatranLb.cpp:587] adding new vip: 10.200.1.3:80:6
I0304 14:05:03.319754 5329 KatranLb.cpp:587] adding new vip: fc00:1::1:80:6
I0304 14:05:03.477419 5329 KatranLb.cpp:587] adding new vip: 10.200.1.5:443:17
I0304 14:05:03.478024 5329 KatranLb.cpp:653] modyfing vip: 10.200.1.5:443:17
I0304 14:05:03.636570 5329 KatranLb.cpp:587] adding new vip: fc00:1::2:443:17
I0304 14:05:03.637153 5329 KatranLb.cpp:653] modyfing vip: fc00:1::2:443:17
I0304 14:05:03.794124 5329 BpfTester.cpp:214] Test: packet to UDP based v4 VIP (and v4 real) result: Passed
I0304 14:05:03.794591 5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real) result: Passed
I0304 14:05:03.794898 5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real) + ToS in IPV4 result: Passed
I0304 14:05:03.795253 5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real; any dst ports). result: Passed
I0304 14:05:03.795575 5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v6 real) result: Passed
I0304 14:05:03.795858 5329 BpfTester.cpp:214] Test: packet to TCP based v6 VIP (and v6 real) result: Passed
I0304 14:05:03.796102 5329 BpfTester.cpp:214] Test: packet to TCP based v6 VIP (and v6 real) with ToS / tc set result: Passed
I0304 14:05:03.796380 5329 BpfTester.cpp:214] Test: v4 ICMP echo-request result: Passed
I0304 14:05:03.796711 5329 BpfTester.cpp:214] Test: v6 ICMP echo-request result: Passed
I0304 14:05:03.797286 5329 BpfTester.cpp:214] Test: v4 ICMP dest-unreachabe fragmentation-needed result: Passed
I0304 14:05:03.797821 5329 BpfTester.cpp:214] Test: v6 ICMP packet-too-big result: Passed
I0304 14:05:03.798318 5329 BpfTester.cpp:214] Test: drop of IPv4 packet w/ options result: Passed
I0304 14:05:03.798739 5329 BpfTester.cpp:214] Test: drop of IPv4 fragmented packet result: Passed
I0304 14:05:03.799027 5329 BpfTester.cpp:214] Test: drop of IPv6 fragmented packet result: Passed
I0304 14:05:03.799335 5329 BpfTester.cpp:214] Test: pass of v4 packet with dst not equal to any configured VIP result: Passed
I0304 14:05:03.799625 5329 BpfTester.cpp:214] Test: pass of v6 packet with dst not equal to any configured VIP result: Passed
I0304 14:05:03.799966 5329 BpfTester.cpp:214] Test: pass of arp packet result: Passed
I0304 14:05:03.800016 5329 BpfTester.cpp:214] Test: LRU hit result: Passed
I0304 14:05:03.800035 5329 BpfTester.cpp:214] Test: packet #1 dst port hashing only result: Passed
I0304 14:05:03.800050 5329 BpfTester.cpp:214] Test: packet #2 dst port hashing only result: Passed
I0304 14:05:03.800062 5329 BpfTester.cpp:214] Test: ipinip packet result: Passed
I0304 14:05:03.800076 5329 BpfTester.cpp:214] Test: ipv6inipv6 packet result: Passed
I0304 14:05:03.800089 5329 BpfTester.cpp:214] Test: ipv4inipv6 packet result: Passed
I0304 14:05:03.800107 5329 BpfTester.cpp:214] Test: QUIC: long header. Client Initial type. LRU miss result: Passed
I0304 14:05:03.800124 5329 BpfTester.cpp:214] Test: QUIC: long header. 0-RTT Protected. CH. LRU hit. result: Passed
I0304 14:05:03.800452 5329 BpfTester.cpp:214] Test: QUIC: long header. Handshake. v4 vip v6 real. Conn Id based. result: Passed
I0304 14:05:03.800478 5329 BpfTester.cpp:214] Test: QUIC: long header. Retry. v4 vip v6 real. Conn Id based. result: Passed
I0304 14:05:03.800493 5329 BpfTester.cpp:214] Test: QUIC: long header. client initial. v6 vip v6 real. LRU miss result: Passed
I0304 14:05:03.800504 5329 BpfTester.cpp:214] Test: QUIC: short header. No connection id. CH. LRU hit result: Passed
I0304 14:05:03.800515 5329 BpfTester.cpp:214] Test: QUIC: short header w/ connection id result: Passed
I0304 14:05:03.800526 5329 BpfTester.cpp:214] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I0304 14:05:03.800536 5329 BpfTester.cpp:214] Test: QUIC: short header w/ conn id. host id = 0. CH. LRU hit result: Passed
I0304 14:05:03.800544 5329 katran_tester.cpp:378] Testing counter's sanity. Printing on errors only
I0304 14:05:03.800731 5329 katran_tester.cpp:418] incorrect stats about katran library internals: number of failed bpf syscalls is non zero
E0304 14:05:03.800858 5329 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0304 14:05:03.800873 5329 KatranLb.cpp:1350] Error gathering map stats for vip_map
I0304 14:05:03.800879 5329 katran_tester.cpp:356] vip_map: current size is incorrect: 0
E0304 14:05:03.800889 5329 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0304 14:05:03.800896 5329 KatranLb.cpp:1350] Error gathering map stats for reals
I0304 14:05:03.800904 5329 katran_tester.cpp:356] reals: current size is incorrect: 0
I0304 14:05:03.800909 5329 katran_tester.cpp:435] Testing of counters is complete
E0304 14:05:03.801021 5329 KatranSimulator.cpp:169] src and dst must have same address family
E0304 14:05:03.801036 5329 KatranSimulator.cpp:162] malformed src or dst ip address. src: aaaa dst: bbbb
E0304 14:05:03.801044 5329 BpfLoader.cpp:97] Can't find prog with name: cls-hc
I0304 14:05:03.801050 5329 katran_tester.cpp:309] Healthchecking not enabled. Skipping HC related tests
+ sudo sh -c '/home/bule/katran/_build/build/katran/decap/testing/xdpdecap_tester -decap_prog /home/bule/katran/_build/deps/bpfprog/bpf/decap_kern.o -test_from_fixtures=true '
libbpf: failed to open /home/bule/katran/_build/deps/bpfprog/bpf/decap_kern.o: No such file or directory
Segmentation fault (core dumped)
any sugguest?
thanks
I have a server that use 802.3ad bonding, and there's an issue in driver mode.
Since there were some troubles with building katran's dependences on my vm(out of memory when building fizz), so i wrote a balancer-user program(modified from kernel's bpf sample: xdp_tx_iptunnel_user.c) and complied it with katran's bpf files. The 'balancer_user.c' and some modifies are here:
balancer_user.c.txt
balaner_kern.diff.txt
It is only used to do a test. It will load katran's bpf file and set vip and rs, and only works when running seprate instance on per link in "generic xdp" mode.
Below is my env.
katran's version is 4c1fe70.
$ ifconfig
bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
inet 172.30.4.3 netmask 255.255.255.0 broadcast 172.30.4.255
inet6 fe80::ec6:ccff:fe9e:a9ce prefixlen 64 scopeid 0x20<link>
ether aa:bb:cc:dd:ee:ff txqueuelen 1000 (Ethernet)
RX packets 23856910322 bytes 2133122139232 (1.9 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 33499171263 bytes 22037784326757 (20.0 TiB)
TX errors 0 dropped 642 overruns 0 carrier 0 collisions 0
eth0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether aa:bb:cc:dd:ee:ff txqueuelen 1000 (Ethernet)
RX packets 1214891 bytes 766042605 (730.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1353364 bytes 719038863 (685.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether aa:bb:cc:dd:ee:ff txqueuelen 1000 (Ethernet)
RX packets 23202 bytes 14578863 (13.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 25149 bytes 14607499 (13.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2797883824 bytes 6603725657284 (6.0 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2797883824 bytes 6603725657284 (6.0 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tunl0: flags=193<UP,RUNNING,NOARP> mtu 1480
inet 172.30.4.253 netmask 255.255.255.255
tunnel txqueuelen 1000 (IPIP Tunnel)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
"172.30.4.253" is the vip.
$ cat /etc/sysconfig/network-scripts/ifcfg-bond1
#IP Config for bond1:
DEVICE=bond1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=4 miimon=100 ad_select=1 updelay=200 xmit_hash_policy=2"
IPV6INIT=no
TYPE=Unknown
NM_CONTROLLED=no
IPADDR='172.30.4.3'
NETMASK='255.255.255.0'
GATEWAY='172.30.4.1'
$ cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
...
$ ethtool -i eth0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x800003e2
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
$ uname -mrs
Linux 4.18.8-1.el7.elrepo.x86_64 x86_64
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ether aa:bb:cc:dd:ee:11 brd ff:ff:ff:ff:ff:ff
6: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
Here are my commands:
$ ./balancer
Start a XDP prog which encapsulates incoming packets
in an IPv4/v6 header and XDP_TX it out. The dst <VIP:PORT>
is used to select packets to encapsulate
Usage: ./balancer [...]
-i <ifindex> Interface Index
-a <vip-service-address> IPv4 or IPv6
-p <vip-service-port> Service port
-s <source-ip> Used in the IPTunnel header
-d <dest-ip> Used in the IPTunnel header, ip range (e.g. 1.1.1.1,2.2.2.2) is also allowed
-m <dest-MAC> Used in sending the IP Tunneled pkt
-T <stop-after-X-seconds> Default: 0 (forever)
-P <IP-Protocol> Default is TCP
-S use skb-mode
-N enforce native mode
-h Display this help
It works fine in generic mode(with '-S'):
sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S
sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S
the output is:
sum: 318928 pkts, rate: 159464 pkts/s
sum: 4213744 pkts, rate: 2106872 pkts/s
sum: 4241320 pkts, rate: 2120660 pkts/s
sum: 4251751 pkts, rate: 2125875 pkts/s
sum: 4269300 pkts, rate: 2134650 pkts/s
sum: 4288777 pkts, rate: 2144388 pkts/s
sum: 4295820 pkts, rate: 2147910 pkts/s
When using driver mode(with '-N'), it won't work properly.
sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N
sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N
sometimes it works:
$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
* Trying 172.30.4.253...
* Connected to 172.30.4.253 (172.30.4.253) port 80 (#0)
> GET /t HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.4.253
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.14.1
< Date: Tue, 25 Dec 2018 07:15:00 GMT
< Content-Type: application/octet-stream
< Content-Length: 2
< Connection: keep-alive
<
* Connection #0 to host 172.30.4.253 left intact
OK
and sometimes not:
$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
* Trying 172.30.4.253...
* Connection timed out
* Failed connect to 172.30.4.253:80; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection timed out
I tried curl
many times, almost 70% was failed.
And also i tried to run with "bond1" in generic mode, it couldn't work.
sudo ./balancer -i 6 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 10.210.9.100,10.210.9.101,10.210.9.102 -m 68:05:ca:21:d6:e5 -S
the result is:
$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
* Trying 172.30.4.253...
* Connection refused
* Failed connect to 172.30.4.253:80; Connection refused
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection refused
Hi
I'm trying to build katran on an Ubuntu 18.04 system.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
I'm using gcc version 7.5.0
and the latest commit of katran
$ git log
commit 096536b137bc18da66db946f9312e61dc7e330dc (HEAD -> master, origin/master, origin/HEAD)
Author: svcscm <[email protected]>
Date: Thu Sep 17 12:14:34 2020 -0700
Updating submodules
Summary:
GitHub commits:
https://github.com/facebook/fb303/commit/2cc39c81a4ab352d160c266516c5f1804406e10e
https://github.com/facebook/fbthrift/commit/d0ee2d31a4606864f2bfc88aaa0dff9d4dcf5467
https://github.com/facebook/fbzmq/commit/7690bf6b0e664cdc8a242b989f631ce1be1cf595
https://github.com/facebook/litho/commit/829f75ded87fe92ff926e943644fa193169efd57
https://github.com/facebook/proxygen/commit/d5c788f4c1f02cb810d458e44483f81f740e7c0e
https://github.com/facebook/rocksdb/commit/b9750c7c3c8c9b4fe5597a42fd73267c69281fe5
https://github.com/facebook/wangle/commit/1cf06f76dd7c0d6e0eb7083c81d1872afd40d48f
https://github.com/facebook/watchman/commit/49c01fccf225c35358995ea44efcf59c45547acc
https://github.com/facebookexperimental/rust-shed/commit/612fc34d70f911915b092ec89e0cdd6eb272b462
https://github.com/facebookincubator/fizz/commit/c2c96af540c50d26c2739431ebb8458804f74f8f
https://github.com/facebookincubator/katran/commit/8e217fa90cac4d4f39db8e1dc3ba7eed11c8d73a
https://github.com/facebookincubator/mvfst/commit/df966d4ffca17873ab35323b2e0a356e38dbd9b9
https://github.com/facebookincubator/profilo/commit/e45cb217266da4e0e053d36afd12ffd89b8338f5
https://github.com/rsocket/rsocket-cpp/commit/f2b05dc5e855570c7f98131c1c6970868ef5b0f3
Reviewed By: yns88
fbshipit-source-id: 765c8b0fce704a6d3503392df73714b3054b1392
Following is the error on running ./build_katran.sh
Scanning dependencies of target katran_server_grpc
[ 98%] Building CXX object example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o
/mydata/local/katran/example_grpc/katran_server.cpp: In function ‘int main(int, char**)’:
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
};
^
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:62: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o' failed
make[2]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o] Error 1
CMakeFiles/Makefile2:2282: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Please take a look. Thanks!
./build_katran is broken
-- Found folly: /usr/local
-- Found fizz: /usr/local
CMake Error at CMakeLists.txt:30 (find_package):
By not providing "FindGlog.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Glog", but
CMake did not find one.
Could not find a package configuration file provided by "Glog" with any of
the following names:
GlogConfig.cmake
glog-config.cmake
Add the installation prefix of "Glog" to CMAKE_PREFIX_PATH or set
"Glog_DIR" to a directory containing one of the above files. If "Glog"
provides a separate development package or SDK, be sure it has been
installed.
-- Configuring incomplete, errors occurred!
See also "/home/tehnerd/katran/build/CMakeFiles/CMakeOutput.log".
tehnerd@devubuntu:~/katran$
if i start to install fizz, it became broken on glog (same error) etc etc
what is the blessed way to build open source version of katran?
root@katran-big:/home/syla/katran# ./os_run_tester.sh
if i restart katran_server then config data miss take,
if i have to two network interface Wan <--->Lan,how can i configure
like this:
client(172.16.1.100)--->Wan(eth1:172.16.1.101)(vip:172.16.1.200)<--->Lan(eth0:192.168.1.10)--->server(192.168.1.20)
Hi folks,
when tesing healthcheck, i found katran terminate with "called after throwing an instance of
std::invalid_argument" with attaching healthcheck bpf to main interface. After added some log, i found it's the mnl_cb_run
in modifyTcBpfFilter
return -1 and set errno to INVAL
.
I have digged into libmnl source https://github.com/threatstack/libmnl/blob/master/src/callback.c while there is no INVAL
in in __mnl_cb_run
. would anyone help me to figure this out?
Thanks
When I run katran_grpc_server
with the following commands:
./katran_server_grpc -hc_forwarding=false -default_mac xx:xx:xx:xx:xx:xx -intf ens1f1
I get the following error:
I0413 19:11:23.831044 1 KatranGrpcService.cpp:68] Starting Katran
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error loading .BTF into kernel: -22.
E0413 19:11:23.941128 1 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0413 19:11:23.941141 1 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E0413 19:11:23.941144 1 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E0413 19:11:23.941172 1 BpfAdapter.cpp:503] Error receiving netlink message: Cannot allocate memory [12]
terminate called after throwing an instance of 'std::invalid_argument'
what(): can't attach main bpf prog to main inteface
*** Aborted at 1586805083 (Unix time, try 'date -d @1586805083') ***
*** Signal 6 (SIGABRT) (0x1) received by PID 1 (pthread TID 0x7f95729118c0) (linux TID 1) (maybe from PID 1, UID 0) (code: -6), stack trace: ***
@ 000055cdf085dae2 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
@ 000055cdf085dbad folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
@ 00007f95724e688f (unknown)
@ 00007f95717f6e97 gsignal
@ 00007f95717f8800 abort
@ 00007f9571fd3956 (unknown)
@ 00007f9571fd9ab5 (unknown)
@ 00007f9571fd9af0 std::terminate()
@ 00007f9571fd9d23 __cxa_throw
@ 000055cdf0c1734f katran::KatranLb::attachBpfProgs()
/katran/katran/lib/KatranLb.cpp:411
@ 000055cdf0a1187c lb::katran::KatranGrpcService::KatranGrpcService(katran::KatranConfig const&)
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 000055cdf06ddf1f RunServer(katran::KatranConfig&, int, std::shared_ptr<folly::EventBase>)
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 000055cdf06de77b main
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 00007f95717d9b96 __libc_start_main
@ 000055cdf06ddb39 _start
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
*** Aborted at 1586805084 (Unix time, try 'date -d @1586805084') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 1 (pthread TID 0x7f95729118c0) (linux TID 1) (code: 128), stack trace: ***
@ 000055cdf085dae2 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
@ 000055cdf085dbad folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
/katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
@ 00007f95724e688f (unknown)
@ 00007f95717f88f0 abort
@ 00007f9571fd3956 (unknown)
@ 00007f9571fd9ab5 (unknown)
@ 00007f9571fd9af0 std::terminate()
@ 00007f9571fd9d23 __cxa_throw
@ 000055cdf0c1734f katran::KatranLb::attachBpfProgs()
/katran/katran/lib/KatranLb.cpp:411
@ 000055cdf0a1187c lb::katran::KatranGrpcService::KatranGrpcService(katran::KatranConfig const&)
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 000055cdf06ddf1f RunServer(katran::KatranConfig&, int, std::shared_ptr<folly::EventBase>)
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 000055cdf06de77b main
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
@ 00007f95717d9b96 __libc_start_main
@ 000055cdf06ddb39 _start
-> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
and dmesg -T
shows the followings output:
[Mon Apr 13 23:46:18 2020] traps: katran_server_g[35859] general protection ip:7f2a18a698f0 sp:7ffe98b69ea0 error:0 in libc-2.27.so[7f2a18a29000+1e7000]
[Mon Apr 13 23:46:18 2020] traps: katran_server_g[35857] general protection ip:7f7541b758f0 sp:7ffe16337750 error:0 in libc-2.27.so[7f7541b35000+1e7000]
[Mon Apr 13 23:47:19 2020] traps: katran_server_g[36045] general protection ip:7f1f7048d8f0 sp:7fff148ce510 error:0 in libc-2.27.so[7f1f7044d000+1e7000]
[Mon Apr 13 23:47:19 2020] traps: katran_server_g[36060] general protection ip:7f0d22cae8f0 sp:7fff9120b0e0 error:0 in libc-2.27.so[7f0d22c6e000+1e7000]
[Mon Apr 13 23:48:19 2020] traps: katran_server_g[36262] general protection ip:7fcb70e318f0 sp:7ffd549ea720 error:0 in libc-2.27.so[7fcb70df1000+1e7000]
[Mon Apr 13 23:48:19 2020] traps: katran_server_g[36251] general protection ip:7f880bd458f0 sp:7fff048b66b0 error:0 in libc-2.27.so[7f880bd05000+1e7000]
[Mon Apr 13 23:49:20 2020] traps: katran_server_g[36460] general protection ip:7eff78eca8f0 sp:7ffc6724b8e0 error:0 in libc-2.27.so[7eff78e8a000+1e7000]
[Mon Apr 13 23:49:20 2020] traps: katran_server_g[36474] general protection ip:7f2808bb08f0 sp:7ffc60646cc0 error:0 in libc-2.27.so[7f2808b70000+1e7000]
my kernel-version is: 4.15.0-96-generic
I also tried on 5.6.3-050603-lowlatency
CPU Model:
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
NIC Model:
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
ixgbe version: 5.6.5
any ideas?
Needs a logo (I'm building a slide of BPF technologies).
I get this error when I run os_run_tester.sh
+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/bule/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/bule/katran/_build/deps
+ sudo sh -c '/home/bule/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
[sudo] password for bule:
libbpf: elf: skipping unrecognized data section(6) .maps.ctl_array
libbpf: elf: skipping unrecognized data section(7) .maps.vip_map
libbpf: elf: skipping unrecognized data section(8) .maps.fallback_cache
libbpf: elf: skipping unrecognized data section(9) .maps.ch_rings
libbpf: elf: skipping unrecognized data section(10) .maps.reals
libbpf: elf: skipping unrecognized data section(11) .maps.reals_stats
libbpf: elf: skipping unrecognized data section(12) .maps.stats
libbpf: elf: skipping unrecognized data section(13) .maps.quic_mapping
libbpf: elf: skipping unrecognized data section(25) .eh_frame
libbpf: elf: skipping relo section(26) .rel.eh_frame for section(25) .eh_frame
libbpf: load bpf program failed: Invalid argument
libbpf: failed to load program 'balancer_ingress'
libbpf: failed to load object '/home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o'
E0922 09:58:04.089079 1681 BpfLoader.cpp:307] error while trying to load bpf object: /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o
terminate called after throwing an instance of 'std::invalid_argument'
what(): can't load main bpf program
Aborted (core dumped)
ubuntu 18.04 and kernel 4.15
how hard is it going to be to add clang-format style file into katran's repo? (clang-format -dump-config to generate). right now if i'm running clang-format -i on chaged files i can see lots of unrelated changes. could be because of some custom options of clang-format which is using internally @ fb. would be nice for everyone (internal + external ) to be in sync on how clang-format is configured
Hi folks.
I spun up a fresh VM with Ubuntu 18.04 to test this, and I'm getting a SIGSEGV running the os_run_tester.sh
script.
After changing build/CMakeCache.txt: like this:
...
CMAKE_BUILD_TYPE:STRING=Debug
...
CMAKE_CXX_FLAGS_DEBUG:STRING=-g3 -fno-stack-protector
...
I then used gdb to run the tester like so:
(gdb) file /usr/local/src/katran/build/katran/lib/testing/katran_tester
Reading symbols from /usr/local/src/katran/build/katran/lib/testing/katran_tester...done.
(gdb) run -balancer_prog /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true
Starting program: /usr/local/src/katran/build/katran/lib/testing/katran_tester -balancer_prog /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
E0617 19:44:14.898008 24596 BpfLoader.cpp:160] Can't read section size for index: 2
I0617 19:44:14.898075 24596 BpfLoader.cpp:434] Skipping section: 2 of file: /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o
I0617 19:44:14.931807 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0617 19:44:14.931880 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0617 19:44:14.931902 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0617 19:44:14.931906 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0617 19:44:14.931908 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
I0617 19:44:14.939867 24596 KatranLb.cpp:394] adding new vip: 10.200.1.1:80:17
I0617 19:44:15.020308 24596 KatranLb.cpp:394] adding new vip: 10.200.1.1:80:6
I0617 19:44:15.091914 24596 KatranLb.cpp:394] adding new vip: 10.200.1.2:0:6
I0617 19:44:15.162443 24596 KatranLb.cpp:394] adding new vip: 10.200.1.4:0:6
I0617 19:44:15.231927 24596 KatranLb.cpp:459] modyfing vip: 10.200.1.4:0:6
I0617 19:44:15.231992 24596 KatranLb.cpp:394] adding new vip: 10.200.1.3:80:6
I0617 19:44:15.301601 24596 KatranLb.cpp:394] adding new vip: fc00:1::1:80:6
I0617 19:44:15.370398 24596 KatranLb.cpp:394] adding new vip: 10.200.1.5:443:17
I0617 19:44:15.370491 24596 KatranLb.cpp:459] modyfing vip: 10.200.1.5:443:17
I0617 19:44:15.440598 24596 KatranLb.cpp:394] adding new vip: fc00:1::2:443:17
I0617 19:44:15.440691 24596 KatranLb.cpp:459] modyfing vip: fc00:1::2:443:17
I0617 19:44:15.512671 24596 XdpTester.cpp:166] Test: packet to UDP based v4 VIP (and v4 real) result: Passed
I0617 19:44:15.512732 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v4 real) result: Passed
I0617 19:44:15.512744 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v4 real; any dst ports). result: Passed
I0617 19:44:15.512753 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v6 real) result: Passed
I0617 19:44:15.512763 24596 XdpTester.cpp:166] Test: packet to TCP based v6 VIP (and v6 real) result: Passed
I0617 19:44:15.512769 24596 XdpTester.cpp:166] Test: v4 ICMP echo-request result: Passed
I0617 19:44:15.512776 24596 XdpTester.cpp:166] Test: v6 ICMP echo-request result: Passed
I0617 19:44:15.512784 24596 XdpTester.cpp:166] Test: v4 ICMP dest-unreachabe fragmentation-needed result: Passed
I0617 19:44:15.512814 24596 XdpTester.cpp:166] Test: v6 ICMP packet-too-big result: Passed
I0617 19:44:15.512897 24596 XdpTester.cpp:166] Test: drop of IPv4 packet w/ options result: Passed
I0617 19:44:15.512924 24596 XdpTester.cpp:166] Test: drop of IPv4 fragmented packet result: Passed
I0617 19:44:15.512948 24596 XdpTester.cpp:166] Test: drop of IPv6 fragmented packet result: Passed
I0617 19:44:15.512956 24596 XdpTester.cpp:166] Test: pass of v4 packet with dst not equal to any configured VIP result: Passed
I0617 19:44:15.512964 24596 XdpTester.cpp:166] Test: pass of v6 packet with dst not equal to any configured VIP result: Passed
I0617 19:44:15.512971 24596 XdpTester.cpp:166] Test: pass of arp packet result: Passed
I0617 19:44:15.512979 24596 XdpTester.cpp:166] Test: LRU hit result: Passed
I0617 19:44:15.512990 24596 XdpTester.cpp:166] Test: packet #1 dst port hashing only result: Passed
I0617 19:44:15.512998 24596 XdpTester.cpp:166] Test: packet #2 dst port hashing only result: Passed
I0617 19:44:15.513005 24596 XdpTester.cpp:166] Test: ipinip packet result: Passed
I0617 19:44:15.513031 24596 XdpTester.cpp:166] Test: ipv6inipv6 packet result: Passed
I0617 19:44:15.513054 24596 XdpTester.cpp:166] Test: ipv4inipv6 packet result: Passed
I0617 19:44:15.513063 24596 XdpTester.cpp:166] Test: QUIC: long header. Client Initial type result: Passed
I0617 19:44:15.513092 24596 XdpTester.cpp:166] Test: QUIC: long header. 0-RTT Protected result: Passed
I0617 19:44:15.513114 24596 XdpTester.cpp:166] Test: QUIC: long header. v4 vip v6 real result: Passed
I0617 19:44:15.513147 24596 XdpTester.cpp:166] Test: QUIC: long header. v6 vip v6 real result: Passed
I0617 19:44:15.513252 24596 XdpTester.cpp:166] Test: QUIC: short header. no connection id result: Passed
I0617 19:44:15.513362 24596 XdpTester.cpp:166] Test: QUIC: short header w/ connection id result: Passed
I0617 19:44:15.513447 24596 XdpTester.cpp:166] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I0617 19:44:15.513480 24596 katran_tester.cpp:166] Testing counter's sanity. Printing on errors only
Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
Here is the BT:
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()
And a list:
(gdb) list
177 stats = lb.getLruMissStats();
178 if ((stats.v1 != 2) || (stats.v2 != 6)) {
179 VLOG(2) << "TCP syns: " << stats.v1 << " TCP non-syns: " << stats.v2;
180 LOG(INFO) << "per pckt type LRU miss counter is incorrect";
181 }
182 stats = lb.getLruFallbackStats();
183 if (stats.v1 != 13) {
184 VLOG(2) << "FallbackLRU hits: " << stats.v1;
185 LOG(INFO) << "LRU fallback counter is incorrect";
186 }
Here are the particulars of the test box:
# uname -a
Linux katran 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.3.0-16ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --with-as=/usr/bin/x86_64-linux-gnu-as --with-ld=/usr/bin/x86_64-linux-gnu-ld --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)
I'm out of my depth here, and was hoping you all might look at it. Let me know what other info you need or if there is any other way I can help. Reproducing should be as easy as spinning up a stock Ubuntu 18.04 VM and running build_katran.sh
.
Thanks!
Hi, my config is similar to this one about healthcheck while for some reason in our cvm environment, we modified balance_kern.c, DSR is replaced by TUNNEL and return packet will go through LB.
now packet forwarding is well but health check is not. Actually i can capture hc return packet with tcpdump, but the hc process do not get the response.
configure in my environment
LB(katran):
RS:
configured with tunnel back
-eth0 172.33.35.7 mark 10000
health check program:
bind socket with 172.33.4.100 and send SYNC to 172.33.4.78(vip) with mark 10000.
for test, i modified balancer_kern, when recving IPIP pacekt, directly return XDP_PASS to kernel.
but my hc program do not recv ACK while i can capture ACK packet in LB with tcpdump.
strangely when i ifconfig down the vip in LB, hc program can recv ACK.
Hey, I try to run Katran in a setup where more reals sit at one physical server node together with Katran.
So there are two nodes master (10.80.96.1) and test (10.80.96.3). master runs two reals(10.80.96.70, 10.80.96.68) that are part of one VIP (10.80.96.13) and test is used to create requests to the VIP.
root@master# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.80.96.13/32 brd 10.80.96.13 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s20f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether 0c:c4:7a:e5:46:d6 brd ff:ff:ff:ff:ff:ff
3: enp0s20f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
link/ether 0c:c4:7a:e5:46:d7 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 xdpgeneric/id:142 qdisc noqueue state UP group default qlen 1000
link/ether 0c:c4:7a:e5:46:d6 brd ff:ff:ff:ff:ff:ff
inet 147.x.x.x/31 brd 255.255.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet 10.80.96.1/31 brd 255.255.255.255 scope global bond0:0
valid_lft forever preferred_lft forever
inet6 2604:1380:2000:8f00::1/127 scope global
valid_lft forever preferred_lft forever
inet6 fe80::ec4:7aff:fee5:46d6/64 scope link
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:30:1c:4b:ae brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:30ff:fe1c:4bae/64 scope link
valid_lft forever preferred_lft forever
8: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0a:58:0a:50:60:41 brd ff:ff:ff:ff:ff:ff
inet 10.80.96.65/29 scope global kube-bridge
valid_lft forever preferred_lft forever
inet6 fe80::9487:55ff:fe03:69a1/64 scope link
valid_lft forever preferred_lft forever
11: kube-dummy-if: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether f2:7d:88:7a:96:d3 brd ff:ff:ff:ff:ff:ff
inet 10.96.0.1/32 brd 10.96.0.1 scope link kube-dummy-if
valid_lft forever preferred_lft forever
inet 10.96.0.10/32 brd 10.96.0.10 scope link kube-dummy-if
valid_lft forever preferred_lft forever
inet 10.105.217.13/32 brd 10.105.217.13 scope link kube-dummy-if
valid_lft forever preferred_lft forever
inet6 fe80::f07d:88ff:fe7a:96d3/64 scope link
valid_lft forever preferred_lft forever
12: vethde26d90b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default
link/ether ba:f2:1e:a0:15:2f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::b8f2:1eff:fea0:152f/64 scope link
valid_lft forever preferred_lft forever
13: veth7b135f01@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default
link/ether aa:20:29:d3:29:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::a820:29ff:fed3:29e2/64 scope link
valid_lft forever preferred_lft forever
14: veth1956ecdc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default
link/ether 2a:d7:28:63:92:3e brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::28d7:28ff:fe63:923e/64 scope link
valid_lft forever preferred_lft forever
15: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
16: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 127.0.0.42/32 scope host ipip0
valid_lft forever preferred_lft forever
18: veth9c036093@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default
link/ether fa:0a:06:f9:28:5e brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::f80a:6ff:fef9:285e/64 scope link
valid_lft forever preferred_lft forever
19: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
link/tunnel6 :: brd ::
20: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN group default qlen 1000
link/tunnel6 :: brd ::
inet6 fe80::ac1a:bbff:fea6:2184/64 scope link
valid_lft forever preferred_lft forever
Every node has assigned subnet of ips that can sit there. In this case 10.80.96.64/29 through kube-bridge
:
root@master:~# ip route
default via 147.x.x.x dev bond0 onlink
10.0.0.0/8 via 10.80.96.0 dev bond0
10.80.96.0/31 dev bond0 proto kernel scope link src 10.80.96.1
10.80.96.64/29 dev kube-bridge proto kernel scope link src 10.80.96.65
147.x.x.x/31 dev bond0 proto kernel scope link src 147.x.x.x
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
root@master:~# ./katran/katranc -l
2019/02/06 12:13:31 vips len 1
VIP: 10.80.96.13 Port: 80 Protocol: tcp
Vip's flags:
->10.80.96.70 weight: 1
->10.80.96.68 weight: 1
exiting
tcpdump at master:
12:24:05.312977 IP 10.80.96.3.60764 > 10.80.96.13.http: Flags [S], seq 949265021, win 29200, options [mss 1460,sackOK,TS val 2208753565 ecr 0,nop,wscale 8], length 0
12:24:05.313047 IP 10.80.96.13.http > 10.80.96.3.60764: Flags [R.], seq 0, ack 949265022, win 0, length 0
root@test:~# curl 10.80.96.13:80
curl: (7) Failed to connect to 10.80.96.13 port 80: Connection refused
Reals 10.80.96.70:80, 10.80.96.68:80 could be accessed, it's a simple nginx.
root@test:~# curl 10.80.96.70
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
VIP 10.80.96.13/32 gets announced using BGP and is set at the interface lo. ipip tunnels are present and up.
Request on 10.80.96.13:80 returns connection refused, but tcpdump shows traffic got to the node.
I think it is caused by the fact I use bridge. Is it the case or do you see any other cause? Is there any way to mitigate this issue? @tehnerd
root@master:~# ps faux | grep katran
root 23059 0.0 0.0 13108 1064 pts/5 S+ 12:56 0:00 \_ grep --color=auto katran
root 3682 0.0 0.0 18376 2964 ? Ss 12:13 0:00 | \_ /bin/bash /katran/start_katran_grpc_server_smart.sh
root 3731 0.0 0.1 479676 11512 ? SLl 12:13 0:00 | \_ /katran/katran_server_grpc -balancer_prog /katran/balancer_kern.o -default_mac 54:1e:56:64:67:00 -forwarding_cores=0,1,2,3 -hc_forwarding=false -intf=bond0 -lru_size=10000
root@master:~# ip route | grep default
default via 147.x.x.x dev bond0 onlink
root@master:~# ip n show | grep 147.x.x.x
147.x.x.x dev bond0 lladdr 54:1e:56:64:67:00 REACHABLE
root@master:~# ethtool -l enp0s20f0
Channel parameters for enp0s20f0:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@master:~# ethtool -l enp0s20f1
Channel parameters for enp0s20f1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
I ran ./build_katran.sh and got this error:
/home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc: In member function ‘virtual void X509Test_ZeroLengthsWithX509PARAM_Test::TestBody()’:
/home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc:712:10: error: declaration of ‘struct X509Test_ZeroLengthsWithX509PARAM_Test::TestBody()::Test’ shadows a previous local [-Werror=shadow]
struct Test {
^~~~
In file included from /home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc:19:
/home/kahou/katran/deps/grpc/third_party/boringssl/third_party/googletest/include/gtest/gtest.h:375:23: note: shadowed declaration is here
class GTEST_API_ Test {
^
cc1plus: all warnings being treated as errors
make[2]: *** [third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/build.make:635: third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/x509/x509_test.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2348: third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
What's the throughput(pps)?Could you show me the data chart?
Hello! I get the way non-IPIP packets are processed on L4 nodes. But what does happen to them once they get to L7 node? Is there also Katran that does the decapsulation? I've been reading the BPF code and there are some signs Katran does this when INLINE_DECAP
is set. And then there is a function process_encaped_pckt
that actually does the decapsulation I think.
I assume that Kartan is also supposed to run on L7 nodes and do the decapsulation of IPIP packets. Could you please tell whether my assumption is correct or not?
Hi,
I am trying to build katran on my ubuntu 19.10
user:~/katran$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan
and using katran release
$ git log
commit 3d50460170b2e7d5d05d85e011f5fe5ef196cddf (HEAD -> master, origin/master, origin/HEAD)
Author: svcscm <[email protected]>
Date: Wed Jan 29 10:45:21 2020 -0800
Updating submodules
Summary:
GitHub commits:
https://github.com/facebook/litho/commit/fada02ad90dcf14e29fe22acbb3485595d6b93de
https://github.com/facebook/rocksdb/commit/a07a9dc904b692db67052e58ccd0b5d2e96aff92
https://github.com/facebookincubator/fizz/commit/fbb77357c42ece2f64c1aa3fc6a4e3ec17708b44
https://github.com/facebookincubator/profilo/commit/3b3e46f2b308aec845c7b3718ab4ef36d0019707
Reviewed By: 2d2d2d2d2d
fbshipit-source-id: 0cc55e3757528f2f9db4c32dbaef32174c808706
Here is the issue.
-- Found Protobuf: /home/bfn/katran/_build/deps/lib/libprotobuf.a;-lpthread (found version "3.11.2")
cmd is /home/bfn/katran/_build/deps/bin/protoc
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ABSL_BAD_OPTIONAL
linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc
ABSL_BASE_THROW
linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc
ABSL_STRINGS
linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc
-- Configuring incomplete, errors occurred!
See also "/home/bfn/katran/_build/build/CMakeFiles/CMakeOutput.log".
See also "/home/bfn/katran/_build/build/CMakeFiles/CMakeError.log".
Please take a look. Thanks.
w/ latest upstream diff:
commit 4f8508ca1cb363fce2b16a6ed3200d511f8f59ac (refs/remotes/upstream/master)
Author: Udip Pant <[email protected]>
Date: Wed Oct 9 09:21:16 2019 -0700
add unit tests for feature to copy ToS value from inner packet
Reviewed By: mjoras
Differential Revision: D17826785
fbshipit-source-id: 3eacfa8947a42f1aea9b0d1345d6b6d40ddcacec
running katran_tester:
./os_run_tester.sh
+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/tehnerd/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/tehnerd/katran/_build/deps
+ sudo sh -c '/home/tehnerd/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/tehnerd/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
libbpf: Error in bpf_create_map_xattr(lru_maps_mapping):ERROR: strerror_r(-524)=22(-524). Retrying without BTF.
E1009 18:34:46.199492 21152 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E1009 18:34:46.199611 21152 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E1009 18:34:46.199625 21152 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E1009 18:34:46.199632 21152 BpfLoader.cpp:87] Can't find map w/ name: pckt_srcs
I1009 18:34:46.199656 21152 KatranLb.cpp:481] adding new vip: 10.200.1.1:80:17
I1009 18:34:46.274731 21152 KatranLb.cpp:481] adding new vip: 10.200.1.1:80:6
I1009 18:34:46.350360 21152 KatranLb.cpp:481] adding new vip: 10.200.1.2:0:6
I1009 18:34:46.418886 21152 KatranLb.cpp:481] adding new vip: 10.200.1.4:0:6
I1009 18:34:46.486299 21152 KatranLb.cpp:546] modyfing vip: 10.200.1.4:0:6
I1009 18:34:46.486353 21152 KatranLb.cpp:481] adding new vip: 10.200.1.3:80:6
I1009 18:34:46.553889 21152 KatranLb.cpp:481] adding new vip: fc00:1::1:80:6
I1009 18:34:46.620697 21152 KatranLb.cpp:481] adding new vip: 10.200.1.5:443:17
I1009 18:34:46.620771 21152 KatranLb.cpp:546] modyfing vip: 10.200.1.5:443:17
I1009 18:34:46.687752 21152 KatranLb.cpp:481] adding new vip: fc00:1::2:443:17
I1009 18:34:46.687876 21152 KatranLb.cpp:546] modyfing vip: fc00:1::2:443:17
I1009 18:34:46.755780 21152 XdpTester.cpp:168] Test: packet to UDP based v4 VIP (and v4 real) result: Passed
I1009 18:34:46.755833 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real) result: Passed
I1009 18:34:46.755846 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real) + ToS in IPV4 result: Passed
I1009 18:34:46.755856 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real; any dst ports). result: Passed
I1009 18:34:46.755867 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v6 real) result: Passed
I1009 18:34:46.755877 21152 XdpTester.cpp:168] Test: packet to TCP based v6 VIP (and v6 real) result: Passed
I1009 18:34:46.755887 21152 XdpTester.cpp:168] Test: packet to TCP based v6 VIP (and v6 real) with ToS / tc set result: Passed
I1009 18:34:46.755895 21152 XdpTester.cpp:168] Test: v4 ICMP echo-request result: Passed
I1009 18:34:46.755903 21152 XdpTester.cpp:168] Test: v6 ICMP echo-request result: Passed
I1009 18:34:46.755913 21152 XdpTester.cpp:168] Test: v4 ICMP dest-unreachabe fragmentation-needed result: Passed
I1009 18:34:46.755923 21152 XdpTester.cpp:168] Test: v6 ICMP packet-too-big result: Passed
I1009 18:34:46.755933 21152 XdpTester.cpp:168] Test: drop of IPv4 packet w/ options result: Passed
I1009 18:34:46.755940 21152 XdpTester.cpp:168] Test: drop of IPv4 fragmented packet result: Passed
I1009 18:34:46.755949 21152 XdpTester.cpp:168] Test: drop of IPv6 fragmented packet result: Passed
I1009 18:34:46.755957 21152 XdpTester.cpp:168] Test: pass of v4 packet with dst not equal to any configured VIP result: Passed
I1009 18:34:46.755966 21152 XdpTester.cpp:168] Test: pass of v6 packet with dst not equal to any configured VIP result: Passed
I1009 18:34:46.755975 21152 XdpTester.cpp:168] Test: pass of arp packet result: Passed
I1009 18:34:46.755982 21152 XdpTester.cpp:168] Test: LRU hit result: Passed
I1009 18:34:46.755992 21152 XdpTester.cpp:168] Test: packet #1 dst port hashing only result: Passed
I1009 18:34:46.756001 21152 XdpTester.cpp:168] Test: packet #2 dst port hashing only result: Passed
I1009 18:34:46.756009 21152 XdpTester.cpp:168] Test: ipinip packet result: Passed
I1009 18:34:46.756018 21152 XdpTester.cpp:168] Test: ipv6inipv6 packet result: Passed
I1009 18:34:46.756027 21152 XdpTester.cpp:168] Test: ipv4inipv6 packet result: Passed
I1009 18:34:46.756036 21152 XdpTester.cpp:168] Test: QUIC: long header. Client Initial type. LRU miss result: Passed
I1009 18:34:46.756045 21152 XdpTester.cpp:168] Test: QUIC: long header. 0-RTT Protected. CH. LRU hit. result: Passed
I1009 18:34:46.756054 21152 XdpTester.cpp:168] Test: QUIC: long header. Handshake. v4 vip v6 real. Conn Id based. result: Passed
I1009 18:34:46.756062 21152 XdpTester.cpp:168] Test: QUIC: long header. Retry. v4 vip v6 real. Conn Id based. result: Passed
I1009 18:34:46.756072 21152 XdpTester.cpp:168] Test: QUIC: long header. client initial. v6 vip v6 real. LRU miss result: Passed
I1009 18:34:46.756081 21152 XdpTester.cpp:168] Test: QUIC: short header. No connection id. CH. LRU hit result: Passed
I1009 18:34:46.756089 21152 XdpTester.cpp:168] Test: QUIC: short header w/ connection id result: Passed
I1009 18:34:46.756098 21152 XdpTester.cpp:168] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I1009 18:34:46.756106 21152 XdpTester.cpp:168] Test: QUIC: short header w/ conn id. host id = 0. CH. LRU hit result: Passed
I1009 18:34:46.756217 21152 katran_tester.cpp:327] Testing counter's sanity. Printing on errors only
I1009 18:34:46.756355 21152 katran_tester.cpp:377] Testing of counters is complete
E1009 18:34:46.756445 21152 KatranSimulator.cpp:169] src and dst must have same address family
E1009 18:34:46.756454 21152 KatranSimulator.cpp:162] malformed src or dst ip address. src: aaaa dst: bbbb
aka everything is green/passed. however if i check what actual packets were generated (it is input/output format):
./os_run_tester.sh -pcap_output=/tmp/test.pcap
18:35:45.391440 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto UDP (17), length 43)
192.168.1.1.31337 > 10.200.1.1.80: [udp sum ok] UDP, length 15
18:35:45.391454 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 63)
172.16.104.123 > 10.0.0.3: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto UDP (17), length 43)
192.168.1.1.31337 > 10.200.1.1.80: [udp sum ok] UDP, length 15
18:35:45.391502 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391510 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
172.16.104.123 > 10.0.0.3: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
bellow v4 input has tos output does not
18:35:45.391521 IP (tos 0x8c, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
no tos in output packet:
18:35:45.391525 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
172.16.104.123 > 10.0.0.3: IP (tos 0x8c, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
18:35:45.391533 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.2.42: Flags [.], cksum 0x2809 (correct), seq 0:15, ack 0, win 8192, length 15
18:35:45.391537 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
172.16.104.123 > 10.0.0.2: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.2.42: Flags [.], cksum 0x2809 (correct), seq 0:15, ack 1, win 8192, length 15
18:35:45.391544 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.3.80: Flags [.], cksum 0x27e2 (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391549 IP6 (hlim 64, next-header IPIP (4) payload length: 55) 100::bac1:101 > fc00::1: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
192.168.1.1.31337 > 10.200.1.3.80: Flags [.], cksum 0x27e2 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
18:35:45.391556 IP6 (hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391561 IP6 (hlim 64, next-header IPv6 (41) payload length: 75) 100::7a69:1 > fc00::3: IP6 (hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
18:35:45.391570 IP6 (class 0x8c, hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
v6 seems working as expected:
18:35:45.391574 IP6 (class 0x8c, hlim 64, next-header IPv6 (41) payload length: 75) 100::7a69:1 > fc00::3: IP6 (class 0x8c, hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
the issue is that tos population is inside "decrement_ttl" stanza:
https://github.com/facebookincubator/katran/blob/master/katran/lib/bpf/balancer_kern.c#L230
which is set for true only for IPIP decapsulated packets when INLINE_DECAP is used (e.g. for stable anycast usecase)
Hi,
I played with bpf/xdp to prototype an UDP loadbalancer.
For UDP checksum, I reused your ipv4_l4_csum
function, but I fall in a verifier issue :
R? min value is negative, either use unsigned or 'var &= const'
We currently discussing about this tough issue on bcc mailing list.
I would like to know if you already faced this kind of issue and you have some input about that ?
Thx
[katran machine(Ubuntu 18.04 kvm guest) info]
root@jockey-test:/home/ubuntu/katran/example_grpc# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 8
root@jockey-test:/home/ubuntu/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac fa:16:3e:6d:25:0b -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=eth0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000
I0821 11:33:39.125748 8470 KatranGrpcService.cpp:67] Starting Katran
E0821 11:33:39.125957 8470 BpfLoader.cpp:160] Can't read section size for index: 2
I0821 11:33:39.125982 8470 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/balancer_kern.o
I0821 11:33:39.158964 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0821 11:33:39.158993 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0821 11:33:39.159008 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0821 11:33:39.159010 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0821 11:33:39.159013 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0821 11:33:39.166762 8470 BpfLoader.cpp:160] Can't read section size for index: 2
I0821 11:33:39.166790 8470 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
E0821 11:33:39.166806 8470 BpfLoader.cpp:160] Can't read section size for index: 12
I0821 11:33:39.166810 8470 BpfLoader.cpp:434] Skipping section: 12 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
I0821 11:33:39.167026 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0821 11:33:39.167047 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0821 11:33:39.167052 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0821 11:33:39.167070 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0821 11:33:39.167073 8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0821 11:33:39.170778 8470 BpfLoader.cpp:97] Can't find name w/ name: lpm_src_v4
E0821 11:33:39.170791 8470 BpfLoader.cpp:97] Can't find name w/ name: decap_dst
Server listening on 0.0.0.0:50051
I0821 11:34:03.237671 8479 KatranLb.cpp:429] adding new vip: 10.200.200.1:22:6
I0821 11:36:18.393283 8487 KatranLb.cpp:429] adding new vip: 10.200.200.1:23:6
root@jockey-test:/home/ubuntu/katran/example_grpc# ./katran_goclient -l
2018/08/21 11:48:52 vips len 2
VIP: 10.200.200.1 Port: 23 Protocol: tcp
Vip's flags:
->10.0.100.218 weight: 1
VIP: 10.200.200.1 Port: 22 Protocol: tcp
Vip's flags:
->10.0.100.218 weight: 1
root@jockey-test:/home/ubuntu/katran/example_grpc# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4076 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:0c:0f:46 brd ff:ff:ff:ff:ff:ff
prog/xdp id 206548 tag 9ac36823edf016c9 jited
3: br-87c747560664: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:05:b4:52:71 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:8b:17:c6:a4 brd ff:ff:ff:ff:ff:ff
17: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
18: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
19: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/tunnel6 :: brd ::
20: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/tunnel6 :: brd ::
When client telnet to VIP 10.200.200.1 port 22, katran_goclient prints "lru miss 100.00%" line.
root@jockey-test:/home/ubuntu/katran/example_grpc# ./katran_goclient -s -lru
summary: 21 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 21 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
What is the policy for open sourced version of katran regarding kernel's support. do you guys pro actively testing things on 4.15 (ubuntu 18.04. which seems officially supported by katran. at least all examples are using it) or not? do you consider it as regression if newer version of code does not work on it (4.15) and will try to fix it? or ? (i'm looking on new patches for quic conn id 24 bit and i have a feeling that verifier wont be happy w/ this patches on 4.15. were they tested on unbuntu 18.04?)
hi,
as example:
myclient(10.43.1.208)---->(220.248.68.38 interface enp4s0f1)(vip: 220.248.68.37)katran_server(lan_10.24.247.28)---real server(10.24.247.31)
on myclient:
telnet 220.248.68.37 22
no respone and i dont know whats wrong with me?
#start command:
/root/katran/build/katran/lib/xdproot -bpfprog /root/katran/deps/linux/bpfprog/bpf/xdp_root.o -bpfpath=/sys/fs/bpf/jmp_enp4s0f1 -intf=enp4s0f1
E0814 09:52:57.754595 14135 BpfLoader.cpp:160] Can't read section size for index: 2
E0814 09:52:57.755061 14135 BpfLoader.cpp:160] Can't read section size for index: 12
can't pin root array
root@ubnutu:~/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac 10:f3:11:7a:b1:96 -forwarding_cores=0 -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=enp4s0f1 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000 -map_path /sys/fs/bpf/jmp_enp4s0f1 -prog_pos=2
I0814 10:00:01.356066 14182 KatranGrpcService.cpp:67] Starting Katran
E0814 10:00:01.357009 14182 BpfLoader.cpp:160] Can't read section size for index: 2
I0814 10:00:01.357031 14182 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/balancer_kern.o
I0814 10:00:01.417520 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0814 10:00:01.417557 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0814 10:00:01.417563 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0814 10:00:01.417569 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0814 10:00:01.417574 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0814 10:00:01.435165 14182 BpfLoader.cpp:160] Can't read section size for index: 2
I0814 10:00:01.435199 14182 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
E0814 10:00:01.435222 14182 BpfLoader.cpp:160] Can't read section size for index: 12
I0814 10:00:01.435230 14182 BpfLoader.cpp:434] Skipping section: 12 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
I0814 10:00:01.435448 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0814 10:00:01.435462 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0814 10:00:01.435469 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0814 10:00:01.435479 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0814 10:00:01.435485 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0814 10:00:01.443327 14182 BpfLoader.cpp:97] Can't find name w/ name: lpm_src_v4
E0814 10:00:01.443353 14182 BpfLoader.cpp:97] Can't find name w/ name: decap_dst
Server listening on 0.0.0.0:50051
root@ubnutu:# ./katran_goclient -l# ping 10.24.247.31
2018/08/14 05:49:25 vips len 1
VIP: 220.248.68.37 Port: 22 Protocol: tcp
Vip's flags:
->10.24.247.31 weight: 1
exiting
root@ubnutu:
PING 10.24.247.31 (10.24.247.31) 56(84) bytes of data.
64 bytes from 10.24.247.31: icmp_seq=1 ttl=64 time=0.399 ms
root@ubnutu:~# ./katran_goclient -s -lru
summary: 14 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 14 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
#on real server 10.24.247.31
[root@node3:~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 220.248.68.37/32 scope global lo
valid_lft forever preferred_lft forever
Hi folks,
I have a bunch of servers that use 802.3ad bonding. Can I just run a separate instance of katran for each member ethernet nic? (dividing up the interrupts and such correctly, of course.) If I understand correctly, Katran is low enough in the stack where it wouldn't know or care about the bonding. Is this correct?
Thanks!
Hi:
I configure 1 VIP and 2 real servers with weight value 1
and 10
.
But the 2 real servers filled in ch_rings
bpf map have almost the same number (The ratio is 1:1
).
I think this line in the source code is the reason. Anyone can tell me the reason why assigning 1
to the weight of the endpoint ?
HI, the recent study of your project kartan, I installed in the test environment, according to your EXAMPLE.md file, I find the source code did not find hc_it_client.py, you can provide this file. Thank you!
From my test, if the packet from client has no room for adding ipip header, Katran can't handle it. Is it right? I didn't see any codes in xdp-balancer to handle this situation. Can it be fixed?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.