facebookincubator/katran

A high performance layer 4 load balancer

katran's Issues

Question about connection tracking

Since connection tracking is implemented by per cpu lru maps, is there affinity of incoming packets to individual cpus? In other words, if a source ip and port pair is assigned a real on a cpu, will another cpu process a packet from the same source ip and port later and assign a different real?

lru_maps_mapping is BPF_MAP_TYPE_ARRAY_OF_MAPS containing a BPF_MAP_TYPE_LRU_HASH map per cpu. Will BPF_MAP_TYPE_LRU_HASH or BPF_MAP_TYPE_LRU_PERCPU_HASH with the BPF_F_NO_COMMON_LRU flag achieve the same goal?

2a8eb124625a7d316f64374501c6201067a523ec breakes testing on ubuntu 18.04

is 18.04 still supported? what the official position of FB folks about it?

I0303 07:28:53.732266 28170 katran_tester.cpp:378] Testing counter's sanity. Printing on errors only
I0303 07:28:53.732388 28170 katran_tester.cpp:418] incorrect stats about katran library internals: number of failed bpf syscalls is non zero
E0303 07:28:53.732395 28170 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0303 07:28:53.732400 28170 KatranLb.cpp:1350] Error gathering map stats for vip_map
I0303 07:28:53.732403 28170 katran_tester.cpp:356] vip_map: current size is incorrect: 0
E0303 07:28:53.732409 28170 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0303 07:28:53.732411 28170 KatranLb.cpp:1350] Error gathering map stats for reals
I0303 07:28:53.732414 28170 katran_tester.cpp:356] reals: current size is incorrect: 0

build simple_katran_client failed

[ 95%] Building CXX object example/CMakeFiles/simple_katran_client.dir/client/Main.cpp.o
collect2: error: ld returned 1 exit status
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:134: recipe for target 'example_grpc/katran_server_grpc' failed
make[2]: *** [example_grpc/katran_server_grpc] Error 1
CMakeFiles/Makefile2:1242: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 97%] Linking CXX executable simple_katran_client
collect2: error: ld returned 1 exit status
example/CMakeFiles/simple_katran_client.dir/build.make:147: recipe for target 'example/simple_katran_client' failed
make[2]: *** [example/simple_katran_client] Error 1
CMakeFiles/Makefile2:858: recipe for target 'example/CMakeFiles/simple_katran_client.dir/all' failed
make[1]: *** [example/CMakeFiles/simple_katran_client.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Benchmarks

Good day, I was wondering if you could add some benchmarks that includes throughput and requests a second (or packets per second) at various level of concurrency with a single node. Bonus if the benchmark compares it's results to other layer 4 load balancers to get relative speed differences

Repo General Questions

I've been playing around with the katran library for a few days now and I was curious about the following things:

1. The repo (once built) includes fully functioning example servers. Is there a reason these haven't been moved from "examples" to just top level apps provided as a full L4 load balancing solutions within this repo? (By that I mean something like moving example_grpc to just be the standard provided katran server.)

2. Since these are only examples right now, is it advised to only take these and use them in a library form / for testing and users should always be building their own full server using the katran lib?

3. I made a simple prometheus metrics exporter for the example_grpc server, is this something that you would want added to this repo?

4. Do you have any examples of how to create a pcap after you have built with KATRAN_INTROSPECTION enabled?

Typo in example?

I am following the example of https://github.com/facebookincubator/katran/blob/master/EXAMPLE.md and under Configuration of forwarding plane section step 4:

VIP must be configured on the real
$ sudo ip a a 10.200.200.1/32 dev lo

Should the VIP is on the server instead of the real?

performance of katran.

I tested the performance of katran, there is no difference between driver mode and generic mode, and they are all lower than lvs.

The configuration of lb machine:

$ nproc 
56
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz

$ cpu_layout.py
======================================================================
Core and Socket Information (as reported by '/sys/devices/system/cpu')
======================================================================

cores =  [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14]
sockets =  [0, 1]

        Socket 0        Socket 1
        --------        --------
Core 0  [0, 28]         [14, 42]
Core 1  [1, 29]         [15, 43]
Core 2  [2, 30]         [16, 44]
Core 3  [3, 31]         [17, 45]
Core 4  [4, 32]         [18, 46]
Core 5  [5, 33]         [19, 47] 
Core 6  [6, 34]         [20, 48]
Core 8  [7, 35]         [21, 49]
Core 9  [8, 36]         [22, 50]
Core 10 [9, 37]         [23, 51]
Core 11 [10, 38]        [24, 52]
Core 12 [11, 39]        [25, 53]
Core 13 [12, 40]        [26, 54]
Core 14 [13, 41]        [27, 55]

$ lspci
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
        Subsystem: Huawei Technologies Co., Ltd. Device d110
        Physical Slot: 0-2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 26
        NUMA node: 0
        Region 0: Memory at 92200000 (64-bit, prefetchable) [size=2M]
        Region 2: I/O ports at 2020 [size=32]
        Region 4: Memory at 92404000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at 92480000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

02:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
        Subsystem: Huawei Technologies Co., Ltd. Device d110
        Physical Slot: 0-2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin B routed to IRQ 152
        NUMA node: 0
        Region 0: Memory at 92000000 (64-bit, prefetchable) [size=2M]
        Region 2: I/O ports at 2000 [size=32]
        Region 4: Memory at 92400000 (64-bit, prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       63
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       8

$ ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       63
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       8

# irqbalance is disabled.
# smp affinity of eth0: (0-7)
95 000000,00000001
96 000000,00000002
97 000000,00000004
98 000000,00000008
99 000000,00000010
100 000000,00000020
101 000000,00000040
109 000000,00000080

# smp affinity of eth1: (8-15)
102 000000,00000100
103 000000,00000200
104 000000,00000400
105 000000,00000800
106 000000,00001000
107 000000,00002000
108 000000,00004000
160 000000,00008000

configured service:

$ ./katran_goclient -l -rpc_port 50052
2018/12/29 20:37:37 vips len 1
VIP:         10.210.9.103 Port:     80 Protocol: tcp
Vip's flags: 
 ->172.30.4.8        weight: 1
 ->172.30.4.7        weight: 1
 ->172.30.4.4        weight: 1
 ->10.210.9.100      weight: 1
 ->10.210.9.102      weight: 1

$ ./katran_goclient -l -rpc_port 50051
2018/12/29 20:37:38 vips len 1
VIP:         10.210.9.103 Port:     80 Protocol: tcp
Vip's flags: 
 ->172.30.4.8        weight: 1
 ->172.30.4.7        weight: 1
 ->172.30.4.4        weight: 1
 ->10.210.9.100      weight: 1
 ->10.210.9.102      weight: 1

two clients ran wrk.

summary stats of lb:

$ ./katran_goclient -s -sum  -rpc_port 50051
summary: 1281055 pkts/sec 98705851 bytes/sec
summary: 1176094 pkts/sec 90136858 bytes/sec
summary: 1278174 pkts/sec 97861917 bytes/sec
summary: 1279117 pkts/sec 97803088 bytes/sec


$ ./katran_goclient -s -sum  -rpc_port 50052
summary: 856653 pkts/sec 65913453 bytes/sec
summary: 842773 pkts/sec 64840720 bytes/sec
summary: 847563 pkts/sec 65222415 bytes/sec
summary: 880912 pkts/sec 67892596 bytes/sec

cpu usage:

%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu8  :  0.0 us,  0.0 sy,  0.0 ni,  6.4 id,  0.0 wa,  0.0 hi, 93.6 si,  0.0 st
%Cpu9  :  0.0 us,  0.0 sy,  0.0 ni,  9.2 id,  0.0 wa,  0.0 hi, 90.8 si,  0.0 st
%Cpu10 :  0.0 us,  0.0 sy,  0.0 ni,  9.8 id,  0.0 wa,  0.0 hi, 90.2 si,  0.0 st
%Cpu11 :  0.0 us,  0.0 sy,  0.0 ni,  9.2 id,  0.0 wa,  0.0 hi, 90.8 si,  0.0 st
%Cpu12 :  0.0 us,  0.0 sy,  0.0 ni, 19.0 id,  0.0 wa,  0.0 hi, 81.0 si,  0.0 st
%Cpu13 :  0.0 us,  0.0 sy,  0.0 ni, 27.3 id,  0.0 wa,  0.0 hi, 72.7 si,  0.0 st
%Cpu14 :  0.0 us,  0.0 sy,  0.0 ni, 40.5 id,  0.0 wa,  0.0 hi, 59.5 si,  0.0 st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni, 38.5 id,  0.0 wa,  0.0 hi, 61.5 si,  0.0 st

client results:

$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t  -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
  32 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   112.87ms  261.00ms   1.86s    89.40%
    Req/Sec    29.17k     3.76k   65.05k    77.06%
  61118879 requests in 1.10m, 9.84GB read
  Socket errors: connect 0, read 3, write 380, timeout 38989
Requests/sec: 928741.35
Transfer/sec:    153.08MB

$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t  -v                       
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
  32 threads and 10000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   106.98ms  261.37ms   1.99s    90.09%
    Req/Sec    38.68k     9.50k  114.19k    86.95%
  90094314 requests in 1.22m, 14.50GB read
  Socket errors: connect 0, read 3, write 112, timeout 37468
Requests/sec: 1231554.03
Transfer/sec:    202.99MB

Same machine and same configuration with lvs:

$ sudo ipvsadm -l -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.210.9.103:80 wrr
  -> 172.30.4.4:80                Tunnel  1      0          0         
  -> 172.30.4.7:80                Tunnel  1      0          0         
  -> 172.30.4.8:80                Tunnel  1      0          0         
  -> 10.210.9.100:80              Tunnel  1      0          0         
  -> 10.210.9.102:80              Tunnel  1      0          0  

cpu usage:

%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,  5.2 id,  0.0 wa,  0.0 hi, 94.8 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,  7.0 id,  0.0 wa,  0.0 hi, 93.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,  5.2 id,  0.0 wa,  0.0 hi, 94.8 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,  5.7 id,  0.0 wa,  0.0 hi, 94.3 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,  5.3 id,  0.0 wa,  0.0 hi, 94.7 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,  6.3 id,  0.0 wa,  0.0 hi, 93.7 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,  7.8 id,  0.0 wa,  0.0 hi, 92.2 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,  3.1 id,  0.0 wa,  0.0 hi, 96.9 si,  0.0 st
%Cpu8  :  0.0 us,  0.0 sy,  0.0 ni, 36.8 id,  0.0 wa,  0.0 hi, 63.2 si,  0.0 st
%Cpu9  :  0.0 us,  0.0 sy,  0.0 ni, 50.4 id,  0.0 wa,  0.0 hi, 49.6 si,  0.0 st
%Cpu10 :  0.0 us,  0.0 sy,  0.0 ni, 50.7 id,  0.0 wa,  0.0 hi, 49.3 si,  0.0 st
%Cpu11 :  0.0 us,  0.0 sy,  0.0 ni, 61.4 id,  0.0 wa,  0.0 hi, 38.6 si,  0.0 st
%Cpu12 :  0.0 us,  0.0 sy,  0.0 ni, 57.8 id,  0.0 wa,  0.0 hi, 42.2 si,  0.0 st
%Cpu13 :  0.0 us,  0.0 sy,  0.0 ni, 63.8 id,  0.0 wa,  0.0 hi, 36.2 si,  0.0 st
%Cpu14 :  0.0 us,  0.0 sy,  0.0 ni, 63.5 id,  0.0 wa,  0.0 hi, 36.5 si,  0.0 st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni, 35.6 id,  0.0 wa,  0.0 hi, 64.4 si,  0.0 st

client results:

$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t  -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
  32 threads and 10000 connections
^C  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.93ms  241.20ms   1.99s    89.54%
    Req/Sec    63.61k    12.08k  159.62k    82.01%
  113727680 requests in 0.94m, 18.31GB read
  Socket errors: connect 0, read 0, write 30, timeout 11326
Requests/sec: 2021869.63
Transfer/sec:    333.26MB

$ wrk -c 10000 -t 32 -d 300s http://10.210.9.103/t  -v
wrk 4.0.2-2-g91655b5 [epoll] Copyright (C) 2012 Will Glozer
Running 5m test @ http://10.210.9.103/t
  32 threads and 10000 connections
^C  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.67ms  241.55ms   1.98s    89.59%
    Req/Sec    65.79k    10.29k  143.66k    82.61%
  119779234 requests in 0.95m, 19.28GB read
  Socket errors: connect 0, read 1, write 44, timeout 12137
Requests/sec: 2093134.23
Transfer/sec:    344.92MB

Is this normal? Is there any way to increase the performance?
Thanks.

libbpf: Error loading BTF: Invalid argument(22)

HI
I got some ERRORs.
any suggestion?

~/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/bpfprog/bpf/balancer_kern.o -default_mac 52:54:00:12:35:02 -forwarding_cores=0 -healthchecker_prog ./deps/bpfprog/bpf/healthchecking_ipip.o -intf=eth0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000

I0423 05:13:18.267051  8437 KatranGrpcService.cpp:67] Starting Katran
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error finalizing and loading .BTF into kernel: -22. Ignored and continue.
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error finalizing and loading .BTF into kernel: -22. Ignored and continue.
E0423 05:13:18.308995  8437 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0423 05:13:18.310480  8437 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
Server listening on 0.0.0.0:50051
^CI0423 05:13:20.733433  8438 GrpcSignalHandler.cpp:37] Signal: 2, stopping service in 10000 milliseconds.
I0423 05:13:30.743686  8438 GrpcSignalHandler.cpp:42] Stopping Katran!

Failed to build with build_katran.sh

Hi,

I am trying to build Katran with the latest commit 23add38cf9839b2bacafbf7330767853f69c3810 and Ubuntu 18.04, but it is failed due to the linkage error below. This is also the same issue Travis CI build has encountered.

Scanning dependencies of target katran_server_grpc
[ 98%] Building CXX object example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o
[100%] Linking CXX executable katran_server_grpc
../../deps/grpc/build/libgpr.a(host_port.cc.o): In function grpc_core::(anonymous namespace)::DoSplitHostPort(absl::string_view, absl::string_view*, absl::string_view*, bool*)': /home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:77: undefined reference to absl::string_view::find(char, unsigned long) const'
/home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:79: undefined reference to absl::string_view::find(char, unsigned long) const' /home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:53: undefined reference to absl::string_view::find(char, unsigned long) const'
/home/ubuntu/workspace/katran/_build/deps/grpc/src/core/lib/gprpp/host_port.cc:70: undefined reference to absl::string_view::find(char, unsigned long) const' ../../deps/grpc/build/libgpr.a(host_port.cc.o): In function grpc_core::(anonymous namespace)::DoSplitHostPort(absl::string_view, absl::string_view*, absl::string_view*, bool*)':
/home/ubuntu/workspace/katran/_build/deps/grpc/third_party/abseil-cpp/absl/strings/string_view.h:382: undefined reference to `absl::base_internal::ThrowStdOutOfRange(char const*)'
collect2: error: ld returned 1 exit status
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:154: recipe for target 'example_grpc/katran_server_grpc' failed
make[2]: *** [example_grpc/katran_server_grpc] Error 1
CMakeFiles/Makefile2:2101: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

Questions about forwarding mode

Hi developers. I have some experience with traditional LVS technology but just started to learn about this library, so please bear with me if my questions are not on the right track.

From the documentation I learned katran's forwarding mode is a bit like LVS' TUN mode, which needs L4 machines to wrap packets to IPIP protocol and requires the L7 have public network reachability. Will tunneling be a huge computational overhead for the L4 machines?

I think in most of deployment scenarios, L7 and L4 are just in the same local network and doesn't need IPIP to work. Why not support something like LVS's

• DR: L4 -direct routing-> L7 -> Client. Like TUN, but requires L4 and L7 to be L2 reachable and doesn't require tunneling. Or
• FullNAT: L4 -SNAT-> L7 -> L4 -DNAT-> Client. Doesn't require L2 reachability but packets needs to be returned to L4 to reach client

mode?

What is the design considerations of using tunnel? Thanks.

./build_katran.sh error

uname -a
Linux kruztw 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

gcc --version
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE

Question about lru

In bpf files, lru seems the cache of get_packet_dst result. How can lru handle packets when dst real server already down, but old dst still in lru map? New connection with same 5-tuple will still find the old lru dst.
Other question, why lru only handle udp timeout in connection_table_lookup, how about tcp connections?

Linux networking stack can't forward ipip packet.

Hi,

I am trying to reproduce the example here using virtual machines (i.e., client, server, router, katran). And the below diagram illustrates my environment:

I am using ubuntu 18.04 with the kernel 4.15.0-55-generic for the virtual machines.

client --(net1)-- router --(net2)-- katran
                    |
                 (net3)
                    |
                 server

net1: 192.168.150.0/24
net2: 10.10.10.0/24
net3: 50.10.10.0/24

client ip: 192.168.150.10
katran ip: 10.10.10.10
server ip: 50.10.10.10
router ip:
    net1-if: 192.168.150.1
    net2-if: 10.10.10.1
    net3-if: 50.10.10.1

The routing table of the router is as below:

10.10.10.0/24 dev ens5 proto kernel scope link src 10.10.10.1
20.200.200.1 via 10.10.10.1 dev ens5
30.10.10.0/24 dev ens7 proto kernel scope link src 30.10.10.12
50.10.10.0/24 dev ens6 proto kernel scope link src 50.10.10.1
192.168.150.0/24 dev ens2 proto kernel scope link src 192.168.150.1

It seems the katran sends ipip packet correctly to the router after resolving VIP (i.e., 20.200.200.1). Here is a tcpdump from the router

15:51:55.996433 52:54:00:23:97:ef > 52:54:00:12:e8:4c, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 80)
    172.16.242.206 > 50.10.10.10: (tos 0x0, ttl 63, id 21090, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.150.10.50276 > 20.200.200.1.22: Flags [S], cksum 0x7ddd (correct), seq 1289943810, win 29200, options [mss 1460,sackOK,TS val 2434696983 ecr 0,nop,wscale 7], length 0
15:51:57.017960 52:54:00:23:97:ef > 52:54:00:12:e8:4c, ethertype IPv4 (0x0800), length 94: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 80)
    172.16.242.206 > 50.10.10.10: (tos 0x0, ttl 63, id 21091, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.150.10.50276 > 20.200.200.1.22: Flags [S], cksum 0x79df (correct), seq 1289943810, win 29200, options [mss 1460,sackOK,TS val 2434698005 ecr 0,nop,wscale 7], length 0

But the problem is the router can't forward ipip packets to the server. It drops the packets before it calls the ip_recv() as shown the below kernel trace log. I expect the networking stack of the router calls the ip_recv() function to handle ipip packet.

This is the log of kernel call trace from the router when it receives ipip packets:

swapper     0 [003]  2339.377884: skb:kfree_skb: skbaddr=0xffff913f34574600 protocol=4 location=0xffffffffa6e5371f
            7fffa6e40696 kfree_skb ([kernel.kallsyms])
            7fffa6e5571f __netif_receive_skb_core ([kernel.kallsyms])
            7fffa6e55c08 __netif_receive_skb ([kernel.kallsyms])
            7fffa6e57f55 netif_receive_skb_internal ([kernel.kallsyms])
            7fffa6e58c55 napi_gro_receive ([kernel.kallsyms])
            7fffc004145a cp_rx_poll ([kernel.kallsyms])
            7fffa6e58380 net_rx_action ([kernel.kallsyms])
            7fffa72020e4 __softirqentry_text_start ([kernel.kallsyms])
            7fffa66969b8 irq_exit ([kernel.kallsyms])
            7fffa700462a do_IRQ ([kernel.kallsyms])
            7fffa7002904 ret_from_intr ([kernel.kallsyms])
            7fffa6fa3672 native_safe_halt ([kernel.kallsyms])
            7fffa6fa3330 default_idle ([kernel.kallsyms])
            7fffa663b035 arch_cpu_idle ([kernel.kallsyms])
            7fffa6fa3803 default_idle_call ([kernel.kallsyms])
            7fffa66d6db2 do_idle ([kernel.kallsyms])
            7fffa66d7013 cpu_startup_entry ([kernel.kallsyms])
            7fffa665b37b start_secondary ([kernel.kallsyms])
            7fffa66020d5 secondary_startup_64 ([kernel.kallsyms])

Am I missing something to config the router?

thx~
Junho

License question

Are you folks set on using GNU 2.0 as a license? Are you willing to consider using MIT or Apache?

katran server not sending the packets to the backend real servers

Hi,

I am following the example setup as described in - https://github.com/facebookincubator/katran/blob/master/EXAMPLE.md. There are two ubuntu servers.
Server-1
Ubuntu 18.04.1 kernel version - 4.18.0.
This is to host katran server. Interface to bind ens4f0.201. IP assigned to this interface is IP-1.
Server-2
Ubuntu 16.04.4 LTS. Kernel version 4.10.0-27-generic.
This is to host back end real servers. Interface to bind is ens4f0.116. IP assigned to this interface is IP-2.

Ping is working fine between these two servers over these interfaces.

---

1. "sudo ip link add name ipip0 type ipip external" command doesn't work on the Server-2. Not sure because of Ubuntu 16.04. Hence i modified the tunnel creation command in both the server-1( hosting Katran server) and server-2 ( backend real server) as follows -

Server-1 ( hosting Katran server)
sudo ip tunnel add tun0 mode ipip local IP-1 remote IP-2 dev ens4f0.201
sudo ip link set up dev tun0

Server-2 ( hosting real backend servers)
sudo ip tunnel add tun0 mode ipip local IP-2 remote IP-1 dev ens4f0.116
sudo ip link set up dev tun0

Katan Server started -
sudo ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac -forwarding_cores=0 -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=ens4f0.201 -ipip_intf=tun0 -ipip6_intf=ipip60 -lru_size=10000

Note: i don't have IPv6 and hence here ipip60 is dummy. Without -ipip6_intf , it crashes.

VIP and real server IP added.
katran-master/example_grpc/goclient/bin$ ./main -A -t :22
2018/08/30 00:41:20 Adding service: 10.38.42.103:22 6
2018/08/30 00:41:20 Vip modified
exiting
katran-master/example_grpc/goclient/bin$ ./main -a -t :22 -r IP-2

Note: VIP is another IP from the same subnet of IP-1.

When ssh is attempted to the VIP 10.38.42.103, it's not working. Can you please tell me what is wrong in the setup?"ip a" doesn't show that VIP is bound to the interface where Katran server is running.

2. As mentioned in documentation, "(katran works in DSR (Direct Sever Return) mode)". As i can see, Katran is sending the packets in IPIP tunnel to the back end server. Does it support forwarding the packets with just changing the destination MAC?

Include port in real definition

Hello, I'd like to make it possible to redirect packets to the port other than vip's port.
eg:
user --> katran(192.168.0.42:80) --> ... --> real(10.0.0.2:8080)

Could you please provide me a high level guide what needs to be changed in order to make it work? I will try to implement it.

About healthchecking

HI， VIP gets announced using OSPF(quagga) and is set at the interface lo. ipip tunnels are present and up.
Because VIP was set at lo interface, healthcheck packets were not transmitted outside from enp0s3.
but, healthchecking_ipip.o work in enp0s3. What are the ways to solve this problem? Thanks.

travis is broken

is it a requirements for .travis.yml to be the way it is right now for ship-it/import-it to work? right now, because of this, travis make no sense and all PR/scheduled build are red. could we (at least temporary, till fbbuild would be able to work w/ katran) return to previous version of it?

Documentation question

Hi, and sorry in advance for being naive. I have an interesting application for Katran and was wondering if you could let me know whether you think Katran would be a good fit.

I need to collect and encapsulate packets relating to TCP sessions which may be received at any of a number of distinct geographically distributed egress points from a large IP network. Packets returning from the internet relating to a particular TCP session may appear at any of a number of different edge routers.

I'm wondering if I could install Katran in each egress data center and configure it to encapsulate each TCP session and route it to a consistent server within a centralized cluster of processing servers somewhere inside the network. In reality, the centralized cluster is itself a TCP-to-TCP proxy that understands the relevant L7 protocol; however, we only need Katran to help with marshalling the packets from the edge. Once we're inside the network, the network's fabric can route packets properly to their destination from the filtering servers.

It seems like this should be possible because from Katran's point of view, whether Katran is literally at the "top of the rack" or in a completely different rack somewhere else in the world should make no difference. Note that latency here is not a real concern given the underlying protocol we are proxying.

BpfAdapter.cpp:472] Error receiving netlink message: Operation not supported [95]

ip checksum issue ?

Hi,
I'm pretty much a novice in XDP and low level network protocol but reading rfc791 I understand that checksum should be calculate using the whole ip header.

E.g. :

 Checksum

    The internet header checksum is recomputed if the internet header is
    changed.  For example, a reduction of the time to live, additions or
    changes to internet options, or due to fragmentation.  This checksum
    at the internet level is intended to protect the internet header
    fields from transmission errors.

So would it be more correct to calculate checksum using iph->ihl instead of sizeof(struct iphdr) in balancer_helpers.h ?

(bonus question why using >> 1 instead of / 2 ? 2nd solution seems semantically more appropriate, is there any reason or this is just a programming preference ?)

small probability outer ip header checksum error

we test katran in our environment。and find some outer ip header checksum error about （1/100000）rate 。
I check the code in katran/lib/bpf/pckt_encap.h。
in some Small probability 。(csum & 0xffff) + (csum >> 16) may be overflow in 16bit。 it‘s may cause the problem I had mention

after fix code under groud。 this problem disappear。

- iph->check = ~((csum & 0xffff) + (csum >> 16));
+ csum = (csum & 0xffff) + (csum >> 16);
+ csum += (csum >>16);
+ iph->check = ~(csum);

thanks。

os_run_tester.sh errors

I gots some ERRORs when I executed os_run_tester.sh.

bule@bule:~/katran$ ./os_run_tester.sh
+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/bule/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/bule/katran/_build/deps
+ sudo sh -c '/home/bule/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error loading .BTF into kernel: -22.
E0304 14:05:02.500411  5329 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0304 14:05:02.500869  5329 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E0304 14:05:02.501143  5329 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E0304 14:05:02.501462  5329 BpfLoader.cpp:87] Can't find map w/ name: pckt_srcs
E0304 14:05:02.501798  5329 BpfLoader.cpp:87] Can't find map w/ name: hc_pckt_srcs_map
E0304 14:05:02.502049  5329 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0304 14:05:02.502434  5329 KatranLb.cpp:1350] Error gathering map stats for vip_map
E0304 14:05:02.502846  5329 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0304 14:05:02.503111  5329 KatranLb.cpp:1350] Error gathering map stats for reals
I0304 14:05:02.503408  5329 katran_tester.cpp:356] reals: current size is incorrect: 0
I0304 14:05:02.503669  5329 katran_tester.cpp:368] Initial testing of counters is complete
I0304 14:05:02.503984  5329 KatranLb.cpp:587] adding new vip: 10.200.1.1:80:17
I0304 14:05:02.664216  5329 KatranLb.cpp:587] adding new vip: 10.200.1.1:80:6
I0304 14:05:02.827368  5329 KatranLb.cpp:587] adding new vip: 10.200.1.2:0:6
I0304 14:05:03.000072  5329 KatranLb.cpp:587] adding new vip: 10.200.1.4:0:6
I0304 14:05:03.156711  5329 KatranLb.cpp:653] modyfing vip: 10.200.1.4:0:6
I0304 14:05:03.157140  5329 KatranLb.cpp:587] adding new vip: 10.200.1.3:80:6
I0304 14:05:03.319754  5329 KatranLb.cpp:587] adding new vip: fc00:1::1:80:6
I0304 14:05:03.477419  5329 KatranLb.cpp:587] adding new vip: 10.200.1.5:443:17
I0304 14:05:03.478024  5329 KatranLb.cpp:653] modyfing vip: 10.200.1.5:443:17
I0304 14:05:03.636570  5329 KatranLb.cpp:587] adding new vip: fc00:1::2:443:17
I0304 14:05:03.637153  5329 KatranLb.cpp:653] modyfing vip: fc00:1::2:443:17
I0304 14:05:03.794124  5329 BpfTester.cpp:214] Test: packet to UDP based v4 VIP (and v4 real)                     result: Passed
I0304 14:05:03.794591  5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real)                     result: Passed
I0304 14:05:03.794898  5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real) + ToS in IPV4       result: Passed
I0304 14:05:03.795253  5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v4 real; any dst ports).     result: Passed
I0304 14:05:03.795575  5329 BpfTester.cpp:214] Test: packet to TCP based v4 VIP (and v6 real)                     result: Passed
I0304 14:05:03.795858  5329 BpfTester.cpp:214] Test: packet to TCP based v6 VIP (and v6 real)                     result: Passed
I0304 14:05:03.796102  5329 BpfTester.cpp:214] Test: packet to TCP based v6 VIP (and v6 real) with ToS / tc set   result: Passed
I0304 14:05:03.796380  5329 BpfTester.cpp:214] Test: v4 ICMP echo-request                                         result: Passed
I0304 14:05:03.796711  5329 BpfTester.cpp:214] Test: v6 ICMP echo-request                                         result: Passed
I0304 14:05:03.797286  5329 BpfTester.cpp:214] Test: v4 ICMP dest-unreachabe fragmentation-needed                 result: Passed
I0304 14:05:03.797821  5329 BpfTester.cpp:214] Test: v6 ICMP packet-too-big                                       result: Passed
I0304 14:05:03.798318  5329 BpfTester.cpp:214] Test: drop of IPv4 packet w/ options                               result: Passed
I0304 14:05:03.798739  5329 BpfTester.cpp:214] Test: drop of IPv4 fragmented packet                               result: Passed
I0304 14:05:03.799027  5329 BpfTester.cpp:214] Test: drop of IPv6 fragmented packet                               result: Passed
I0304 14:05:03.799335  5329 BpfTester.cpp:214] Test: pass of v4 packet with dst not equal to any configured VIP   result: Passed
I0304 14:05:03.799625  5329 BpfTester.cpp:214] Test: pass of v6 packet with dst not equal to any configured VIP   result: Passed
I0304 14:05:03.799966  5329 BpfTester.cpp:214] Test: pass of arp packet                                           result: Passed
I0304 14:05:03.800016  5329 BpfTester.cpp:214] Test: LRU hit                                                      result: Passed
I0304 14:05:03.800035  5329 BpfTester.cpp:214] Test: packet #1 dst port hashing only                              result: Passed
I0304 14:05:03.800050  5329 BpfTester.cpp:214] Test: packet #2 dst port hashing only                              result: Passed
I0304 14:05:03.800062  5329 BpfTester.cpp:214] Test: ipinip packet                                                result: Passed
I0304 14:05:03.800076  5329 BpfTester.cpp:214] Test: ipv6inipv6 packet                                            result: Passed
I0304 14:05:03.800089  5329 BpfTester.cpp:214] Test: ipv4inipv6 packet                                            result: Passed
I0304 14:05:03.800107  5329 BpfTester.cpp:214] Test: QUIC: long header. Client Initial type. LRU miss             result: Passed
I0304 14:05:03.800124  5329 BpfTester.cpp:214] Test: QUIC: long header. 0-RTT Protected. CH. LRU hit.             result: Passed
I0304 14:05:03.800452  5329 BpfTester.cpp:214] Test: QUIC: long header. Handshake. v4 vip v6 real. Conn Id based. result: Passed
I0304 14:05:03.800478  5329 BpfTester.cpp:214] Test: QUIC: long header. Retry. v4 vip v6 real. Conn Id based.     result: Passed
I0304 14:05:03.800493  5329 BpfTester.cpp:214] Test: QUIC: long header. client initial. v6 vip v6 real. LRU miss  result: Passed
I0304 14:05:03.800504  5329 BpfTester.cpp:214] Test: QUIC: short header. No connection id. CH. LRU hit            result: Passed
I0304 14:05:03.800515  5329 BpfTester.cpp:214] Test: QUIC: short header w/ connection id                          result: Passed
I0304 14:05:03.800526  5329 BpfTester.cpp:214] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I0304 14:05:03.800536  5329 BpfTester.cpp:214] Test: QUIC: short header w/ conn id. host id = 0. CH. LRU hit      result: Passed
I0304 14:05:03.800544  5329 katran_tester.cpp:378] Testing counter's sanity. Printing on errors only
I0304 14:05:03.800731  5329 katran_tester.cpp:418] incorrect stats about katran library internals: number of failed bpf syscalls is non zero
E0304 14:05:03.800858  5329 BpfAdapter.cpp:356] Error while retrieving map metadata for vip_map: Unknown error -1
E0304 14:05:03.800873  5329 KatranLb.cpp:1350] Error gathering map stats for vip_map
I0304 14:05:03.800879  5329 katran_tester.cpp:356] vip_map: current size is incorrect: 0
E0304 14:05:03.800889  5329 BpfAdapter.cpp:356] Error while retrieving map metadata for reals: Unknown error -1
E0304 14:05:03.800896  5329 KatranLb.cpp:1350] Error gathering map stats for reals
I0304 14:05:03.800904  5329 katran_tester.cpp:356] reals: current size is incorrect: 0
I0304 14:05:03.800909  5329 katran_tester.cpp:435] Testing of counters is complete
E0304 14:05:03.801021  5329 KatranSimulator.cpp:169] src and dst must have same address family
E0304 14:05:03.801036  5329 KatranSimulator.cpp:162] malformed src or dst ip address. src: aaaa dst: bbbb
E0304 14:05:03.801044  5329 BpfLoader.cpp:97] Can't find prog with name: cls-hc
I0304 14:05:03.801050  5329 katran_tester.cpp:309] Healthchecking not enabled. Skipping HC related tests
+ sudo sh -c '/home/bule/katran/_build/build/katran/decap/testing/xdpdecap_tester -decap_prog /home/bule/katran/_build/deps/bpfprog/bpf/decap_kern.o -test_from_fixtures=true '
libbpf: failed to open /home/bule/katran/_build/deps/bpfprog/bpf/decap_kern.o: No such file or directory
Segmentation fault (core dumped)

any sugguest?
thanks

katran xdp issue when running with bonding in driver mode

I have a server that use 802.3ad bonding, and there's an issue in driver mode.

Since there were some troubles with building katran's dependences on my vm(out of memory when building fizz), so i wrote a balancer-user program(modified from kernel's bpf sample: xdp_tx_iptunnel_user.c) and complied it with katran's bpf files. The 'balancer_user.c' and some modifies are here:
balancer_user.c.txt
balaner_kern.diff.txt

It is only used to do a test. It will load katran's bpf file and set vip and rs, and only works when running seprate instance on per link in "generic xdp" mode.

Below is my env.
katran's version is 4c1fe70.

$ ifconfig
bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet 172.30.4.3  netmask 255.255.255.0  broadcast 172.30.4.255
        inet6 fe80::ec6:ccff:fe9e:a9ce  prefixlen 64  scopeid 0x20<link>
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 23856910322  bytes 2133122139232 (1.9 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33499171263  bytes 22037784326757 (20.0 TiB)
        TX errors 0  dropped 642 overruns 0  carrier 0  collisions 0

eth0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 1214891  bytes 766042605 (730.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1353364  bytes 719038863 (685.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether aa:bb:cc:dd:ee:ff  txqueuelen 1000  (Ethernet)
        RX packets 23202  bytes 14578863 (13.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25149  bytes 14607499 (13.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2797883824  bytes 6603725657284 (6.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2797883824  bytes 6603725657284 (6.0 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunl0: flags=193<UP,RUNNING,NOARP>  mtu 1480
        inet 172.30.4.253  netmask 255.255.255.255
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

"172.30.4.253" is the vip.

$ cat /etc/sysconfig/network-scripts/ifcfg-bond1
#IP Config for bond1:
DEVICE=bond1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS="mode=4 miimon=100 ad_select=1 updelay=200 xmit_hash_policy=2"
IPV6INIT=no
TYPE=Unknown
NM_CONTROLLED=no
IPADDR='172.30.4.3'
NETMASK='255.255.255.0'
GATEWAY='172.30.4.1'

$ cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
...

$ ethtool -i eth0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x800003e2
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

$ uname -mrs
Linux 4.18.8-1.el7.elrepo.x86_64 x86_64

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:11 brd ff:ff:ff:ff:ff:ff
6: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff

Here are my commands:

$ ./balancer 
Start a XDP prog which encapsulates incoming packets
in an IPv4/v6 header and XDP_TX it out.  The dst <VIP:PORT>
is used to select packets to encapsulate

Usage: ./balancer [...]
    -i <ifindex> Interface Index
    -a <vip-service-address> IPv4 or IPv6
    -p <vip-service-port> Service  port
    -s <source-ip> Used in the IPTunnel header
    -d <dest-ip> Used in the IPTunnel header, ip range (e.g. 1.1.1.1,2.2.2.2) is also allowed
    -m <dest-MAC> Used in sending the IP Tunneled pkt
    -T <stop-after-X-seconds> Default: 0 (forever)
    -P <IP-Protocol> Default is TCP
    -S use skb-mode
    -N enforce native mode
    -h Display this help

It works fine in generic mode(with '-S'):

sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S

sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -S

the output is:

sum:    318928 pkts, rate:    159464 pkts/s
sum:   4213744 pkts, rate:   2106872 pkts/s
sum:   4241320 pkts, rate:   2120660 pkts/s
sum:   4251751 pkts, rate:   2125875 pkts/s
sum:   4269300 pkts, rate:   2134650 pkts/s
sum:   4288777 pkts, rate:   2144388 pkts/s
sum:   4295820 pkts, rate:   2147910 pkts/s

When using driver mode(with '-N'), it won't work properly.

sudo ./balancer -i 2 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N

sudo ./balancer -i 3 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 172.30.4.4,172.30.4.5,172.30.4.6 -m aa:bb:cc:dd:ee:00 -N

sometimes it works:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connected to 172.30.4.253 (172.30.4.253) port 80 (#0)
> GET /t HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.4.253
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.14.1
< Date: Tue, 25 Dec 2018 07:15:00 GMT
< Content-Type: application/octet-stream
< Content-Length: 2
< Connection: keep-alive
< 
* Connection #0 to host 172.30.4.253 left intact
OK

and sometimes not:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connection timed out
* Failed connect to 172.30.4.253:80; Connection timed out
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection timed out

I tried curl many times, almost 70% was failed.

And also i tried to run with "bond1" in generic mode, it couldn't work.

sudo ./balancer -i 6 -a 172.30.4.253 -p 80 -s 172.30.4.3 -d 10.210.9.100,10.210.9.101,10.210.9.102 -m 68:05:ca:21:d6:e5  -S

the result is:

$ curl 172.30.4.253/t -v
* About to connect() to 172.30.4.253 port 80 (#0)
*   Trying 172.30.4.253...
* Connection refused
* Failed connect to 172.30.4.253:80; Connection refused
* Closing connection 0
curl: (7) Failed connect to 172.30.4.253:80; Connection refused

build_katran.sh compilation error

Hi

I'm trying to build katran on an Ubuntu 18.04 system.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

I'm using gcc version 7.5.0 and the latest commit of katran

$ git log

commit 096536b137bc18da66db946f9312e61dc7e330dc (HEAD -> master, origin/master, origin/HEAD)
Author: svcscm <[email protected]>
Date:   Thu Sep 17 12:14:34 2020 -0700

    Updating submodules
    
    Summary:
    GitHub commits:
    
    https://github.com/facebook/fb303/commit/2cc39c81a4ab352d160c266516c5f1804406e10e
    https://github.com/facebook/fbthrift/commit/d0ee2d31a4606864f2bfc88aaa0dff9d4dcf5467
    https://github.com/facebook/fbzmq/commit/7690bf6b0e664cdc8a242b989f631ce1be1cf595
    https://github.com/facebook/litho/commit/829f75ded87fe92ff926e943644fa193169efd57
    https://github.com/facebook/proxygen/commit/d5c788f4c1f02cb810d458e44483f81f740e7c0e
    https://github.com/facebook/rocksdb/commit/b9750c7c3c8c9b4fe5597a42fd73267c69281fe5
    https://github.com/facebook/wangle/commit/1cf06f76dd7c0d6e0eb7083c81d1872afd40d48f
    https://github.com/facebook/watchman/commit/49c01fccf225c35358995ea44efcf59c45547acc
    https://github.com/facebookexperimental/rust-shed/commit/612fc34d70f911915b092ec89e0cdd6eb272b462
    https://github.com/facebookincubator/fizz/commit/c2c96af540c50d26c2739431ebb8458804f74f8f
    https://github.com/facebookincubator/katran/commit/8e217fa90cac4d4f39db8e1dc3ba7eed11c8d73a
    https://github.com/facebookincubator/mvfst/commit/df966d4ffca17873ab35323b2e0a356e38dbd9b9
    https://github.com/facebookincubator/profilo/commit/e45cb217266da4e0e053d36afd12ffd89b8338f5
    https://github.com/rsocket/rsocket-cpp/commit/f2b05dc5e855570c7f98131c1c6970868ef5b0f3
    
    Reviewed By: yns88
    
    fbshipit-source-id: 765c8b0fce704a6d3503392df73714b3054b1392

Following is the error on running ./build_katran.sh

Scanning dependencies of target katran_server_grpc
[ 98%] Building CXX object example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o
/mydata/local/katran/example_grpc/katran_server.cpp: In function ‘int main(int, char**)’:
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
   };
   ^
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
/mydata/local/katran/example_grpc/katran_server.cpp:127:3: sorry, unimplemented: non-trivial designated initializers not supported
example_grpc/CMakeFiles/katran_server_grpc.dir/build.make:62: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o' failed
make[2]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/katran_server.cpp.o] Error 1
CMakeFiles/Makefile2:2282: recipe for target 'example_grpc/CMakeFiles/katran_server_grpc.dir/all' failed
make[1]: *** [example_grpc/CMakeFiles/katran_server_grpc.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

Please take a look. Thanks!

build script is broken

./build_katran is broken

-- Found folly: /usr/local
-- Found fizz: /usr/local
CMake Error at CMakeLists.txt:30 (find_package):
  By not providing "FindGlog.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Glog", but
  CMake did not find one.

  Could not find a package configuration file provided by "Glog" with any of
  the following names:

    GlogConfig.cmake
    glog-config.cmake

  Add the installation prefix of "Glog" to CMAKE_PREFIX_PATH or set
  "Glog_DIR" to a directory containing one of the above files.  If "Glog"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/home/tehnerd/katran/build/CMakeFiles/CMakeOutput.log".
tehnerd@devubuntu:~/katran$

if i start to install fizz, it became broken on glog (same error) etc etc

what is the blessed way to build open source version of katran?

xdp-balancer not found

root@katran-big:/home/syla/katran# ./os_run_tester.sh

• '[' -z '' ']'
  ++ pwd
• KATRAN_BUILD_DIR=/home/syla/katran/_build/build
• '[' -z '' ']'
  ++ pwd
• DEPS_DIR=/home/syla/katran/_build/deps
• sudo sh -c '/home/syla/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/syla/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
  libbpf: Error loading ELF section .BTF: -22.
  E0212 21:58:01.396488 1322 BpfLoader.cpp:97] Can't find prog with name: xdp-balancer
  terminate called after throwing an instance of 'std::invalid_argument'
  what(): can't get fd for prog: xdp-balancer, error: Invalid argument
  Aborted (core dumped)

How can I use config file to run katran

if i restart katran_server then config data miss take,
if i have to two network interface Wan <--->Lan，how can i configure
like this:
client(172.16.1.100)--->Wan(eth1:172.16.1.101)(vip:172.16.1.200)<--->Lan(eth0:192.168.1.10)--->server(192.168.1.20)

attach healthchecking bpf prog to main inteface failed

Hi folks,
when tesing healthcheck, i found katran terminate with "called after throwing an instance of
std::invalid_argument" with attaching healthcheck bpf to main interface. After added some log, i found it's the mnl_cb_run in modifyTcBpfFilter return -1 and set errno to INVAL.

I have digged into libmnl source https://github.com/threatstack/libmnl/blob/master/src/callback.c while there is no INVAL in in __mnl_cb_run . would anyone help me to figure this out?
Thanks

BpfAdapter.cpp:503] Error receiving netlink message: Cannot allocate memory [12]

When I run katran_grpc_server with the following commands:
./katran_server_grpc -hc_forwarding=false -default_mac xx:xx:xx:xx:xx:xx -intf ens1f1

I get the following error:

I0413 19:11:23.831044     1 KatranGrpcService.cpp:68] Starting Katran
libbpf: Error loading BTF: Invalid argument(22)
libbpf: Error loading .BTF into kernel: -22.
E0413 19:11:23.941128     1 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E0413 19:11:23.941141     1 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E0413 19:11:23.941144     1 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E0413 19:11:23.941172     1 BpfAdapter.cpp:503] Error receiving netlink message: Cannot allocate memory [12]
terminate called after throwing an instance of 'std::invalid_argument'
  what():  can't attach main bpf prog to main inteface
*** Aborted at 1586805083 (Unix time, try 'date -d @1586805083') ***
*** Signal 6 (SIGABRT) (0x1) received by PID 1 (pthread TID 0x7f95729118c0) (linux TID 1) (maybe from PID 1, UID 0) (code: -6), stack trace: ***
    @ 000055cdf085dae2 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
                       /katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
    @ 000055cdf085dbad folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
    @ 00007f95724e688f (unknown)
    @ 00007f95717f6e97 gsignal
    @ 00007f95717f8800 abort
    @ 00007f9571fd3956 (unknown)
    @ 00007f9571fd9ab5 (unknown)
    @ 00007f9571fd9af0 std::terminate()
    @ 00007f9571fd9d23 __cxa_throw
    @ 000055cdf0c1734f katran::KatranLb::attachBpfProgs()
                       /katran/katran/lib/KatranLb.cpp:411
    @ 000055cdf0a1187c lb::katran::KatranGrpcService::KatranGrpcService(katran::KatranConfig const&)
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 000055cdf06ddf1f RunServer(katran::KatranConfig&, int, std::shared_ptr<folly::EventBase>)
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 000055cdf06de77b main
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 00007f95717d9b96 __libc_start_main
    @ 000055cdf06ddb39 _start
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
*** Aborted at 1586805084 (Unix time, try 'date -d @1586805084') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 1 (pthread TID 0x7f95729118c0) (linux TID 1) (code: 128), stack trace: ***
    @ 000055cdf085dae2 folly::symbolizer::(anonymous namespace)::innerSignalHandler(int, siginfo_t*, void*)
                       /katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
    @ 000055cdf085dbad folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /katran/deps/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
    @ 00007f95724e688f (unknown)
    @ 00007f95717f88f0 abort
    @ 00007f9571fd3956 (unknown)
    @ 00007f9571fd9ab5 (unknown)
    @ 00007f9571fd9af0 std::terminate()
    @ 00007f9571fd9d23 __cxa_throw
    @ 000055cdf0c1734f katran::KatranLb::attachBpfProgs()
                       /katran/katran/lib/KatranLb.cpp:411
    @ 000055cdf0a1187c lb::katran::KatranGrpcService::KatranGrpcService(katran::KatranConfig const&)
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 000055cdf06ddf1f RunServer(katran::KatranConfig&, int, std::shared_ptr<folly::EventBase>)
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 000055cdf06de77b main
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp
    @ 00007f95717d9b96 __libc_start_main
    @ 000055cdf06ddb39 _start
                       -> /katran/deps/folly/folly/io/async/EventBaseManager.cpp

and dmesg -T shows the followings output:

[Mon Apr 13 23:46:18 2020] traps: katran_server_g[35859] general protection ip:7f2a18a698f0 sp:7ffe98b69ea0 error:0 in libc-2.27.so[7f2a18a29000+1e7000]
[Mon Apr 13 23:46:18 2020] traps: katran_server_g[35857] general protection ip:7f7541b758f0 sp:7ffe16337750 error:0 in libc-2.27.so[7f7541b35000+1e7000]
[Mon Apr 13 23:47:19 2020] traps: katran_server_g[36045] general protection ip:7f1f7048d8f0 sp:7fff148ce510 error:0 in libc-2.27.so[7f1f7044d000+1e7000]
[Mon Apr 13 23:47:19 2020] traps: katran_server_g[36060] general protection ip:7f0d22cae8f0 sp:7fff9120b0e0 error:0 in libc-2.27.so[7f0d22c6e000+1e7000]
[Mon Apr 13 23:48:19 2020] traps: katran_server_g[36262] general protection ip:7fcb70e318f0 sp:7ffd549ea720 error:0 in libc-2.27.so[7fcb70df1000+1e7000]
[Mon Apr 13 23:48:19 2020] traps: katran_server_g[36251] general protection ip:7f880bd458f0 sp:7fff048b66b0 error:0 in libc-2.27.so[7f880bd05000+1e7000]
[Mon Apr 13 23:49:20 2020] traps: katran_server_g[36460] general protection ip:7eff78eca8f0 sp:7ffc6724b8e0 error:0 in libc-2.27.so[7eff78e8a000+1e7000]
[Mon Apr 13 23:49:20 2020] traps: katran_server_g[36474] general protection ip:7f2808bb08f0 sp:7ffc60646cc0 error:0 in libc-2.27.so[7f2808b70000+1e7000]

my kernel-version is: 4.15.0-96-generic
I also tried on 5.6.3-050603-lowlatency

CPU Model:
Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

NIC Model:
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection

ixgbe version: 5.6.5

any ideas?

katran logo

Needs a logo (I'm building a slide of BPF technologies).

can't load main bpf program run os_run_tester.sh

I get this error when I run os_run_tester.sh

+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/bule/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/bule/katran/_build/deps
+ sudo sh -c '/home/bule/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
[sudo] password for bule:
libbpf: elf: skipping unrecognized data section(6) .maps.ctl_array
libbpf: elf: skipping unrecognized data section(7) .maps.vip_map
libbpf: elf: skipping unrecognized data section(8) .maps.fallback_cache
libbpf: elf: skipping unrecognized data section(9) .maps.ch_rings
libbpf: elf: skipping unrecognized data section(10) .maps.reals
libbpf: elf: skipping unrecognized data section(11) .maps.reals_stats
libbpf: elf: skipping unrecognized data section(12) .maps.stats
libbpf: elf: skipping unrecognized data section(13) .maps.quic_mapping
libbpf: elf: skipping unrecognized data section(25) .eh_frame
libbpf: elf: skipping relo section(26) .rel.eh_frame for section(25) .eh_frame
libbpf: load bpf program failed: Invalid argument
libbpf: failed to load program 'balancer_ingress'
libbpf: failed to load object '/home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o'
E0922 09:58:04.089079  1681 BpfLoader.cpp:307] error while trying to load bpf object: /home/bule/katran/_build/deps/bpfprog/bpf/balancer_kern.o
terminate called after throwing an instance of 'std::invalid_argument'
  what():  can't load main bpf program
Aborted (core dumped)

ubuntu 18.04 and kernel 4.15

add .clang-format into repository?

how hard is it going to be to add clang-format style file into katran's repo? (clang-format -dump-config to generate). right now if i'm running clang-format -i on chaged files i can see lots of unrelated changes. could be because of some custom options of clang-format which is using internally @ fb. would be nice for everyone (internal + external ) to be in sync on how clang-format is configured

SIGSEGV running os_run_tester.sh

Hi folks.

I spun up a fresh VM with Ubuntu 18.04 to test this, and I'm getting a SIGSEGV running the os_run_tester.sh script.

After changing build/CMakeCache.txt: like this:

...
CMAKE_BUILD_TYPE:STRING=Debug
...
CMAKE_CXX_FLAGS_DEBUG:STRING=-g3 -fno-stack-protector
...

I then used gdb to run the tester like so:

(gdb) file /usr/local/src/katran/build/katran/lib/testing/katran_tester
Reading symbols from /usr/local/src/katran/build/katran/lib/testing/katran_tester...done.
(gdb) run -balancer_prog /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true
Starting program: /usr/local/src/katran/build/katran/lib/testing/katran_tester -balancer_prog /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
E0617 19:44:14.898008 24596 BpfLoader.cpp:160] Can't read section size for index: 2
I0617 19:44:14.898075 24596 BpfLoader.cpp:434] Skipping section: 2 of file: /usr/local/src/katran/deps/linux/bpfprog/bpf/balancer_kern.o
I0617 19:44:14.931807 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0617 19:44:14.931880 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0617 19:44:14.931902 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0617 19:44:14.931906 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0617 19:44:14.931908 24596 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
I0617 19:44:14.939867 24596 KatranLb.cpp:394] adding new vip: 10.200.1.1:80:17
I0617 19:44:15.020308 24596 KatranLb.cpp:394] adding new vip: 10.200.1.1:80:6
I0617 19:44:15.091914 24596 KatranLb.cpp:394] adding new vip: 10.200.1.2:0:6
I0617 19:44:15.162443 24596 KatranLb.cpp:394] adding new vip: 10.200.1.4:0:6
I0617 19:44:15.231927 24596 KatranLb.cpp:459] modyfing vip: 10.200.1.4:0:6
I0617 19:44:15.231992 24596 KatranLb.cpp:394] adding new vip: 10.200.1.3:80:6
I0617 19:44:15.301601 24596 KatranLb.cpp:394] adding new vip: fc00:1::1:80:6
I0617 19:44:15.370398 24596 KatranLb.cpp:394] adding new vip: 10.200.1.5:443:17
I0617 19:44:15.370491 24596 KatranLb.cpp:459] modyfing vip: 10.200.1.5:443:17
I0617 19:44:15.440598 24596 KatranLb.cpp:394] adding new vip: fc00:1::2:443:17
I0617 19:44:15.440691 24596 KatranLb.cpp:459] modyfing vip: fc00:1::2:443:17
I0617 19:44:15.512671 24596 XdpTester.cpp:166] Test: packet to UDP based v4 VIP (and v4 real)                     result: Passed
I0617 19:44:15.512732 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v4 real)                     result: Passed
I0617 19:44:15.512744 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v4 real; any dst ports).     result: Passed
I0617 19:44:15.512753 24596 XdpTester.cpp:166] Test: packet to TCP based v4 VIP (and v6 real)                     result: Passed
I0617 19:44:15.512763 24596 XdpTester.cpp:166] Test: packet to TCP based v6 VIP (and v6 real)                     result: Passed
I0617 19:44:15.512769 24596 XdpTester.cpp:166] Test: v4 ICMP echo-request                                         result: Passed
I0617 19:44:15.512776 24596 XdpTester.cpp:166] Test: v6 ICMP echo-request                                         result: Passed
I0617 19:44:15.512784 24596 XdpTester.cpp:166] Test: v4 ICMP dest-unreachabe fragmentation-needed                 result: Passed
I0617 19:44:15.512814 24596 XdpTester.cpp:166] Test: v6 ICMP packet-too-big                                       result: Passed
I0617 19:44:15.512897 24596 XdpTester.cpp:166] Test: drop of IPv4 packet w/ options                               result: Passed
I0617 19:44:15.512924 24596 XdpTester.cpp:166] Test: drop of IPv4 fragmented packet                               result: Passed
I0617 19:44:15.512948 24596 XdpTester.cpp:166] Test: drop of IPv6 fragmented packet                               result: Passed
I0617 19:44:15.512956 24596 XdpTester.cpp:166] Test: pass of v4 packet with dst not equal to any configured VIP   result: Passed
I0617 19:44:15.512964 24596 XdpTester.cpp:166] Test: pass of v6 packet with dst not equal to any configured VIP   result: Passed
I0617 19:44:15.512971 24596 XdpTester.cpp:166] Test: pass of arp packet                                           result: Passed
I0617 19:44:15.512979 24596 XdpTester.cpp:166] Test: LRU hit                                                      result: Passed
I0617 19:44:15.512990 24596 XdpTester.cpp:166] Test: packet #1 dst port hashing only                              result: Passed
I0617 19:44:15.512998 24596 XdpTester.cpp:166] Test: packet #2 dst port hashing only                              result: Passed
I0617 19:44:15.513005 24596 XdpTester.cpp:166] Test: ipinip packet                                                result: Passed
I0617 19:44:15.513031 24596 XdpTester.cpp:166] Test: ipv6inipv6 packet                                            result: Passed
I0617 19:44:15.513054 24596 XdpTester.cpp:166] Test: ipv4inipv6 packet                                            result: Passed
I0617 19:44:15.513063 24596 XdpTester.cpp:166] Test: QUIC: long header. Client Initial type                       result: Passed
I0617 19:44:15.513092 24596 XdpTester.cpp:166] Test: QUIC: long header. 0-RTT Protected                           result: Passed
I0617 19:44:15.513114 24596 XdpTester.cpp:166] Test: QUIC: long header. v4 vip v6 real                            result: Passed
I0617 19:44:15.513147 24596 XdpTester.cpp:166] Test: QUIC: long header. v6 vip v6 real                            result: Passed
I0617 19:44:15.513252 24596 XdpTester.cpp:166] Test: QUIC: short header. no connection id                         result: Passed
I0617 19:44:15.513362 24596 XdpTester.cpp:166] Test: QUIC: short header w/ connection id                          result: Passed
I0617 19:44:15.513447 24596 XdpTester.cpp:166] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I0617 19:44:15.513480 24596 katran_tester.cpp:166] Testing counter's sanity. Printing on errors only

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

Here is the BT:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000000000000000 in ?? ()

And a list:

(gdb) list
177	  stats = lb.getLruMissStats();
178	  if ((stats.v1 != 2) || (stats.v2 != 6)) {
179	    VLOG(2) << "TCP syns: " << stats.v1 << " TCP non-syns: " << stats.v2;
180	    LOG(INFO) << "per pckt type LRU miss counter is incorrect";
181	  }
182	  stats = lb.getLruFallbackStats();
183	  if (stats.v1 != 13) {
184	    VLOG(2) << "FallbackLRU hits: " << stats.v1;
185	    LOG(INFO) << "LRU fallback counter is incorrect";
186	  }

Here are the particulars of the test box:

# uname -a
Linux katran 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.3.0-16ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --with-as=/usr/bin/x86_64-linux-gnu-as --with-ld=/usr/bin/x86_64-linux-gnu-ld --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)

I'm out of my depth here, and was hoping you all might look at it. Let me know what other info you need or if there is any other way I can help. Reproducing should be as easy as spinning up a stock Ubuntu 18.04 VM and running build_katran.sh.

Thanks!

healthcheck problem

Hi， my config is similar to this one about healthcheck while for some reason in our cvm environment, we modified balance_kern.c, DSR is replaced by TUNNEL and return packet will go through LB.

now packet forwarding is well but health check is not. Actually i can capture hc return packet with tcpdump, but the hc process do not get the response.

configure in my environment

LB（katran）:

• eth0 172.33.4.100 --intf
• eth0:1 172.33.4.75 for tunnel back
• eth0:2 172.33.4.78 vip
• lo --hc_intf
• ipip0 --ipip_intf

RS:
configured with tunnel back
-eth0 172.33.35.7 mark 10000

health check program:
bind socket with 172.33.4.100 and send SYNC to 172.33.4.78(vip) with mark 10000.

for test, i modified balancer_kern, when recving IPIP pacekt, directly return XDP_PASS to kernel.

but my hc program do not recv ACK while i can capture ACK packet in LB with tcpdump.

strangely when i ifconfig down the vip in LB, hc program can recv ACK.

Reals at bridged interface

Hey, I try to run Katran in a setup where more reals sit at one physical server node together with Katran.
So there are two nodes master (10.80.96.1) and test (10.80.96.3). master runs two reals(10.80.96.70, 10.80.96.68) that are part of one VIP (10.80.96.13) and test is used to create requests to the VIP.

root@master# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.80.96.13/32 brd 10.80.96.13 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s20f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0c:c4:7a:e5:46:d6 brd ff:ff:ff:ff:ff:ff
3: enp0s20f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 0c:c4:7a:e5:46:d7 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 xdpgeneric/id:142 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:e5:46:d6 brd ff:ff:ff:ff:ff:ff
    inet 147.x.x.x/31 brd 255.255.255.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 10.80.96.1/31 brd 255.255.255.255 scope global bond0:0
       valid_lft forever preferred_lft forever
    inet6 2604:1380:2000:8f00::1/127 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fee5:46d6/64 scope link 
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:30:1c:4b:ae brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:30ff:fe1c:4bae/64 scope link 
       valid_lft forever preferred_lft forever
8: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:58:0a:50:60:41 brd ff:ff:ff:ff:ff:ff
    inet 10.80.96.65/29 scope global kube-bridge
       valid_lft forever preferred_lft forever
    inet6 fe80::9487:55ff:fe03:69a1/64 scope link 
       valid_lft forever preferred_lft forever
11: kube-dummy-if: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether f2:7d:88:7a:96:d3 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.1/32 brd 10.96.0.1 scope link kube-dummy-if
       valid_lft forever preferred_lft forever
    inet 10.96.0.10/32 brd 10.96.0.10 scope link kube-dummy-if
       valid_lft forever preferred_lft forever
    inet 10.105.217.13/32 brd 10.105.217.13 scope link kube-dummy-if
       valid_lft forever preferred_lft forever
    inet6 fe80::f07d:88ff:fe7a:96d3/64 scope link 
       valid_lft forever preferred_lft forever
12: vethde26d90b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether ba:f2:1e:a0:15:2f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::b8f2:1eff:fea0:152f/64 scope link 
       valid_lft forever preferred_lft forever
13: veth7b135f01@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether aa:20:29:d3:29:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::a820:29ff:fed3:29e2/64 scope link 
       valid_lft forever preferred_lft forever
14: veth1956ecdc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether 2a:d7:28:63:92:3e brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::28d7:28ff:fe63:923e/64 scope link 
       valid_lft forever preferred_lft forever
15: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
16: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 127.0.0.42/32 scope host ipip0
       valid_lft forever preferred_lft forever
18: veth9c036093@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP group default 
    link/ether fa:0a:06:f9:28:5e brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::f80a:6ff:fef9:285e/64 scope link 
       valid_lft forever preferred_lft forever
19: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
20: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN group default qlen 1000
    link/tunnel6 :: brd ::
    inet6 fe80::ac1a:bbff:fea6:2184/64 scope link 
       valid_lft forever preferred_lft forever

Every node has assigned subnet of ips that can sit there. In this case 10.80.96.64/29 through kube-bridge:

root@master:~# ip route
default via 147.x.x.x dev bond0 onlink 
10.0.0.0/8 via 10.80.96.0 dev bond0 
10.80.96.0/31 dev bond0 proto kernel scope link src 10.80.96.1 
10.80.96.64/29 dev kube-bridge proto kernel scope link src 10.80.96.65 
147.x.x.x/31 dev bond0 proto kernel scope link src 147.x.x.x
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

root@master:~# ./katran/katranc -l
2019/02/06 12:13:31 vips len 1
VIP:          10.80.96.13 Port:     80 Protocol: tcp
Vip's flags: 
 ->10.80.96.70       weight: 1
 ->10.80.96.68       weight: 1
exiting

tcpdump at master:

12:24:05.312977 IP 10.80.96.3.60764 > 10.80.96.13.http: Flags [S], seq 949265021, win 29200, options [mss 1460,sackOK,TS val 2208753565 ecr 0,nop,wscale 8], length 0
12:24:05.313047 IP 10.80.96.13.http > 10.80.96.3.60764: Flags [R.], seq 0, ack 949265022, win 0, length 0

root@test:~# curl 10.80.96.13:80
curl: (7) Failed to connect to 10.80.96.13 port 80: Connection refused

Reals 10.80.96.70:80, 10.80.96.68:80 could be accessed, it's a simple nginx.

root@test:~# curl 10.80.96.70
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

VIP 10.80.96.13/32 gets announced using BGP and is set at the interface lo. ipip tunnels are present and up.

Request on 10.80.96.13:80 returns connection refused, but tcpdump shows traffic got to the node.
I think it is caused by the fact I use bridge. Is it the case or do you see any other cause? Is there any way to mitigate this issue? @tehnerd

root@master:~# ps faux | grep katran
root     23059  0.0  0.0  13108  1064 pts/5    S+   12:56   0:00          \_ grep --color=auto katran
root      3682  0.0  0.0  18376  2964 ?        Ss   12:13   0:00      |   \_ /bin/bash /katran/start_katran_grpc_server_smart.sh
root      3731  0.0  0.1 479676 11512 ?        SLl  12:13   0:00      |       \_ /katran/katran_server_grpc -balancer_prog /katran/balancer_kern.o -default_mac 54:1e:56:64:67:00 -forwarding_cores=0,1,2,3 -hc_forwarding=false -intf=bond0 -lru_size=10000

root@master:~# ip route | grep default
default via 147.x.x.x dev bond0 onlink 
root@master:~# ip n show | grep 147.x.x.x
147.x.x.x dev bond0 lladdr 54:1e:56:64:67:00 REACHABLE

root@master:~# ethtool -l enp0s20f0
Channel parameters for enp0s20f0:
Pre-set maximums:
RX:		0
TX:		0
Other:		1
Combined:	8
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	4

root@master:~# ethtool -l enp0s20f1
Channel parameters for enp0s20f1:
Pre-set maximums:
RX:		0
TX:		0
Other:		1
Combined:	8
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	4

Compilation error from build_katran.sh

I ran ./build_katran.sh and got this error:

/home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc: In member function ‘virtual void X509Test_ZeroLengthsWithX509PARAM_Test::TestBody()’:
/home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc:712:10: error: declaration of ‘struct X509Test_ZeroLengthsWithX509PARAM_Test::TestBody()::Test’ shadows a previous local [-Werror=shadow]
   struct Test {
          ^~~~
In file included from /home/kahou/katran/deps/grpc/third_party/boringssl/crypto/x509/x509_test.cc:19:
/home/kahou/katran/deps/grpc/third_party/boringssl/third_party/googletest/include/gtest/gtest.h:375:23: note: shadowed declaration is here
 class GTEST_API_ Test {
                       ^
cc1plus: all warnings being treated as errors
make[2]: *** [third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/build.make:635: third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/x509/x509_test.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2348: third_party/boringssl/crypto/CMakeFiles/crypto_test.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

How about the performance?

What's the throughput(pps)?Could you show me the data chart?

marianne

Decapsulation of IPIP packets

Hello! I get the way non-IPIP packets are processed on L4 nodes. But what does happen to them once they get to L7 node? Is there also Katran that does the decapsulation? I've been reading the BPF code and there are some signs Katran does this when INLINE_DECAP is set. And then there is a function process_encaped_pckt that actually does the decapsulation I think.

I assume that Kartan is also supposed to run on L7 nodes and do the decapsulation of IPIP packets. Could you please tell whether my assumption is correct or not?

Failed to run ./build_katran.sh

Hi,

I am trying to build katran on my ubuntu 19.10

user:~/katran$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan

and using katran release

$ git log
commit 3d50460170b2e7d5d05d85e011f5fe5ef196cddf (HEAD -> master, origin/master, origin/HEAD)
Author: svcscm <[email protected]>
Date:   Wed Jan 29 10:45:21 2020 -0800

    Updating submodules
    
    Summary:
    GitHub commits:
    
    https://github.com/facebook/litho/commit/fada02ad90dcf14e29fe22acbb3485595d6b93de
    https://github.com/facebook/rocksdb/commit/a07a9dc904b692db67052e58ccd0b5d2e96aff92
    https://github.com/facebookincubator/fizz/commit/fbb77357c42ece2f64c1aa3fc6a4e3ec17708b44
    https://github.com/facebookincubator/profilo/commit/3b3e46f2b308aec845c7b3718ab4ef36d0019707
    
    Reviewed By: 2d2d2d2d2d
    
    fbshipit-source-id: 0cc55e3757528f2f9db4c32dbaef32174c808706

Here is the issue.

-- Found Protobuf: /home/bfn/katran/_build/deps/lib/libprotobuf.a;-lpthread (found version "3.11.2")
cmd is /home/bfn/katran/_build/deps/bin/protoc
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ABSL_BAD_OPTIONAL
    linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc
ABSL_BASE_THROW
    linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc
ABSL_STRINGS
    linked by target "katran_server_grpc" in directory /home/bfn/katran/example_grpc

-- Configuring incomplete, errors occurred!
See also "/home/bfn/katran/_build/build/CMakeFiles/CMakeOutput.log".
See also "/home/bfn/katran/_build/build/CMakeFiles/CMakeError.log".

Please take a look. Thanks.

TOS for ipv4 copy seems not working

w/ latest upstream diff:

commit 4f8508ca1cb363fce2b16a6ed3200d511f8f59ac (refs/remotes/upstream/master)
Author: Udip Pant <[email protected]>
Date:   Wed Oct 9 09:21:16 2019 -0700

    add unit tests for feature to copy ToS value from inner packet
    
    Reviewed By: mjoras
    
    Differential Revision: D17826785
    
    fbshipit-source-id: 3eacfa8947a42f1aea9b0d1345d6b6d40ddcacec

running katran_tester:

./os_run_tester.sh 
+ '[' -z '' ']'
++ pwd
+ KATRAN_BUILD_DIR=/home/tehnerd/katran/_build/build
+ '[' -z '' ']'
++ pwd
+ DEPS_DIR=/home/tehnerd/katran/_build/deps
+ sudo sh -c '/home/tehnerd/katran/_build/build/katran/lib/testing/katran_tester -balancer_prog /home/tehnerd/katran/_build/deps/bpfprog/bpf/balancer_kern.o -test_from_fixtures=true '
libbpf: Error in bpf_create_map_xattr(lru_maps_mapping):ERROR: strerror_r(-524)=22(-524). Retrying without BTF.
E1009 18:34:46.199492 21152 BpfLoader.cpp:87] Can't find map w/ name: lpm_src_v4
E1009 18:34:46.199611 21152 BpfLoader.cpp:87] Can't find map w/ name: decap_dst
E1009 18:34:46.199625 21152 BpfLoader.cpp:87] Can't find map w/ name: event_pipe
E1009 18:34:46.199632 21152 BpfLoader.cpp:87] Can't find map w/ name: pckt_srcs
I1009 18:34:46.199656 21152 KatranLb.cpp:481] adding new vip: 10.200.1.1:80:17
I1009 18:34:46.274731 21152 KatranLb.cpp:481] adding new vip: 10.200.1.1:80:6
I1009 18:34:46.350360 21152 KatranLb.cpp:481] adding new vip: 10.200.1.2:0:6
I1009 18:34:46.418886 21152 KatranLb.cpp:481] adding new vip: 10.200.1.4:0:6
I1009 18:34:46.486299 21152 KatranLb.cpp:546] modyfing vip: 10.200.1.4:0:6
I1009 18:34:46.486353 21152 KatranLb.cpp:481] adding new vip: 10.200.1.3:80:6
I1009 18:34:46.553889 21152 KatranLb.cpp:481] adding new vip: fc00:1::1:80:6
I1009 18:34:46.620697 21152 KatranLb.cpp:481] adding new vip: 10.200.1.5:443:17
I1009 18:34:46.620771 21152 KatranLb.cpp:546] modyfing vip: 10.200.1.5:443:17
I1009 18:34:46.687752 21152 KatranLb.cpp:481] adding new vip: fc00:1::2:443:17
I1009 18:34:46.687876 21152 KatranLb.cpp:546] modyfing vip: fc00:1::2:443:17
I1009 18:34:46.755780 21152 XdpTester.cpp:168] Test: packet to UDP based v4 VIP (and v4 real)                     result: Passed
I1009 18:34:46.755833 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real)                     result: Passed
I1009 18:34:46.755846 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real) + ToS in IPV4       result: Passed
I1009 18:34:46.755856 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v4 real; any dst ports).     result: Passed
I1009 18:34:46.755867 21152 XdpTester.cpp:168] Test: packet to TCP based v4 VIP (and v6 real)                     result: Passed
I1009 18:34:46.755877 21152 XdpTester.cpp:168] Test: packet to TCP based v6 VIP (and v6 real)                     result: Passed
I1009 18:34:46.755887 21152 XdpTester.cpp:168] Test: packet to TCP based v6 VIP (and v6 real) with ToS / tc set   result: Passed
I1009 18:34:46.755895 21152 XdpTester.cpp:168] Test: v4 ICMP echo-request                                         result: Passed
I1009 18:34:46.755903 21152 XdpTester.cpp:168] Test: v6 ICMP echo-request                                         result: Passed
I1009 18:34:46.755913 21152 XdpTester.cpp:168] Test: v4 ICMP dest-unreachabe fragmentation-needed                 result: Passed
I1009 18:34:46.755923 21152 XdpTester.cpp:168] Test: v6 ICMP packet-too-big                                       result: Passed
I1009 18:34:46.755933 21152 XdpTester.cpp:168] Test: drop of IPv4 packet w/ options                               result: Passed
I1009 18:34:46.755940 21152 XdpTester.cpp:168] Test: drop of IPv4 fragmented packet                               result: Passed
I1009 18:34:46.755949 21152 XdpTester.cpp:168] Test: drop of IPv6 fragmented packet                               result: Passed
I1009 18:34:46.755957 21152 XdpTester.cpp:168] Test: pass of v4 packet with dst not equal to any configured VIP   result: Passed
I1009 18:34:46.755966 21152 XdpTester.cpp:168] Test: pass of v6 packet with dst not equal to any configured VIP   result: Passed
I1009 18:34:46.755975 21152 XdpTester.cpp:168] Test: pass of arp packet                                           result: Passed
I1009 18:34:46.755982 21152 XdpTester.cpp:168] Test: LRU hit                                                      result: Passed
I1009 18:34:46.755992 21152 XdpTester.cpp:168] Test: packet #1 dst port hashing only                              result: Passed
I1009 18:34:46.756001 21152 XdpTester.cpp:168] Test: packet #2 dst port hashing only                              result: Passed
I1009 18:34:46.756009 21152 XdpTester.cpp:168] Test: ipinip packet                                                result: Passed
I1009 18:34:46.756018 21152 XdpTester.cpp:168] Test: ipv6inipv6 packet                                            result: Passed
I1009 18:34:46.756027 21152 XdpTester.cpp:168] Test: ipv4inipv6 packet                                            result: Passed
I1009 18:34:46.756036 21152 XdpTester.cpp:168] Test: QUIC: long header. Client Initial type. LRU miss             result: Passed
I1009 18:34:46.756045 21152 XdpTester.cpp:168] Test: QUIC: long header. 0-RTT Protected. CH. LRU hit.             result: Passed
I1009 18:34:46.756054 21152 XdpTester.cpp:168] Test: QUIC: long header. Handshake. v4 vip v6 real. Conn Id based. result: Passed
I1009 18:34:46.756062 21152 XdpTester.cpp:168] Test: QUIC: long header. Retry. v4 vip v6 real. Conn Id based.     result: Passed
I1009 18:34:46.756072 21152 XdpTester.cpp:168] Test: QUIC: long header. client initial. v6 vip v6 real. LRU miss  result: Passed
I1009 18:34:46.756081 21152 XdpTester.cpp:168] Test: QUIC: short header. No connection id. CH. LRU hit            result: Passed
I1009 18:34:46.756089 21152 XdpTester.cpp:168] Test: QUIC: short header w/ connection id                          result: Passed
I1009 18:34:46.756098 21152 XdpTester.cpp:168] Test: QUIC: short header w/ connection id but non-existing mapping result: Passed
I1009 18:34:46.756106 21152 XdpTester.cpp:168] Test: QUIC: short header w/ conn id. host id = 0. CH. LRU hit      result: Passed
I1009 18:34:46.756217 21152 katran_tester.cpp:327] Testing counter's sanity. Printing on errors only
I1009 18:34:46.756355 21152 katran_tester.cpp:377] Testing of counters is complete
E1009 18:34:46.756445 21152 KatranSimulator.cpp:169] src and dst must have same address family
E1009 18:34:46.756454 21152 KatranSimulator.cpp:162] malformed src or dst ip address. src: aaaa dst: bbbb

aka everything is green/passed. however if i check what actual packets were generated (it is input/output format):

 ./os_run_tester.sh  -pcap_output=/tmp/test.pcap

18:35:45.391440 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto UDP (17), length 43)
    192.168.1.1.31337 > 10.200.1.1.80: [udp sum ok] UDP, length 15
18:35:45.391454 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 63)
    172.16.104.123 > 10.0.0.3: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto UDP (17), length 43)
    192.168.1.1.31337 > 10.200.1.1.80: [udp sum ok] UDP, length 15
18:35:45.391502 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391510 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
    172.16.104.123 > 10.0.0.3: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP

bellow v4 input has tos output does not 
18:35:45.391521 IP (tos 0x8c, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP

no tos in output packet:
18:35:45.391525 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
    172.16.104.123 > 10.0.0.3: IP (tos 0x8c, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.1.80: Flags [.], cksum 0x27e4 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP


18:35:45.391533 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.2.42: Flags [.], cksum 0x2809 (correct), seq 0:15, ack 0, win 8192, length 15
18:35:45.391537 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto IPIP (4), length 75)
    172.16.104.123 > 10.0.0.2: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.2.42: Flags [.], cksum 0x2809 (correct), seq 0:15, ack 1, win 8192, length 15
18:35:45.391544 IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.3.80: Flags [.], cksum 0x27e2 (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391549 IP6 (hlim 64, next-header IPIP (4) payload length: 55) 100::bac1:101 > fc00::1: IP (tos 0x0, ttl 64, id 1, offset 0, flags [none], proto TCP (6), length 55)
    192.168.1.1.31337 > 10.200.1.3.80: Flags [.], cksum 0x27e2 (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
18:35:45.391556 IP6 (hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 0, win 8192, length 15: HTTP
18:35:45.391561 IP6 (hlim 64, next-header IPv6 (41) payload length: 75) 100::7a69:1 > fc00::3: IP6 (hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP
18:35:45.391570 IP6 (class 0x8c, hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP

v6 seems working as expected:
18:35:45.391574 IP6 (class 0x8c, hlim 64, next-header IPv6 (41) payload length: 75) 100::7a69:1 > fc00::3: IP6 (class 0x8c, hlim 64, next-header TCP (6) payload length: 35) fc00:2::1.31337 > fc00:1::1.80: Flags [.], cksum 0xfd4f (correct), seq 0:15, ack 1, win 8192, length 15: HTTP

the issue is that tos population is inside "decrement_ttl" stanza:
https://github.com/facebookincubator/katran/blob/master/katran/lib/bpf/balancer_kern.c#L230
which is set for true only for IPIP decapsulated packets when INLINE_DECAP is used (e.g. for stable anycast usecase)

R? min value is negative when calling ipv4_l4_csum

Hi,

I played with bpf/xdp to prototype an UDP loadbalancer.
For UDP checksum, I reused your ipv4_l4_csum function, but I fall in a verifier issue :

R? min value is negative, either use unsigned or 'var &= const' 

We currently discussing about this tough issue on bcc mailing list.
I would like to know if you already faced this kind of issue and you have some input about that ?

Thx

lru miss 100%

[katran machine(Ubuntu 18.04 kvm guest) info]

root@jockey-test:/home/ubuntu/katran/example_grpc# ethtool  -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       8

root@jockey-test:/home/ubuntu/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o  -default_mac fa:16:3e:6d:25:0b -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o  -intf=eth0 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000
I0821 11:33:39.125748  8470 KatranGrpcService.cpp:67] Starting Katran
E0821 11:33:39.125957  8470 BpfLoader.cpp:160] Can't read section size for index: 2
I0821 11:33:39.125982  8470 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/balancer_kern.o
I0821 11:33:39.158964  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0821 11:33:39.158993  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0821 11:33:39.159008  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0821 11:33:39.159010  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0821 11:33:39.159013  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0821 11:33:39.166762  8470 BpfLoader.cpp:160] Can't read section size for index: 2
I0821 11:33:39.166790  8470 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
E0821 11:33:39.166806  8470 BpfLoader.cpp:160] Can't read section size for index: 12
I0821 11:33:39.166810  8470 BpfLoader.cpp:434] Skipping section: 12 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
I0821 11:33:39.167026  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0821 11:33:39.167047  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0821 11:33:39.167052  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0821 11:33:39.167070  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0821 11:33:39.167073  8470 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0821 11:33:39.170778  8470 BpfLoader.cpp:97] Can't find name w/ name: lpm_src_v4
E0821 11:33:39.170791  8470 BpfLoader.cpp:97] Can't find name w/ name: decap_dst
Server listening on 0.0.0.0:50051
I0821 11:34:03.237671  8479 KatranLb.cpp:429] adding new vip: 10.200.200.1:22:6
I0821 11:36:18.393283  8487 KatranLb.cpp:429] adding new vip: 10.200.200.1:23:6

root@jockey-test:/home/ubuntu/katran/example_grpc# ./katran_goclient -l
2018/08/21 11:48:52 vips len 2
VIP:         10.200.200.1 Port:     23 Protocol: tcp
Vip's flags: 
 ->10.0.100.218      weight: 1
VIP:         10.200.200.1 Port:     22 Protocol: tcp
Vip's flags: 
 ->10.0.100.218      weight: 1

root@jockey-test:/home/ubuntu/katran/example_grpc# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4076 xdp qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:0c:0f:46 brd ff:ff:ff:ff:ff:ff
    prog/xdp id 206548 tag 9ac36823edf016c9 jited 
3: br-87c747560664: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:05:b4:52:71 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:8b:17:c6:a4 brd ff:ff:ff:ff:ff:ff
17: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
18: ipip0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
19: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd ::
20: ipip60@NONE: <NOARP,UP,LOWER_UP> mtu 1452 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/tunnel6 :: brd ::

When client telnet to VIP 10.200.200.1 port 22, katran_goclient prints "lru miss 100.00%" line.

root@jockey-test:/home/ubuntu/katran/example_grpc# ./katran_goclient -s -lru
summary: 21 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 21 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec
summary: 1 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 1 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec

kernel support policy.

What is the policy for open sourced version of katran regarding kernel's support. do you guys pro actively testing things on 4.15 (ubuntu 18.04. which seems officially supported by katran. at least all examples are using it) or not? do you consider it as regression if newer version of code does not work on it (4.15) and will try to fix it? or ? (i'm looking on new patches for quic conn id 24 bit and i have a feeling that verifier wont be happy w/ this patches on 4.15. were they tested on unbuntu 18.04?)

katran is not work fine

hi,
as example:
myclient(10.43.1.208)---->(220.248.68.38 interface enp4s0f1)(vip: 220.248.68.37)katran_server(lan_10.24.247.28)---real server(10.24.247.31)

on myclient:
telnet 220.248.68.37 22
no respone and i dont know whats wrong with me?
#start command:

/root/katran/build/katran/lib/xdproot -bpfprog /root/katran/deps/linux/bpfprog/bpf/xdp_root.o -bpfpath=/sys/fs/bpf/jmp_enp4s0f1 -intf=enp4s0f1
E0814 09:52:57.754595 14135 BpfLoader.cpp:160] Can't read section size for index: 2
E0814 09:52:57.755061 14135 BpfLoader.cpp:160] Can't read section size for index: 12
can't pin root array

root@ubnutu:~/katran# ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/linux/bpfprog/bpf/balancer_kern.o -default_mac 10:f3:11:7a:b1:96 -forwarding_cores=0 -healthchecker_prog ./deps/linux/bpfprog/bpf/healthchecking_ipip.o -intf=enp4s0f1 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000 -map_path /sys/fs/bpf/jmp_enp4s0f1 -prog_pos=2
I0814 10:00:01.356066 14182 KatranGrpcService.cpp:67] Starting Katran
E0814 10:00:01.357009 14182 BpfLoader.cpp:160] Can't read section size for index: 2
I0814 10:00:01.357031 14182 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/balancer_kern.o
I0814 10:00:01.417520 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0814 10:00:01.417557 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0814 10:00:01.417563 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0814 10:00:01.417569 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0814 10:00:01.417574 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0814 10:00:01.435165 14182 BpfLoader.cpp:160] Can't read section size for index: 2
I0814 10:00:01.435199 14182 BpfLoader.cpp:434] Skipping section: 2 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
E0814 10:00:01.435222 14182 BpfLoader.cpp:160] Can't read section size for index: 12
I0814 10:00:01.435230 14182 BpfLoader.cpp:434] Skipping section: 12 of file: ./deps/linux/bpfprog/bpf/healthchecking_ipip.o
I0814 10:00:01.435448 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 10
I0814 10:00:01.435462 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 14
I0814 10:00:01.435469 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 16
I0814 10:00:01.435479 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 18
I0814 10:00:01.435485 14182 BpfLoader.cpp:353] relocation for non existing prog w/ idx 20
E0814 10:00:01.443327 14182 BpfLoader.cpp:97] Can't find name w/ name: lpm_src_v4
E0814 10:00:01.443353 14182 BpfLoader.cpp:97] Can't find name w/ name: decap_dst
Server listening on 0.0.0.0:50051

root@ubnutu:# ./katran_goclient -l
2018/08/14 05:49:25 vips len 1
VIP: 220.248.68.37 Port: 22 Protocol: tcp
Vip's flags:
->10.24.247.31 weight: 1
exiting
root@ubnutu:# ping 10.24.247.31
PING 10.24.247.31 (10.24.247.31) 56(84) bytes of data.
64 bytes from 10.24.247.31: icmp_seq=1 ttl=64 time=0.399 ms

root@ubnutu:~# ./katran_goclient -s -lru
summary: 14 pkts/sec. lru hit: 0.00% lru miss: 100.00% (tcp syn: 1.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 14 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec
summary: 0 pkts/sec. lru hit: 0.00% lru miss: 0.00% (tcp syn: 0.00% tcp non-syn: 0.00% udp: 0.00%) fallback lru hit: 0 pkts/sec

#on real server 10.24.247.31
[root@node3:~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 220.248.68.37/32 scope global lo
valid_lft forever preferred_lft forever

katran with bonding

Hi folks,

I have a bunch of servers that use 802.3ad bonding. Can I just run a separate instance of katran for each member ethernet nic? (dividing up the interrupts and such correctly, of course.) If I understand correctly, Katran is low enough in the stack where it wouldn't know or care about the bonding. Is this correct?

Thanks!

weights for maglev's hash

Hi:
I configure 1 VIP and 2 real servers with weight value 1 and 10.
But the 2 real servers filled in ch_rings bpf map have almost the same number (The ratio is 1:1).
I think this line in the source code is the reason. Anyone can tell me the reason why assigning 1 to the weight of the endpoint ?

hc_it_client.py no found!

HI, the recent study of your project kartan, I installed in the test environment, according to your EXAMPLE.md file, I find the source code did not find hc_it_client.py, you can provide this file. Thank you！

Can Katran handle the packets from client when its size equaled to MTU?

From my test, if the packet from client has no room for adding ipip header, Katran can't handle it. Is it right? I didn't see any codes in xdp-balancer to handle this situation. Can it be fixed?