linhua55/lkl_study

study the LKL(linux kernel library) https://github.com/lkl/linux

lkl_study

study the LKL(linux kernel library) https://github.com/lkl/linux

Only apply to 64Bit Linux with ipv4 network.

Rinetd version(with LKL and raw socket backend)

Compile

1. compile static library liblkl.a

https://github.com/linhua55/linux/tree/rinetd_bpf

refer to https://github.com/lkl/linux

Linux(LKL)'s kernel configuration file is the .config file in this repository, it need to be placed at the root directory of the LKL repository.

2. compile rinetd(with lkl)

https://github.com/linhua55/rinetd

refer to https://github.com/linhua55/rinetd/blob/lkl_raw/make.sh

replace /home/vagrant/lkl/linux/tools/lkl/liblkl.a and /home/vagrant/lkl/linux/tools/lkl/include with your actual LKL path.

Release

rinetd(lkl) with bbr powered congestion control

wget "https://github.com/linhua55/lkl_study/releases/download/v1.2/rinetd_bbr_powered" -O /usr/bin/rinetd

rinetd(lkl) with bbr congestion control

wget "https://github.com/linhua55/lkl_study/releases/download/v1.2/rinetd_bbr" -O /usr/bin/rinetd

rinetd(lkl) with pcc congestion control

wget "https://github.com/linhua55/lkl_study/releases/download/v1.2/rinetd_pcc" -O /usr/bin/rinetd

For usage, refer to:

https://gist.github.com/codexss/1d5a834c479bb1532b9f82b23ee2f3fa

https://github.com/mixool/rinetd

https://www.v2ex.com/t/353778#r_4311799

One-key script

Thanks to @phuslu for his one-key script

Usage：

  curl https://raw.githubusercontent.com/linhua55/lkl_study/master/get-rinetd.sh | bash

      The configuration file generated by one-key script is /etc/rinetd-bbr.conf. By default, it only proxy(speed up) port 443and80, modify the port number as needed.

Determine if function

Use top command， view process rinetd's CPU usage. The faster of network speed, the bigger of CPU usage.

Caution：

1. Dependency: iptables, grep, cut, xargs. Usual linux have these tools，But some linux use firewalld instead of iptables, it need install iptables
2. For KVM VPS, need to change venet0:0 to the name of the network interface which have KVM's public IP, normally it is eth0

Some technical details

https://linhua55.github.io/2017/04/24/LKL(Linux%20Kernel%20Library)/

lkl_study's Issues

关于编译出来的liblkl-hijack.so无法联网

我需要使用liblkl-hijack.so来跑haproxy，因为rinetd无法传递真实IP，源码是您的rinetd_bpf分支，配置文件使用本项目的.config，编译好以后运行报错，无法联网

virtio net fd read: Socket operation on non-socket

i=0
cnt=1
iov[0].iov_len=88
write to fd netdev fails: Socket operation on non-socket

是不是这个配置文件只适用于rinetd版本的lkl，需要修改才能使用？

centos 6 x64 运行问题

centos 6 x64 上无法运行rinetd：安装成功后，在“top”进程列表中找不到rinetd进程。同样方式，在centos 7 x64上正常。有方法解决么？想用6，因为资源占用少

无法自动设置 iptables

尝试了 debian \ ubuntu 两种系统，启动 rinetd 后 iptables -L 仍然是空的。

开启rinetd导致ocserv断流

ocserv是用https://github.com/fanyueciyuan/eazy-for-ss/tree/master/ocservauto （https://www.fanyueciyuan.info/fq/ocserv-debian.html) 上的一键脚本安装的。
开启rinetd_bbr_powered或者rinetd_bbr之后，ocserv握手连接正常（使用TCP，端口999），数据流量断流无法打开网页（使用UDP，端口1999），rinetd设置里添不添加999都一样，使用googledrive/一键脚本上的rinetd_bbr_powered也一样，rinetd_pcc没试。关闭rinetd以后anyconnect连接正常可以正常上网。感觉就是开启rinetd以后UDP流量有问题。
VPS是Debian 8.0的OVZ。

使用Linux内核的BPF进行包的过滤

参考 Chion82/kcptun-raw#15
现在是 在用户态过滤的 #3 (comment)

试验：利用 dummy socket去掉 iptables依赖

去掉iptables依赖，可以在 docker（如樱花）中运行

需要修改LKL， lkl不进行三次握手，本地内核进行三次握手。 但是本地内核不处理数据，由LKL处理数据
出自： https://stackoverflow.com/questions/31762305/prevent-kernel-from-processing-tcp-segments-bound-to-a-raw-socket

TODO：

1. 先用WireShark查看， RST原因，究竟是LKL发的，还是本地内核发的
   https://groups.google.com/forum/#!topic/bbr-dev/Nb4a1FPLkJo

如果用lkl-bbr加速HTTP的话，后端就不能获取正确的IP地址了

使用rinetd会“隐藏”掉源IP，相当于代理访问，加速HTTP的话，后端就不能获取正确的IP地址了。
是否有办法能够hack掉这个问题？
我只是提交issues，供作者参考。并没有“要求”的意思，现在能够在ovz上使用bbr已经很不错了

使用反馈 增强版BBR和PCC

增强版BBR经常断流.看youtube只能720p .
换成PCC之后明显不断流了. 由于VPS cpu限制
E5620 @ 2.40GHz 1核(openvz)
只能跑到50m cpu就100%占用了. 几乎内核和ssr server 个占50% .偶尔的内核更多些. 断流几乎没有过.很爽....
4.x内核中的BBR跑到90m 也就3%左右的内核占用. %15左右[kvm cpu MHz : 2599.996]

谢谢...... 以star
现在就去把原来的KCP删掉....[用kcp 老是被认为发包攻击...]

请问在编译lkl 的时候有没有出现无法编译通过的情况

我在编译lkl到kernel/cgroup_freezer.c的时候，提示存在三个未定义的变量，请问您遇到了吗？

请问在端口转发的机器上，rinetd.conf该如何写获得BBR加速呢

情况是有A，B两台VPS，在A进行端口转发到B，客户端SS通过A中转后连接B的SS服务。
iptables配置为：
iptables -t nat -A PREROUTING -p tcp -m tcp --dport A端口 -j DNAT --to-destination B:端口
iptables -t nat -A PREROUTING -p udp -m udp --dport A端口 -j DNAT --to-destination B:端口
iptables -t nat -A POSTROUTING -d 目标IP -p tcp -m tcp --dport B端口 -j SNAT --to-source A
iptables -t nat -A POSTROUTING -d 目标IP -p udp -m udp --dport B端口 -j SNAT --to-source A
可以正常使用SS。
在配置A机器上的rinetd.conf
0.0.0.0 A端口 0.0.0.0 A端口
或另用一个C端口，客户端SS连C端口
0.0.0.0 C端口 0.0.0.0 A端口
都无法连通，请问这种情况下能不能实现lkl加速。

Does LKL_HIJACK_SYSCTL parameter work for rinetd integrated with liblkl?

Greeting,

AFAIK the original liblkl-hijack.so can tune sysctl parameters for optimal performance by setting LKL_HIJACK_SYSCTL , however unfortunately it's rely on support of tun/tap, and now here we are thanks to this wonderful branch using raw-socket instead.

And now looks like the most of users using rinetd integrated with liblkl rather than liblkl-hijack.so. So is LKL_HIJACK_SYSCTL parameter works for rinetd integrated with LKL?

Or Does liblkl-hijack.so compiled from https://github.com/linhua55/linux/tree/rinetd_bpf use raw-socket in replacement of tun/tap ?

DUP in when ping host running rinetd?

When rinetd_bbr_powered is running on VPS, and make a ping from other host, I got

64 bytes from 63.223.xxx.xx (63.223.xxx.xx): icmp_seq=1 ttl=47 time=211 ms
64 bytes from 63.223.xxx.xx (63.223.xxx.xx): icmp_seq=1 ttl=47 time=211 ms (DUP!)
64 bytes from 63.223.xxx.xx (63.223.xxx.xx): icmp_seq=2 ttl=47 time=194 ms
64 bytes from 63.223.xxx.xx (63.223.xxx.xx): icmp_seq=2 ttl=47 time=194 ms (DUP!)

部分OVZ的VPS无法使用

从重装系统到安装LKL-BBR，同样的操作，VPS A正常,VPS B不正常
VPS B启动rinetd_bbr以后，加速的端口就通通无法连接了，结束掉进程以后恢复正常。
不正常的VPS B是directspace家的
如果需要的话，我可以提供root账号进行测试

关于ipv6上能否使用的问题

配置上面加了 <ip> <port> <ip> <port>
可以支持ipv4.请问目前能否对ipv6支持? 谢谢

能否创建一个传统的/etc/init.d这样的启动脚本，有些vps没有systemctl

write to fd netdev fails: No buffer space available

此错误经常发生，使得端口无法连接，手工重启后恢复。

Dec 20 17:07:57 vultr1 systemd[1]: Started rinetd with bbr.
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Linux version 4.10.0+ ([email protected]) (gcc version 7.1.1 20170516 (GCC) ) #70 Wed Jul 12 01:00:04 CST 2017
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] bootmem address range: 0x7ff9fc000000 - 0x7ff9fffff000
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16159
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Kernel command line:  virtio_mmio.device=268@0x1000000:1
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] PID hash table entries: 256 (order: -1, 2048 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] Memory available: 64492k/0k RAM
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] NR_IRQS:4096
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] lkl: irqs initialized
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000003] lkl: time and timers initialized (irq2)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000012] pid_max: default: 4096 minimum: 301
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000545] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.000549] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.112295] console [lkl_console0] enabled
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.112396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.120331] NET: Registered protocol family 16
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.120878] clocksource: Switched to clocksource lkl
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.131057] NET: Registered protocol family 2
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.134798] TCP established hash table entries: 512 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.135073] TCP bind hash table entries: 512 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.135231] TCP: Hash tables configured (established 512 bind 512)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.137512] UDP hash table entries: 128 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.137744] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes)
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.138399] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100010b, IRQ 1.
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.138864] workingset: timestamp_bits=62 max_order=14 bucket_order=0
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.141465] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA.  Trying to continue, but this might not work.
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.149361] NET: Registered protocol family 10
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.152786] Segment Routing with IPv6
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.153543] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.154103] Warning: unable to open an initial console.
Dec 20 17:07:57 vultr1 rinetd-bbr[512]: [    0.155010] This architecture does not have kernel memory protection.
Dec 20 17:07:58 vultr1 rinetd[512]: Starting redirections...
Dec 20 17:08:00 vultr1 rinetd-bbr[512]: [    2.570950] random: fast init done
Dec 20 17:08:37 vultr1 rinetd-bbr[512]: [   39.779961] random: crng init done
Dec 31 17:17:35 vultr1 rinetd-bbr[512]: write to fd netdev fails: No buffer space available

如何删除呢？

今天出现了问题，之前都正常，今天我去配置了一下 config 然后发现，加速后的端口不能访问。关闭服务之后能访问。重新下载脚本安装也不行。我想反安装试试，您能告诉我如何反安装吗？

realse里面的三个版本有什么区别？

bbr默认的，bbr魔改的，那个ppc的有什么特殊地方？

用一键脚本装后 SS UDP转发很容易失败

用的也许是流行最广的秋水逸冰的 SS Python版，还没使用LKL BBR 魔改版（一键脚本）的时候使用SSTAP测试UDP还能发现UDP功能工作正常。

使用LKL RetinD之后再用SSTAP测试UDP就显示UDP转发未工作了。

关于是否生效

使用的加强版，目前连接没问题，而且确定有效，只是不知道如何判断是否发挥了它真正的功效。
启动的日志贴在下面了，其中有几处疑问，已做了明显标记（行末添加了大量#####？？？）
我特意做过对比，重装系统多次，新系统首次启动并未有这种“报错”，但是重启程序就会这么报错了，此时重启vps再启动依然没用，只有重装系统首次启动才没有这种日志。
而且无报错的情况下，速度好像快的多。所以我怀疑我现在的情况是bbr生效，但“加强版”貌似没生效。

PACKET_QDISC_BYPASS, ignoring: Protocol not [available####################################?????????]
[    0.000000] Linux version 4.10.0+ ([email protected]) (gcc version 7.1.1 20170516 (GCC) ) #62 Mon Jul 3 02:14:58 CST 2017
[    0.000000] bootmem address range: 0x7fcd40000000 - 0x7fcd43fff000
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16159
[    0.000000] Kernel command line:  virtio_mmio.device=268@0x1000000:1
[    0.000000] PID hash table entries: 256 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Memory available: 64492k/0k RAM##########################################????????
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:4096
[    0.000000] lkl: irqs initialized
[    0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000001] lkl: time and timers initialized (irq2)
[    0.000007] pid_max: default: 4096 minimum: 301
[    0.000035] Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.000037] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
[    0.010727] console [lkl_console0] enabled
[    0.010733] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.011867] NET: Registered protocol family 16
[    0.012191] clocksource: Switched to clocksource lkl
[    0.012304] NET: Registered protocol family 2
[    0.012532] TCP established hash table entries: 512 (order: 0, 4096 bytes)
[    0.012540] TCP bind hash table entries: 512 (order: 0, 4096 bytes)
[    0.012546] TCP: Hash tables configured (established 512 bind 512)
[    0.012721] UDP hash table entries: 128 (order: 0, 4096 bytes)
[    0.012729] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes)
[    0.012828] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100010b, IRQ 1.
[    0.014900] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    0.014947] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA.  Trying to continue, but this might not work.
[    0.015222] NET: Registered protocol family 10
[    0.016149] Segment Routing with IPv6
[    0.016166] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.016424] Warning: unable to open an initial console.####################################????????
[    0.016440] This architecture does not have kernel memory protection.########################????????
[    2.012309] random: fast init done
[   98.492446] random: crng init done

iptables filter table rules no longer working

Dear author,

Thanks a lot for your work! I have a question about the iptables rules. I have defined some customized iptables rules in filter table to drop incoming traffics from certain IP addresses. However with rinetd-bbr installed, the rules no longer work for the accelerated TCP port, likely because raw table has higher priority than the filter table. To be honest I'm not very familiar with iptables. I was wondering how can I make the filter rules to work again.

Thank you so much!

在编译lol

最后执行systemctl start rinetd.service错误

提示Failed to get D-Bus connection: Unknown error -1

BBR over UDP的可能性

貌似大佬你对lkl很了解，想邀你讨论下udp上面bbr:) 这个想法动机有这么几个:

1. 个人比较抵制kcp这种重复发包，觉得BBR更符合公平与整体效率之上原则。
2. BBR over UDP可以去掉root权限，不用kvm，不用openvz，甚至不需要root权限，甚至是windows/OSX都能跑lkl。只要允许udp流量的平台就能跑，适应性更强。
3. 目前社区很多精力都放在tcp流量伪装上面，其实就是为了绕过ISP针对udp的限速。而部分运营商（比如我这里的联通）并没有这样的qos。BBR over UDP在这种环境非常合适。

目前我能想到的BBR over UDP途径有俩, 一个是quic, 但是go-quic项目没有支持bbr的迹象。第二个就是lkl, 但我看到的libhijack需要绑定整个物理网卡或者一个tap设备，这样必须要有root权限。有没有可能让lkl只需要绑定一个udp socket或者af_unix socket或者pipe，然后把这个当成一个虚拟的L2网卡呢？

执行systemctl stop rinetd后，设置的iptables规则没有被清除掉

没有彻底地把 system函数 替换成 MySystem函数

tool systemctl is not installed, abort.

apt-get install systemd 了，还是安装不成功……

实现 基于 packet socket/TPACKET_V2 (packet mmap) 的 zero copy特性以节省CPU

openVZ的2.6内核不支持TPACKET_V3

参考关键词： virtio-net , vhost-user
http://www.udpwork.com/item/12720.html
https://lwn.net/Articles/346267/ raw socket backend
http://blog.allenx.org/2015/07/13/the-new-feature-vhost-user-in-qemu word explain
http://www.virtualopensystems.com/en/solutions/guides/snabbswitch-qemu/ Vhost-user Architecture
https://lists.gnu.org/archive/html/qemu-devel/2014-05/msg05443.html Data path Vring
http://royluo.org/2014/08/09/virtio-netdev-send/ virtio
http://elixir.free-electrons.com/linux/latest/source/drivers/vhost/vhost.c
http://royluo.org/2014/08/22/vhost/ Vhost 详细数据结构，传递流程
http://royluo.org/2014/08/08/virtio-virtqueue/ Vring数据结构
http://royluo.org/2014/07/26/netdev-virtual-3/
https://lwn.net/Articles/388707/ 重要

1. 内存中数据结构的对应， packet header 和 packet body的分别对应
2. 如何更新packet，内存中packet的更新方法

请问有计划支持最新5.4版本内核的lkl吗？

感谢大大编译的lkl，用的很舒服，拯救了我的OpenVZ VPS
最近发现原版的 https://github.com/lkl/linux 在2022-04-11日更新内核版本到了5.4版本
不知道可以继续编译个最新的版本吗？自己编译的无法使用，感谢~

使用raw socket backend的情况下是否default_qdisc=fq的配置是无法生效的？

是不是只有tap设备才能设置qdisc策略？BBR说明中说需要使用fq策略，如果未使用可能会引起延迟问题，那么使用raw socket backend会不会导致BBR不能很好的工作？

对于多个ip地址的vps要怎样做？

支持同时存在两个虚拟网卡 venet0:0 和venet0:1 的lkl吗？

能否同时支持TCP Fastopen？

开启后ping出现DUP!

64 bytes from : icmp_seq=1 ttl=53 time=288 ms
64 bytes from : icmp_seq=1 ttl=53 time=288 ms (DUP!)