osrg/rustybgp

BGP implemented in the Rust Programming Language

RustyBGP: BGP implementation in Rust

The mission is to develop a high-performance and safe BGP implementation; an experiment to implement aged and rusty BGP protocol in a modern language. RustyBGP is much faster than other OSS implementations. One reason of the high performance is that RustyBGP is designed to exploit multicore processors. Here is a CPU usage comparison with FRR 7.5 during processing 32 peers with 800K prefixes each; RustyBGP (left) uses all the cores while FRR uses only few.

RustyBGP supports the gRPC APIs same as GoBGP; your code to manage GoBGP via the APIs should work with RustyBGP. If you need CLI, GoBGP CLI tool allows you to manage RustyBGP. RustyBGP also supports the same configuration file format as GoBGP (only toml and yaml for now).

Get Started

You can easily build RusyBGP on any system that has Docker running. You don't need Rust development environment. You can build the x86_64 statically-linked binary as follows:

$ git clone https://github.com/osrg/rustybgp.git
$ cd rustybgp
$ docker pull ghcr.io/rust-cross/rust-musl-cross:x86_64-unknown-linux-musl
$ docker run --rm -it -v "$(pwd)":/home/rust/src ghcr.io/rust-cross/rust-musl-cross:x86_64-unknown-linux-musl cargo build --release
$ ls target/x86_64-unknown-linux-musl/release/rustybgpd
target/x86_64-unknown-linux-musl/release/rustybgpd

$ sudo ./target/x86_64-unknown-linux-musl/release/rustybgpd -f gobgpd.conf
Hello, RustyBGP (32 cpus)!

Then you can manage the daemon on a different terminal with GoBGP's CLI command.

$ gobgp neighbor
Peer            AS Up/Down State       |#Received  Accepted
198.51.100.2 65002   never Idle        |        0         0

If you just want to check out the performance, start the daemon with --any-peers option. The daemon accepts any peers without configuration.

$ sudo ./target/x86_64-unknown-linux-musl/release/rustybgpd --as-number 65001 --router-id 203.0.113.1 --any-peers
Hello, RustyBGP (32 cpus)!

Supported Features

Currently, the very basic BGP features are supported; eBGP and iBGP, acstive/passive connection, RPKI, BMP (BGP monitoring protocol), MRT, etc with the following gRPC APIs.

API	Relevant CLI	Note
start_bgp	gobgp global as <VALUE> router-id <IP>
get_bgp	gobgp global
add_peer	gobgp neighbor add <IP> as <VALUE> router-id <IP>	v4/v6 families and addpath (rx) supported
delete_peer	gobgp neighbor del <IP>
list_peer	gobgp neighbor/gobgp neighbor <IP>
enable_peer	gobgp neighbor <IP> enable
disable_peer	gobgp neighbor <IP> disable
add_peer_group
add_dynamic_neighbor
add_path	gobgp global rib add <PREFIX>
delete_path	gobgp global rib del <PREFIX>
list_path	gobgp global rib/gobgp neighbor <IP> [adj-in|adj-out]
add_path_stream	gobgp mrt global inject [FILE]
get_table	gobgp global rib summary
add_policy
list_policy
add_defined_set
list_defined_set
add_statement
list_statement
add_policy_assignment
list_policy_assignment
add_rpki	gobgp rpki server <IP> add
list_rpki	gobgp rpki server
list_rpki_table	gobgp rpki table
enable_mrt
add_bmp	gobgp bmp add	routemonitoring is supported only with adjin
list_bmp	gobgp bmp

Community, discussion and support

You have code or documentation for RustyBGP? Awesome! Send a pull request. No CLA, board members, governance, or other mess. See BUILD.md for info on code contributing.

rustybgp's Issues

RustyBGP performance issues

Working on my next blog post about testing performance of bgp software. Here is a draft: https://github.com/elegantnetwork/elegantnetwork.github.io/blob/master/_drafts/2021-08-120-comparing-open-source-bgp-internet-routes.md, I found a bunch of issues with rustybgp. I cannot replicate the performance that you have shown. Also, there are other issues that I've written up in the draft blog post

Dependent create treebitmap has been deprecated

The treebitmap crate has been deprecated so builds are failing:

$ cargo build --release
    Updating crates.io index
error: no matching package named `treebitmap` found
location searched: registry `crates-io`
required by package `rustybgpd v0.2.0 (/tmp/rustybgp/daemon)`

Error while building

When I tried to build RustyBGP on fresh Linux installation with Docker, I got this:

# docker run --rm -it -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder cargo build --release
...
Compiling tonic-build v0.5.2
Compiling hyper v0.14.25
error[E0599]: no method named `send_replace` found for struct `Arc<tokio::sync::watch::Sender<Option>>` in the current scope
--> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.25/src/client/connect/mod.rs:225:17
|
225 | self.tx.send_replace(Some(connected.clone()));
| ^^^^^^^^^^^^ method not found in `Arc<tokio::sync::watch::Sender<Option>>`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `hyper` due to previous error
warning: build failed, waiting for other jobs to finish...
error: build failed

Is there something wrong with my setup?
I have Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.2

Is the watcher functionality supported?

Does RustyBGP have similar functionality as GoBGP watcher to notify path information to all backend clients?

weird number of prefixes received and resources used when testing with gobgp mrt files

I'm using gobgp to send mrt data from a routeviews mrt file. I'm trying to send 100K routes. For some reason gobgp doesn't actually send that many, but it's close. I have 10 gobgp processes, each in it's own container sending data and one monitor just listening and receiving routes.

What's weird is that here are different numbers of accepted routes, including one that is 93051, just 93% of the data.

$ docker exec bgperf_rustybgp_target /root/gobgp neighbor
Peer         AS  Up/Down State       |#Received  Accepted
10.10.0.4  1004 00:04:29 Establ      |    99305     99305
10.10.0.9  1009 00:02:55 Establ      |    99305     99305
10.10.0.3  1003 00:04:46 Establ      |    99305     99305
10.10.0.8  1008 00:03:12 Establ      |    99272     99272
10.10.0.2  1001 00:04:53 Establ      |        0         0
10.10.0.7  1007 00:03:32 Establ      |    99305     99305
10.10.0.12 1012 00:01:59 Establ      |    93051     93051
10.10.0.6  1006 00:03:53 Establ      |    99236     99236
10.10.0.11 1011 00:02:18 Establ      |    93051     93051
10.10.0.5  1005 00:04:07 Establ      |    99305     99305
10.10.0.10 1010 00:02:39 Establ      |    93051     93051

When I do the same test with FRR 7.5.1, it's consistent.

$ docker exec bgperf_frrouting_target vtysh -c 'sh ip bgp summary'

IPv4 Unicast Summary:
BGP router identifier 10.10.255.254, local AS number 1000 vrf-id 0
BGP table version 99309
RIB entries 181579, using 33 MiB of memory
Peers 11, using 157 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
10.10.0.2       4       1001        13     28020        0    0    0 00:05:35            0    99207
10.10.0.3       4       1003    100014     28019        0    0    0 00:05:20        99305    99207
10.10.0.4       4       1004    100015     25786        0    0    0 00:05:02        99305    99207
10.10.0.5       4       1005    100013     23216        0    0    0 00:04:44        99305    99207
10.10.0.6       4       1006    100012     23213        0    0    0 00:04:27        99305    99207
10.10.0.7       4       1007    100012     23213        0    0    0 00:04:09        99305    99207
10.10.0.8       4       1008    100011     23212        0    0    0 00:03:54        99305    99207
10.10.0.9       4       1009    100011     23212        0    0    0 00:03:36        99305    99207
10.10.0.10      4       1010    100010     23211        0    0    0 00:03:15        99305    99207
10.10.0.11      4       1011    100009     23210        0    0    0 00:02:58        99305    99207
10.10.0.12      4       1012    100008     23210        0    0    0 00:02:39        99305    99207

Total number of neighbors 11

and gobgp as the target

$ docker exec bgperf_gobgp_target /root/gobgp neighbor
Peer         AS  Up/Down State       |#Received  Accepted
10.10.0.2  1001 00:05:52 Establ      |        0         0
10.10.0.3  1003 00:05:43 Establ      |    99305     99305
10.10.0.4  1004 00:05:24 Establ      |    99305     99305
10.10.0.5  1005 00:05:06 Establ      |    99305     99305
10.10.0.6  1006 00:04:50 Establ      |    99305     99305
10.10.0.7  1007 00:04:30 Establ      |    99305     99305
10.10.0.8  1008 00:04:13 Establ      |    99295     99295
10.10.0.9  1009 00:03:54 Establ      |    99305     99305
10.10.0.10 1010 00:03:35 Establ      |    99305     99305
10.10.0.11 1011 00:03:14 Establ      |    99305     99305
10.10.0.12 1012 00:02:58 Establ      |    99305     99305

Now the even more confusing, If I don't stop the tests, they just keep going forever with rustybgpd growing in memory and using lots of CPU. One test I had ran for almost an hour and it was up to > 50 GB of RAM. Even though the amount of received or accepted prefixes does not change.

Actually, I see the same behavior with gobgp as the target. So maybe there's a problem with gobgp as the mrt sender?

With FRR, after gobgp sends the 99305 prefixes then both gobgp and frr don't grow or consume CPU, so it works fine with FRR.

MRT injecting fails with "failed to send: EOF"

Hello,

First of all - thank you for this great tool 🙏

I'm trying to load an MRT dump file(https://data.ris.ripe.net/rrc03/2023.01/bview.20230101.0000.gz) using the gobgp CLI.

It works for ten records, but when I go to 100 or more, it returns:

root@abc:~# gobgp mrt inject global bview.20230101.0000 10
root@abc:~# gobgp mrt inject global bview.20230101.0000 100
failed to send: EOF

Do you have any idea why this could happen?

I appreciate any help you can provide.

Kamil

how to find out that a peer has sent all the routes

in bgperf, it uses the count of how many prefixes received from a peer to know that a test has finished. When adding filter testing, just knowing the number of prefixes accepted isn't enough. It needs to know either the number of prefixes received before filters or the number of prefixes filtered, or that End-of-RIB has been sent. Is there anyway to get any of those things? with BIRD you can see the number of prefixes received and the number accepted. With FRRouting and OpenBGPD you can see when EOR has been sent.

getting a panic with latest code

using bgperf with bgpdump to send traffic

 python3 bgperf.py bench -n5 -trustybgp -p800000 -gbgpdump2 -s --mrt-file=/home/jpietsch/bgperf/rib.20210801.0000

get this output

Hello, RustyBGPd (32 cpus)!
thread '<unnamed>' panicked at 'inconsistent in drop', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.18/src/str
eam/futures_unordered/abort.rs:11:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'panicking twice to abort the program', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/futures-uti
l-0.3.18/src/stream/futures_unordered/abort.rs:6:13
stack backtrace:
   0:     0x56275aa72b6c - std::backtrace_rs::backtrace::libunwind::trace::h3fea1eb2e0ba2ac9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/libunwind.rs
:90:5
   1:     0x56275aa72b6c - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x56275aa72b6c - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x56275aa72b6c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x56275aa9498c - core::fmt::write::h31b9cd1bedd7ea38
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17
   5:     0x56275aa6cad5 - std::io::Write::write_fmt::h1fdf66f83f70913e
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15
   6:     0x56275aa746e0 - std::sys_common::backtrace::_print::he7ac492cd19c3189
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x56275aa746e0 - std::sys_common::backtrace::print::hba20f8920229d8e8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x56275aa746e0 - std::panicking::default_hook::{{closure}}::h714d63979ae18678
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50
   9:     0x56275aa74297 - std::panicking::default_hook::hf1afb64e69563ca8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9
  10:     0x56275aa74d94 - std::panicking::rust_panic_with_hook::h02231a501e274a13
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:624:17
  11:     0x56275a9d83b5 - std::panicking::begin_panic::{{closure}}::hf826dffca46d009a
  12:     0x56275a9d8384 - std::sys_common::backtrace::__rust_end_short_backtrace::h86a5312ea5900044
  13:     0x56275a3102bc - std::panicking::begin_panic::hc8f1edb280a2e8ff
  14:     0x56275a9d82c9 - <futures_util::stream::futures_unordered::abort::abort::DoublePanic as core::ops::drop::Drop>::drop::h0336f65bb09
6aed5
  15:     0x56275a9d8237 - core::ptr::drop_in_place<futures_util::stream::futures_unordered::abort::abort::DoublePanic>::hb07f9406a11a3a1e
  16:     0x56275a9d82ae - futures_util::stream::futures_unordered::abort::abort::h3261bb903d9ba561
  17:     0x56275a6a4cb2 - core::ptr::drop_in_place<futures_util::stream::futures_unordered::FuturesUnordered<futures_util::stream::stream::
next::Next<tokio_stream::wrappers::mpsc_unbounded::UnboundedReceiverStream<rustybgpd::event::ToPeerEvent>>>>::h73aeffd02d895279
  18:     0x56275a7873c7 - rustybgpd::event::Handler::run::{{closure}}::h4fdabcfa710f3e5a
  19:     0x56275a746a0b - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hb7e0bb100bcd89eb
  20:     0x56275a8b6935 - tokio::runtime::task::harness::Harness<T,S>::poll::h6ff7a6c1bb31a4e6
  21:     0x56275a8638b1 - std::thread::local::LocalKey<T>::with::ha297eda2128a4fd6
  22:     0x56275a6d0604 - tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on::h635a09920156b075
  23:     0x56275a7e78b5 - tokio::runtime::Runtime::block_on::hf377527235153e53
  24:     0x56275a838025 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf0a8a3f3550fe502
  25:     0x56275a864523 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h33ccad866afb2795
  26:     0x56275aa77e93 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf2a5e508372e2387
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  27:     0x56275aa77e93 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h46642f5b75478456
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  28:     0x56275aa77e93 - std::sys::unix::thread::Thread::new::thread_start::h1fbdb50adbde1cab
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys/unix/thread.rs:106:17
  29:     0x7f57c2788ea7 - start_thread
  30:     0x7f57c256edef - clone
  31:                0x0 - <unknown>
thread panicked while panicking. aborting

I don't get the problem with only 1 tester

BGP add path support

With add-path receive enabled, rustybgp correctly detects that a peer is advertising the ADD-PATH capability, but doesn't import multiple routes:

Neighbor capabilities:
  multiprotocol:
    ipv4-unicast:   advertised and received
  route-refresh:      received
  graceful-restart:   received
  4-octet-as: advertised and received
  add-path:   received
    Remote:
      ipv4-unicast:  send

Steps to reproduce

1. Create config file bgp.yml:

global:
  config:
    as: 65530
    router-id: 192.168.90.4
    port: 179

  apply-policy:
    config:
      default-import-policy: accept-route
      default-export-policy: reject-route

neighbors:
  - config:
      peer-as: 65530
      neighbor-address: 192.168.90.1
    add-paths:
      config:
        receive: true

2. Build and run rustybgp

git clone https://github.com/osrg/rustybgp
cd rustybgp
git reset --hard 9a94087b81679af2304d1523c1bbde822be7010b
cargo build --release
./target/release/rustybgpd -f bgp.yml

3. gobgp neigh and confirm only ~900k routes received

Using gobgp with the same config works correctly.

Stress testing via rtod

I have a tool that I wrote long ago to stress test large numbers of routes, flapping, etc, etc. I was mostly interested in what went wrong when large numbers of routes appeared or disappeared, which is, indeed, perversely interesting. We test too many things when things are normal, and not when the internet is on fire. Also I hope you'll get a laugh from the readme.

https://github.com/dtaht/rtod#rtod-routing-tables-of-death

Perhaps you will find it useful in your testing. I would rather like to rewrite babeld in rust (waaaaay simpler protocol) but am too clueless thus far to do so. Perhaps by following along here, I'll learn enough about it to make that attempt.

Does rustybpg support multipathing/ECMP (e.g., rfc7911)?

Hi,

Looking through the code, I see /multipath/ and /multipathing/ in the configs and APIs (e.g., src/config/gen.rs) but not elsewhere in the code. Can someone please confirm that means that there are the hooks to implement multipathing but it's not actually implemented yet?

Thank you in advance!

Consider submitting rustybgp to memorysafety.org

It might be a good idea to submit rustybgp to the https://www.memorysafety.org project by ISRG, the parent non-profit of Let's Encrypt. And maybe NTT Labs can sponsor the development of this project, which they seem to be doing anyway.

Why do the CI workflows have gobgp binaries?

Why do the original CI build workflows use gobgp?

from: https://github.com/fujita/dockerfiles

I am trying to have a few github workflows to test the docker builds. And would like to get it to work on alpine.

Route Server mode

I am curious as to the roadmap for this effort. In particular, I am looking for a high speed route server, and trying to find alternatives....