cunicu / cunicu Goto Github PK

View Code? Open in Web Editor NEW

145.0 12.0 10.0 14.84 MB

A zeroconf peer-to-peer mesh VPN using Wireguard® and Interactive Connectivity Establishment (ICE)

Home Page: https://cunicu.li

License: Apache License 2.0

Dockerfile 0.05% Go 93.82% Makefile 0.53% Shell 2.10% C 0.77% JavaScript 1.88% CSS 0.39% Nix 0.46%

wireguard vpn ice golang go mesh nat-traversal vpn-manager wireguard-vpn edge-cloud

cunicu's Introduction

🚧 cunīcu is currently still in an Alpha state and not usable yet

cunīcu is a user-space daemon managing WireGuard® interfaces to establish a mesh of peer-to-peer VPN connections in harsh network environments.

To achieve this, cunīcu utilizes a signaling layer to exchange peer information such as public encryption keys, hostname, advertised networks and reachability information to automate the configuration of the networking links. From a user perspective, cunīcu alleviates the need of manual configuration such as exchange of public keys, IP addresses, endpoints, etc.. Hence, it adopts the design goals of the WireGuard project, to be simple and easy to use.

Thanks to Interactive Connectivity Establishment (ICE), cunīcu is capable to establish direct connections between peers which are located behind NAT firewalls such as home routers. In situations where ICE fails, or direct UDP connectivity is not available, cunīcu falls back to using TURN relays to reroute traffic over an intermediate hop or encapsulate the WireGuard traffic via TURN-TCP.

It relies on the awesome pion/ice package for ICE as well as bundles the a Go user-space implementation of WireGuard in a single binary for systems in which WireGuard kernel support has not landed yet.

With these features, cunīcu can be used to quickly build multi-agent systems or connect field devices such as power grid monitoring infrastructure into a fully connected mesh. Within the ERIGrid 2.0 project, cunīcu is used to interconnect smart grid laboratories for geographically distributed simulation of energy systems.

Documentation

cunīcu's documentation can be found here: cunicu.li/docs.

Authors

Steffen Vogel (@stv0g)

License

cunīcu is licensed under the Apache 2.0 license.

SPDX-FileCopyrightText: 2023 Steffen Vogel [email protected]
SPDX-License-Identifier: Apache-2.0

Funding acknowledgement

The project has been initiated by Steffen Vogel while working at the Institute for Automation of Complex Power Systems (ACS) of RWTH Aachen University.

The development of cunīcu has been supported by the ERIGrid 2.0 project of the H2020 Programme under Grant Agreement No. 870620

Trademark

WireGuard and the WireGuard logo are registered trademarks of Jason A. Donenfeld.

cunicu's People

Contributors

Stargazers

Watchers

Forkers

erigrid2 braginini panyongwei huddhudd j4ckzh0u pjungkamp gnu-linux-libre zakuwaki simhaonline longlongagobysky

cunicu's Issues

Refactoring: separate core Interface/Peer handling from ICE specific code

Make version available via RPC for gRPC signaling server and main daemon

Introduce Syncable interface for features which is used by RPC and daemon

Use controller-runtime envtest package for testing the k8s signaling backend

https://book.kubebuilder.io/reference/envtest.html

Setup {stun,turn}.cunicu.li to accept TLS connections on port 443

Add support for whitelist to accept only a limited list of peers via peer discovery

Add 'domain' option to customize top-level domain for hosts file synchronization entries

Add cunicu to FEIN Aachen e.V. projects list

See: https://fein-aachen.org/projects/cunicu/

Remove advanced CLI options and move them to the config file

Lets try to keep the command line interface as simple as possible for the common use cases.

Advanced settings can be done via a YAML config file or alternatively via environment variables.

This change will require use to use the config file also in the E2E tests.

Use counter/timestamp for signaling message replay protection

Use Ginkgo and Gomega for testing

Create simple and advanced configuration file examples

The complexity of the configuration file is getting out of hand.
However, most of the settings are really advanced and will probably never be used by 99% of users.

Lets move those settings to an advanced example config file.

Add new 'manual' signaling backend

For testing a manual signaling backend would be nice.

Instead of relying on a public signaling server, it would simply print out the signaling messages as Base64 encoded strings in the logfile and ask the user to copy & paste them at the terminal of the receiving peers.

As an added bonus, we could add sub-commands to the cunicu CLI tool to handle the manual exchange of signaling messages:

cunicu signaling send [-f] [-j] [<peer>]
Could print lines in the format of <peer> <msg> of messages <msg> which need to be manually exchanged with the peer <peer>.
The option -f could enable streaming output of messages.
The option -j could enable JSON output.
cunicu signaling recv [-j] (-f|<peer> <msg>)
Could receive signaling messages in the same format as procuced by cunicu signaling send.
The option -j would enable JSON output.
The option -f would read the messages via stdin instead.

Using these commands the user could easily implement their own signaling backend using shell scripts or via RFC1149.

E.g. via SSH:

cunicu signaling send -fj | ssh user@remote-peer cunicu signaling recv -j

Initial work for this feature has been started in the backend-manual branch.

Throw error if config file is world-readable

... and contains secrets?

Use ephemeral session key for signaling messages

Add new signaling backend by routing messages via already connected peers through established tunnels

Instead of relying on a dedicated signalling server, we could also send signaling messages via other peers to whom we already established a tunnel. These peers are then tasked with relaying the messages to the desired destination.

I am currently thinking about using SCTP associations as SCTP provides a couple of interesting properties.

One open question remains: How do we facilitate the routing/relaying of messages between the peers?

Examples

Simple

In this simple example the peers p2 and p3 already have an established connection to peer p1.
However they are still missing a direct connection.
In this case both p2 and p3 could connect the the gRPC service of p1 to exchange signalling messages.

flowchart LR
    p1
    p2
    p3

    p1 --- p2
    p1 --- p3
    p2 -..- p3

Advanced

In this advanced example, p2 and p3 have no established connection to a common peer.
So p1 and p4 would need to relay messages on their behalf.
This would require some more complex routing logic.

flowchart LR
    p1
    p2
    p3
    p4

    p1 --- p4
    p1 --- p2
    p4 --- p3
    p2 -..- p3

Ideas

The main question is: can we delegate routing to the Kernel using its FIB?
If yes, we must not use link-local addresses but use IPv6 addresses with a broader scope.

And we need a routing protocol/daemon embedded into cunico (see #10)

Use more hole punching techniques

Implement routing protocol for partial meshes

Currently, cunicu expects to build a full mesh between all peers to establish full reachability.
In cases where peer connections can not be directly established, we fall back to TURN relays which transparently proxy the Wireguard UDP packets over a third party.

Alternatively, we could also limit ourself to a partial mesh and use existing peer connections to route trafic to other peers.
This would require the implementation of a routing protocol between the peers.

This approach is also used by Tinc-VPN.

Advantages

We could easily implement a signaling backend over the partial mesh (see #9)
We could limit the number of active peer connections as establishing dedicated peer connections could not scale for a large number of peers

Disadvantages

Routed trafic would be interceptable by the peers which as a routing hop
- In contrast: TURN relays would only see encrypted trafic

Open Questions

Which routing protocol to use?
- Should we use an established protocol like IS-IS or BGP and embed an routing daemon into cunicu?
- Existing implementations in Go:
  - bio-rd
  - gobgp
- Or should we implement a custom protocol based on a Link-state or Path-vector routing scheme?
  - Implementing our own would allow us to customize it for the Wireguard / WICE environemnt
    - Use public keys instead of IP addresses to build a routing table
    - Use gRPC as a transport like wice is using for most other signaling and RPC tasks.

Use UDPMux conn directly for wireguard-go.Bind

instead of passing through ice.Conn

Handle change of ListenPort on Wireguard interfaces

We already monitor the managed Wireguard interfaces for changes.
E.g. the addition or removal of new interfaces, peers and handshake times.

However, we currently assume that the ListenPort an interface does not change while wice is managing the interface.

If it does, we need:

Update the eBPF filter for the FilteredUDPConn which backs our UDPMux
Update LocalAddr of UDPMux
Restart ICE for all peers to negotiate new Candidate Pair using updated LocalAddr
Make sure the Proxy instances for all peers are updated to send their data to the new port
Update the NFTables rules used for redirecting trafic of UDPMuxSrflx to the Wireguard interface

Add support for Path MTU Discovery (PMTUD)

cunicu currently attempts at auto-detecting a correct tunnel MTU by taking the link/route MTUs into consideration.
However, this will not be optimal as the path MTU can be smaller than the link MTUs.
Such cases can be detected via PMTUD.

However, there is another twist to this.
In larger WireGuard meshes we somehow need to coordinate all peers to use the smallest of all peer-to-peer path MTUs.
This can be achieved via our signaling backend by including a detected path MTU into the peer descriptions.

Use netlink multicast subscription for notification of Wireguard peer changes

There has been a patch on the Wireguard mailing list add support for netlink multicast subscriptions which allow for listening to changes of the peers of an interface:

https://lore.kernel.org/patchwork/patch/1366219/

However this patch hasnt made it up into the Linux tree yet.

We could use this for accelerating the detection of new peers or peer handshakes.

There is also a thread on the Wireguard mailing list discussing this feature:

https://lists.zx2c4.com/pipermail/wireguard/2021-March/006463.html

Connection Manager

We might want to think about introducing a per-peer connection manager which handles zero, one or more concurrent ICE agents.

Motivation

Connection Upgrades

ICE does not automatically upgrade to better candidate pairs once they become available (e.g. new network interface comes up, upstream topology/routing changes).

Hence, we could periodically create new ICE agents/connections in the background to probe for better candidate pairs and only switch over the WireGuard traffic if we have found a better alternative.

Connection Fallbacks

Keeping multiple ICE connections between two pairs could allow us to quick failover to another candidate pair if the current one breaks.

This is only really of advantage if a quick failover is required and a normal ICE restart would take too long.

Passive peers

In setups with a very high number of peers a fully meshed VPN topology might be undesirable as in results in an excessive number of ICE connections and open ports.

A connection manager could put peers into a passive state in which we do not attempt to establish a direct connection, but rather fall back to dynamic routing (#10).

The connection manager could nominate peers to become active in case we see direct connection attempts or increased traffic to networks connected to that peer.

It also opens a whole new research question: which partial mesh is the optimal one?

Use in-process pipe for user-space Wireguard (Bind interface)

Currently, we communicate with a user-space Wireguard daemon via a standard socket through kernel-space.
This can be optimized by implementing the conn.Bind interface of the wireguard package.

Some initial work started in branch wg-conn-bind.

Refactoring: replace interface{} by any

Accept per-URL credentials for STUN/TURN urls

Relevant RFCs:

Unfortunately, the RFC do not support specifying the TURN credentials as part of the URI.

Support eBPF proxying for relayed candidates

Transform UDP packets with eBPF program to insert/strip TURN channel IDs.

Test co-existence of multiple wice daemons

We have to check if there are any collisions in nftables rules.

Investigate more options for signaling backends

I am still looking for an elegant signaling backend which does not rely on a single service hosted by a third party.
Users should be able to get started using wice without setting up their own signaling server and should not be required to depend on a signaling service hosted by somebody else.

In most cases these self-hosted signaling servers represent a single point of failure.
They also allow the operator of the signaling server to observe which public facing endpoints are exchanging signaling information. (This actual signaling message contents however are always encrypted by wice.)

Ideally we would piggyback to other established P2P networks like I2P, BitTorrent or IPFS.
They already have an established user-base and plenty of public nodes which participate in a DHT.

Ideas

Public Kademlia Networks
libp2p (initial broken implementation exists)
WebTorrent trackers
BitTorrent (Mainline DHT)
Matrix.org
XMPP
Email
Tor
Manual (#11)
Existing nodes via gRPC (#9)

Allow configuration of routing table

Generate website using docusaurus

https://docusaurus.io/

Clarity on functionality.

Hi, I maintain https://github.com/HarvsG/WireGuardMeshes

To be clear wice does not build your mesh for you, it just propagates endpoint information to help with NAT hole-punching? Said another way. If you add a new peer, other peers will not attempt to establish a connection until you have manually updated all other peers?

Add JSON Schema for configuration file

And embed it with Redocusaurus into website.

Remove auto-generated usage docs from repo

Implement integration test-cases

Test cases

#	Topology	# Hosts	Signaling	Proxy	Relay
1	1	2	gRPC	User	❌
2	1	2	gRPC	eBPF + NFTables	❌
3	2	3	gRPC	User	❌
4	2	3	gRPC	eBPF + NFTables	❌
5	3	2	gRPC	eBPF + NFTables	✅ blocked P2P UDP
6	3	2	gRPC	eBPF + NFTables	✅ blocked UDP entirely
7	4	2	gRPC	User	❌
8	5	2	gRPC	User	❌

Topologies

Acronyms

hX Host
swX Switch
fwX Firewall
nX Router with NAT
rX Relay
sX Signaling server

1 - Simple

graph TD
    sw1{{sw1}}
    s1((s1))

    s1 --- sw1
    sw1 --- h1
    sw1 --- h2

2 - Simple with 3-hosts

graph TD
    sw1{{sw1}}
    s1((s1))

    s1 --- sw1
    sw1 --- h1
    sw1 --- h2
    sw1 --- h3

3 - Simple with Relay and firewall

graph TD
    sw1{{sw1}}
    s1((s1))
    r1((r1))

    r1 --- sw1
    s1 --- sw1
    sw1 --- fw1(fw1)
    sw1 --- fw2(fw2)
    fw1 --- h1
    fw2 --- h2

4 - Simple with 1-level NAT and Relay

graph TD
    sw1{{sw1}}
    s1((s1))
    r1((r1))
    n1[/n1/]

    r1 --- sw1
    s1 --- sw1
    sw1 --- n1
    n1 --- h1
    sw1 --- n2
    n2 --- h2

5 - Simple with 2-level CG-NAT and Relay

graph TD
    sw1{{sw1}}
    sw2{{sw2}}
    s1((s1))
    r1((r1))
    n1[/n1/]
    n2[/n2/]
    n3[/n3/]

    r1 --- sw1
    s1 --- sw1
    sw1 --- n1
    n1 --- sw2
    sw2 --- n2
    n2 --- h1
    sw2 --- n3
    n3 --- h2