What does it mean that the library "implements the Wireguard protocol, without the network or tunnel stacks"?

[cloudflare]

What does it mean that the library "implements the Wireguard protocol, without the network or tunnel stacks"?

[vkrasnov]

First we define a few buffers to maximize performance using batch send/recv:

   internal var wireguardReadBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
    internal var wireguardWriteBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
    internal var wireguardTimerBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: maxPacketSize)
    // Preallocated data arrays
    internal var wireguardReadDatagrams = [Data]()
    internal var wireguardReadPackets = [Data]()
    internal var wireguardReadPacketProtocols = [NSNumber]()
    internal var wireguardWriteDatagrams = [Data]()

The encapsulate method looks like this, it is called by packetFlow.readPacketObjects in an infinite loop, unless there is a connection error.

 /**
     Handle packets from the virual interface. Encapsulate and send to server.

     - parameter packets: The raw IP packets that will be encapsulated.
     */
    internal func handleOutboundWarpPackets(_ packets: [NEPacket]) {

        var offset = 0

        for packet in packets {
            if offset + maxPacketSize > bufferSize {
                // Flush if might not have enough space
                sendDatagrams(datagrams: warp.wireguardWriteDatagrams)
                warp.wireguardWriteDatagrams.removeAll(keepingCapacity: true)
                offset = 0
            }

            // We encapsulate as many packets as we can into the wireguardWriteBuffer
            let writeBuff = warp.wireguardWriteBuffer.advanced(by: offset)

            let res = packet.data.withUnsafeBytes { (ptr: UnsafeRawBufferPointer) -> wireguard_result in
                let dataPtr = ptr.baseAddress?.bindMemory(to: UInt8.self, capacity: packet.data.count)
                return wireguard_write(self.warp.wireguardClient, dataPtr, UInt32(packet.data.count), writeBuff, UInt32(maxPacketSize))
            }

            switch res.op {
            case WRITE_TO_NETWORK:
                self.setClientId(buffer: writeBuff)
                warp.wireguardWriteDatagrams.append(Data(bytesNoCopy: writeBuff, count: Int(res.size), deallocator: .none))
                offset += res.size
            case WIREGUARD_DONE:
                break
            case WIREGUARD_ERROR:
                Log.info("Warp: WIREGUARD_ERROR in wireguard_write: \(res.size)")
            default:
                Log.info("Warp: Unexpected return type: \(res.op)")
            }
        }

        if warp.wireguardWriteDatagrams.count > 0 {
            // Send
            self.sendDatagrams(datagrams: warp.wireguardWriteDatagrams)
            warp.wireguardWriteDatagrams.removeAll(keepingCapacity: true)
        }
    }

The decapsulate method (which we pass in setReadHandler) is this:

   /**
     The function the connection calls when datagrams arrive
     */
    // swiftlint:disable function_body_length
    internal func handleInboundWarpPackets(datagrams: [Data]?, err: Error?) {
        guard let datagrams = datagrams else {
            if err != nil && !self.reconnecting.value {
                Log.error(err, message: "Warp: error while reading datagrams")
                return self.handleUDPError(err: err!)
            }
            return
        }

        var offset = 0
        var flushQueue = false

        // This is the main loop
        for var wireguardDatagram in datagrams {
            // Check if we ran out of space and need to flush
            if offset + maxPacketSize >= bufferSize {
                flushReadStructs()
                offset = 0
            }

            let readBuff = warp.wireguardReadBuffer.advanced(by: offset)

            let res = wireguardDatagram.withUnsafeBytes { (ptr: UnsafeRawBufferPointer) -> wireguard_result in
                let dataPtr = ptr.baseAddress?.bindMemory(to: UInt8.self, capacity: maxPacketSize)
                return wireguard_read(self.warp.wireguardClient, dataPtr, UInt32(wireguardDatagram.count), readBuff, UInt32(maxPacketSize))
            }

            switch res.op {
            case WIREGUARD_DONE:
                break
            case WRITE_TO_NETWORK:
                self.setClientId(buffer: readBuff)
                warp.wireguardReadDatagrams.append(Data(bytesNoCopy: readBuff, count: Int(res.size), deallocator: .none))
                offset += res.size
                flushQueue = true
            case WRITE_TO_TUNNEL_IPV4, WRITE_TO_TUNNEL_IPV6:
                let family = (res.op == WRITE_TO_TUNNEL_IPV4) ? afInet : afInet6
                warp.wireguardReadPackets.append(Data(bytesNoCopy: readBuff, count: Int(res.size), deallocator: .none))
                warp.wireguardReadPacketProtocols.append(family)
                offset += res.size
            case WIREGUARD_ERROR:
                Log.info("Warp: WIREGUARD_ERROR in wireguard_read: \(res.size)")
            default:
                Log.info("Warp: Unexpected return type: \(res.op)")
            }
        }

        // If wireguard library wants to send prior pending packets to network, handle that case
        // and repeat calling wireguard_read() until WIREGUARD_DONE or error is returned.
        if flushQueue {
            if offset + maxPacketSize >= bufferSize {
                flushReadStructs()
                offset = 0
            }

            let readBuff = warp.wireguardReadBuffer.advanced(by: offset)
            flushLoop: while true {
                let res = wireguard_read(self.warp.wireguardClient, nil, 0, readBuff, UInt32(maxPacketSize))
                switch res.op {
                case WRITE_TO_NETWORK:
                    self.setClientId(buffer: readBuff)
                    warp.wireguardReadDatagrams.append(Data(bytesNoCopy: readBuff, count: Int(res.size), deallocator: .none))
                    offset += res.size
                case WIREGUARD_DONE:
                    break flushLoop
                case WIREGUARD_ERROR:
                    Log.info("Warp: WIREGUARD_ERROR in repeat wireguard_read: \(res.size)")
                    break flushLoop
                default:
                    Log.info("Warp: Unexpected return type: \(res.op)")
                    break flushLoop
                }
            }
        }

        flushReadStructs()
    }


    func flushReadStructs() {
        if warp.wireguardReadPackets.count > 0 {
            self.packetFlow.writePackets(warp.wireguardReadPackets, withProtocols: warp.wireguardReadPacketProtocols)
        }

        if warp.wireguardReadDatagrams.count > 0 {
            self.sendDatagrams(datagrams: warp.wireguardReadDatagrams)
        }

        warp.wireguardReadPackets.removeAll(keepingCapacity: true)
        warp.wireguardReadDatagrams.removeAll(keepingCapacity: true)
        warp.wireguardReadPacketProtocols.removeAll(keepingCapacity: true)
    }

For roaming, you usually get a UDP error, on which we immediately reconnect.
When you are on cellular and connect to wifi, we used to monitor the path with a callback, but it was not very reliable in iOS. Now instead we check the hasBetterPath property of the current UDP connection every 500ms.

[vkrasnov]

It is functional enough, after all we use it just for that.

Idiomatic means that for example on iOS if you must use NetworkExtensions and NEPacketTunnelProvider.

You also use the platform dependent routing and UDP connection to a server.

On packetFlow.readPackets you call wireguard_write from the ffi code, and send the encapsulated packet over the UDP connection.

On each packet you receive from UDP you call wireguard_read and depending on result either write back to UDP or use packetFlow.writePackets to inject the decapsulated packets.

In addition you need a timed function to call wireguard_tick.

[anjmao]

On iOS you will definitely need to use NetworkExtensions and NEPacketTunnelProvider as on Android VpnService but UDP connection could be managed from Rust side since you can pass created file descriptor to Rust.

@vkrasnov Could you also explain a bit about wireguard_tick fn and why it's needed. To me it looks that this should be internal implementation detail.

[vkrasnov]

Hi @anjmao
You could definitely manage the UDP connection from Rust, but iOS provides much more powerful (and faster) UDP connection API in the form of createUDPSession.

It has multiple benefits over managing your own BSD socket (which Rust provides only minimal abstraction over), including better roaming support, better battery life etc.

If you really wanted to use Rust for that you can also technically extract the filedescrtiptor of the tunnel created by NetworkExtensions and pass it to Rust too.

For WARP we use the idiomatic flow readPackets -> Batch encapsulate with boringtun -> writeMultipleDatagrams

And in the other direction we use the callback in setReadHandler to batch decapsulate multiple datagrams with boringtun followed by writePacketObjects.

[balboah]

is there a code example of this iOS idiomatic loop flow? do you also need to listen for network changes to re-handshake on roaming?

[balboah]

Thanks a lot for the code example. I've been playing a bit with this and got a successful tunnel up.

However I noticed some issues with that NWUDPSession.writeMultipleDatagrams.
First thing was that I also ended up with a timer since the observe KVO which docs recommend seems broken.

Then to conform with "Callers should wait until the completionHandler is executed before issuing another write." I'm using a DispatchSemaphore. But it seems like the completion handler isn't always called, causing a dead lock. Easy to reproduce by toggling flight mode while doing speed tests.
Did you experience this as well? A bit off topic from boringtun so I understand if you have other things to do :)