ayyaruq/zanarkand

Network capture library for realtime FFXIV Frame and FFXIV Message reading from a TCP/IP stream

zanarkand's Issues

Implement Generics

Requires Go 1.18 beta 1, but I dislike beta releases. Schedule looks like 1.18 is released in Feb 2022 tho, so we can get started and test at least. The Frame and Message decoders can all be ported to generics, along with Subscribers as they're basically fulfilling an interface but otherwise identical.

Note that this should use type constraints, as the syntax without constraints is kinda gross, and constraints make it easier to add more decoders in the future.

Usage Examples

Examples of ReadFrame and ReadMessage and maybe some stuff you can do with them. Printing Frames is probably fine, maybe some stuff for live reading vs PCAP files. Maybe Inventory tracking?

Oodle Support

As per the 6.1 or so changes to introduce Oodle compression. There's a few options here and I'm not sure what's best, leaning towards an interface for decompressors and allowing it to be user's choice. The following are what I had in mind:

• LLVM IR from open source Oodle implementations, calling LLIR functions directly
• Transparent eBPF decoding/encoding with kizer's helper
• CGO directly hitting Oodle static library (official enough)
• CGO directly hitting Oodle dynamic library (third party dumps of the Oodle DLL)
• Native Go implementations of Oodle (there's a few around, none great, but we only need enough support for FFXIV)

eBPF is particularly interesting, effectively allowing the Linux kernel network stack to decode packets on receive and re-encode them on send, so anything using the data has full access without any compression in the way. The downside is that it only runs on Linux as eBPF doesn't exist on other platforms unless you count BSD separately.

LLIR is likely the most simple to use and maintain even though it sounds really weird, the downside is that it loses all of the ASM optimisations the official Oodle static lib has, but those might be trivial and third party libs lack them anyway.

Sniffer and Subscriber Contexts

Add a context to sniffers and subscribers, so that cancelling a subscriber or sniffer directly can use a context provided by user or internally to manage shutdown timeouts and drainage. Might need a const somewhere to more optimally configure timeouts, since it's hardcoded right now. A configuration block to the sniffer init maybe?

Replace Reassembler Channel with a proper Buffer

Using a channel for reassembled data has proven less than optimal, specifically it's difficult to debug and is too strict on types and how we return data. Switching to a straight up data buffer and isolate ingress and egress seems like the best way to address this.

Automated Tests

Simple go-test for the decoder and sniffer, connected to CI.

TCP RTT in Frame metadata

To enable more accurate latency tracking than simple ICMP echo requests, add the TCP RTT for each packet to the encapsulating Frame. This could get a bit weird with retransmitted Frames but still useful.

Improved error handling

We currently just kinda yolo return fmt.Errorf() or the occasional errors.New(). A proper set of error types should be used with wrapping to allow users to appropriately respond to different errors. Packet reassembly errors currently can't even be accessed, so that should be addressed too.

Add an alternative to Subscribers to support callbacks

Subscribers currently expose the IPC streams for ingress and egress as channels. While this works well for low volume packets, usually when you know exactly what you want, it's probably worth having a callback style option for users that prefer it.

Relocate Frame Reassembly

Frames being decoded inside the reassembler has been bad for performance and stability. This should be moved out of the reassembler, which is also useful to give some way of accessing the frame timestamp in consuming IPCs (nfi how to best present this tho), could be nice to have since frames have a higher resolution timestamp than IPCs even if it's somewhat less reliable.

Improve Frame and Subscriber connection metadata

To allow users to more easily filter opcodes, improve the Frame and Subscriber interfaces. The main changes here would be as follows:

• Connection direction (ie, is it a Server or Client IPC)
• Wrap the connect type (ie, Zone/Chat/Lobby)

One option is to have Subscriber take callbacks for sent/received frames instead of changing the structure. Additionally, Frame could be implemented as a custom gopacket layer decoder, but I have no idea how that works with tcpassembly. We could also just wrap the FrameHeader straight up into the various Message types.

If we used callbacks like Machina, we could have a function per direction and just track the connection type field from the Frame. It would still be up to the user how to get data out of it so I'm not sure if this is actually useful though. As an example of Subscriber using callbacks, the sig could be updated like this:

func (g *GameEventSubscriber) Subscribe(
  s *Sniffer,
  recv func(c uint16, p []byte),
  send func(c uint16, p []byte)
)

GameEvent and Keepalive messages currently have subscribers, but not the other types, this would be the session and encryption initialisation. I'm a bit hesitant for the latter since I don't want to enable usage for botting, and there's no real value in session parsing.

Allow Subscribers to pre-filter

Add an optional second parameter to Subscribe methods that takes a function to filter the data before it's published in the channel. This is mostly useful for segment/opcode pre-filtering and should reduce load in the consumer.

Subscriber sync pools for readers

Readers should use a pool to reduce allocation overhead. Some large packet bursts can cause problems with the zlib decoder blocking the frame buffer. Pooled readers should be properly closed when the subscriber is shutdown.

Device options for packet source

Currently only pcap for files and interfaces is supported. The ability to select different capture sources should be added for flexibility and performance on different platforms.

This requires a device struct and a way to set it when a sniffer is initialised. Device options should cover as much as gopacket supports, currently as follows:

• AF_Packet
• PCAP
• PF_Ring
• winsock