maier/wirelatency

Go tools for analyzing wire protocols to report fine-grained telemetry

Circonus WireLatency

These are Go-based tools to sniff the wire and reconstruct various protocol sessions to extract various telemetry (namely latency) from transactions within those sessions.

For example, it can be used to track the latency of HTTP API endpoints.

All telemetry data is pushed up to Circonus.

Tooling

protocol_observer --apitoken <circonus_api_token> -wire <description>

The -wire flag can be specified multiple times to specify multiple "wirings." This is useful if you have multiple ports in use for a service. It is recommended to keep separate services monitored by separate protocol_observer instances.

Cassandra

protocol_observer -apitoken <token> -wire cassandra_cql

Will listen on port 9042 for Cassandra CQL framing and disset sessions. Each request/response will be tracked for latency. Queries will generically be tracked as "QUERY", however each prepared query will get its own metric named for the CQL. Latency is tracked on each operation as well as request and response sizes in bytes.

HTTP

protocol_observer -apitoken <token> -wire http:8092:routes.json

This will listen for HTTP traffic on port 8092 using the file routes.json as a mapping of URLs to "names". This allows requests to an API service to be correctly categorized by endpoint. See the same routes.json included.

Each request is recorded and the request and response payload sizes are recorded along with the latency to the first byte of the response and the latency for the entire response.

Metric names are formed as <method>`<endpoint>`<status> where status is the HTTP status codes by the hundreds: 1xx, 2xx, 3xx, 4xx, and 5xx.

Postgres

protocol_observer -apitoken <token> -wire postgres:5432:queries.json

This will listen for Postgres traffic on port 5432. If a config is omitted the default config will be used. If the port is omitted, 5432 will be used.

The protocol observer here distinguishes between regular queries and the execution of prepared statements. All statements are tracked and the latency, request bytes, response bytes and number of tuples effected is recorded. The format of metrics looks like <type>`<attribute> and <type>`SELECT`<attribute> where the "SELECT" is taken from the postgres execute command complete packet. "SELECT" could be "DELETE" or "UPDATE" or anything else Postgres elects to respond with. The <type> will be one of Query or Execute. The <attribute> will be one of "latency", "request_bytes", "response_bytes", or "response_rows."

Additionally, pursuant to your configuration (if supplied) you can record metrics for a query. Prepared and Adhoc queries are handled separately as most people will only want to record latency information on a limited keyspace and thus only leverage the configuration to log prepared queries.

The Query field is a regular expression matching against the query being executed. The Name field helps instruct how the query should be presented. If name is an empty string, no query-specific recording is done. If it is the string "RAW" then the whole query is jammed in the metric name. If it is the string "SHA256" then a hex-encoded sha256 of the query is used. Otherwise the literal string is used to build the metric name.

Kafka

protocol_observer -apitoken <token> -wire kafka:9093

This will listen for Kafka traffic on the specified port (9092 by default).

Message and protocol command latency is recorded as well as message and frame sizes. All production "Produce" and consumption "Fetch" are tracked in _aggregate and by topic. Messages in production and consumption are expanded (including gzip and snappy) to analyze for message timestamps (for latency) and uncompressed payload sizes.

Building

go1.13 is required to build. Most importantly, clone the repository outside of GOPATH.

git clone https://github.com/circonus-labs/wirelatency.git
cd wirelatency/protocol_observer
go build

---

Unless otherwise noted, the source files are distributed under the BSD-style license found in the LICENSE file.