deciduosity/ftdc

tools for collecting, compressing, and using a space-efficient columnar data format based on a document model (bson) and designed for timeseries data.

ftdc -- Golang FTDC Parsing and Generating Library

Overview

FTDC, originally short for full time diagnostic data capture, is MongoDB's internal diagnostic data collection facility. It encodes data in a space-efficient format, which allows MongoDB to record diagnostic information every second, and store weeks of data with only a few hundred megabytes of storage.

The FTDC data format is based on BSON, and provides ways to store series of documents with the same schema/structure in a compressed columnar format.

This library provides a fully-featured and easy to use toolkit for interacting data stored in this format in Go programs. The library itself originated as a project by 2016 Summer interns at MongoDB but has diverged substantially since then, and adds features for generating data in this format.

Use

All documentation is in the godoc.

This library is available for use under the terms of Apache License.

Features

This library supports parsing of the FTDC data format and several ways of iterating these results. Additionally, it provides the ability to create FTDC payloads, and is the only extant (?) tool for generating FTDC data outside of the MongoDB code base.

The library includes tools for generating FTDC payloads and document streams as well as iterators and tools for accessing data from FTDC files. All functionality is part of the ftdc package, and the API is fully documented.

The events and metrics sub-packages provide higher level functionality for collecting data from performance tests (events), and more generalized system metrics collection (metrics).

Development

This project emerged from work at MongoDB to support this fork drops support older versions of Golang, thereby adding support for modules.

Pull requests are welcome. Feel free to create issues with enhancements or bugs.