databento/dbn

Databento Binary Encoding (DBN) - Fast message encoding and storage format for market data

dbn

Databento Binary Encoding (DBN) is an extremely fast message encoding and storage format for normalized market data. The DBN specification includes a simple, self-describing metadata header and a fixed set of struct definitions, which enforce a standardized way to normalize market data.

All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files.

This repository contains both libraries and a CLI tool for working with DBN files and streams. Python bindings for dbn are provided in the databento_dbn package.

For more details, read our introduction to DBN.

Features

• Performant binary encoding and decoding
• Highly compressible with Zstandard
• Extendable fixed-width schemas

Usage

See the respective READMEs for usage details:

• dbn: Rust library crate
• dbn-cli: CLI crate providing a dbn binary
• databento-dbn: Python package

License

Distributed under the Apache 2.0 License.

dbn's Issues

sync encoder does not encode using the symbol_cstr_len for DBN v2 (?)

Bug Report

Expected Behavior

metadata encoder to use the metadata.symbol_cstr_len as length for the fixed length strings.

Actual Behavior

the metadata encoder picks a fixed length for the strings based on version, and ignores the symbol_cstr_len for this purpose, it does serialize symbol_cstr_len tho. to me it looks like if the user sets symbol_cstr_len to something which would be a optimal/good size for their strings, then the decoding would fail, because the decoder actually reads symbol_cstr_len and assumes that the strings are of that size.

Steps to Reproduce the Problem

1. look here https://github.com/databento/dbn/blob/main/rust/dbn/src/encode/dbn/sync.rs#L255 to see that the size is fixed even for V2
2. for decoding, one should note that the string size is first decoded, and then used to the code the strings https://github.com/databento/dbn/blob/main/rust/dbn/src/decode/dbn/sync.rs#L475

Specifications

• dbn version: 2