sirmarcel/son

sequential object notation

son | sequential object notation

son is a data format that builds on JSON and adds one feature inspired by YAML: concatenation of objects with ---. Optionally, the delimiter === can be used once per son file to delimit metadata.

---

Motivation

Why son?

While JSON is perfect for storing structured data, it is inherently impossible to add new portions of data to a file without reading it first. YAML files on the other hand are self extensible by the --- delimiter, but the flexibility YAML offers makes the files inefficient to parse. They are thus unsuited to store significant amounts of data.

son fills the gap by allowing JSON objects to be concatenated with ---. It thus combines the speed and efficiency of JSON with the sequential extensibility of YAML, see example. It further adds to discern metadata from actual data by using ===.

son does not allow to overwrite data. In order to avoid accidental data loss, metada can only be written to fresh files, whereas data can only be appended to files.

Who needs this?

son originated from the need to store computational data that is produced portion by portion on a computer. The requirements were:

• Possible to be read by a human,
• possible to store arbitrary data structures including metadata,
• easy to write and parse by a computer,
• efficient to parse to allow files of up to GB size (takes forever to parse with YAML),
• sequential and incorruptible,
• resilient to data loss.

son is targeted at users who would like to store their data in a format meeting these requirements.

---

Installation

son can be installed from pypi via

pip install son

---

Example

This is a valid son string:

{
  "purpose": "store biography data",
  "version": 0.1
}
===
{
  "first name": "Hildegard",
  "second name": "Kneef",
  "age": 93
}
---
{
  "first name": "Wiglaf",
  "second name": "Droste",
  "age": 57
}
---

It will be parsed into the metadata object, and a list containing the data objects with

>>> import son
>>> metadata, data = son.load('test.son')
>>> print(metadata)
{'purpose': 'store biography data', 'version': 0.1}
>>> print(data)
[{'first name': 'Hildegard', 'second name': 'Kneef', 'age': 93}, {'first name': 'Wiglaf', 'second name': 'Droste', 'age': 57}]

API

son exposes three functions: dump, load, and open.

• dump(obj, file, is_metadata=False, encoding="utf-8", **kwargs) writes a string representation of obj to file. If the file does not exist yet, is_metadata can be True, and obj will be marked as metadata with the === delimiter.
• load(file, verbose=False, encoding="utf-8", **kwargs) loads a son file, returning (metadata, data) where data is a list of de-serialized entries in file.
• open(file, verbose=False, encoding="utf-8", **kwargs) does the same, but returns an iterator generator in place of data. Since this avoids reading the file all at once, this function should be preferred for performance-intensive applications.

The kwargs will be passed to the de-/serialization routines, which can in turn be specified with loader and dumper keyword arguments. They must be callables that turn strings into objects, and vice-versa. By default, we use json.loads and json.dumps.

The public-facing interface can be found in interface.py, the de-/serialization logic in serialize.py and the low-level write/read routines in stream.py.

---

Changelog

v0.4.0: switch to a backend based on generators, allowing large files to be parsed on-the-fly. general cleanup, remove progressbar. contributed by @sirmarcel

v0.3.3: Add documentation via mkdocs and mkdocs-material

v0.3.2: fix for interactively working in ipython console

v0.3.1: inform before file is read, makes more sense when that takes some time

v0.3.0: support for reading compressed .bz2 and .gz files

v0.2.5: progressbar is only shown when a terminal is attached (.isatty())

v0.2.4: progressbar without external dependency

v0.2.3: progress.bar prints to stdout instead of stderr

v0.2.2: optionally be verbose and show progressbar with progress package (optional dependency, install with pip install son[progress])