gbeckers/darr

A Python library for numpy arrays that persist on disk in a format that is simple, self-documented and tool-independent, and maximizes universal readability.

Darr

Darr is a Python science library to work efficiently with potentially very large, disk-based Numpy arrays that are widely readable and self-documented. Every array has its own documentation that includes copy-paste ready code to read it in many popular data science languages, such as R, Julia, Scilab, IDL, Matlab, Maple, and Mathematica, or in Python/Numpy without Darr. Your numerical arrays can be read in other analysis environments with minimal effort and without any need for exporting/copying data.

In essence, Darr makes it trivially easy to share your numerical arrays and metadata with others or with yourself when working in different computing environments, and stores them in a future-proof way.

Universal readability of data is a pillar of good scientific practice. It is also generally a good idea for anyone who wants to save data for the longer term, who wants to flexibly move between analysis environments, or who wants to share data with others without spending much time on figuring out and/or explaining how the receiver can read it. Want to quickly try out an algorithm your colleague wrote in R or Matlab, but no idea how to read your 7-dimensional uint32 numpy array in those environments? A quick copy-paste of code from the documentation included with the array is all that is needed to read it (see example). No need to export anything. Want to share your array with non-Python colleagues? No looking up things, no need to make notes or to provide elaborate explanation. No dependence on complicated formats or specialized libraries.

More rationale for a tool-independent approach to numeric array storage is provided here.

Under the hood, Darr uses NumPy memory-mapped arrays, which is a widely established and trusted way of working with disk-based numerical data, and which makes Darr fully NumPy compatible. This enables efficient out-of-core read/write access to potentially very large arrays. In addition to automatic documentation, Darr adds other functionality to NumPy's memmap, such as easy the appending and truncating of data, support for ragged arrays, the ability to create arrays from iterators, and easy use of metadata. When you change the size of your array, its documentation is automatically kept up to date. Flat binary files and (JSON) text files are accompanied by a README text file that explains how the array and metadata are stored (see example arrays).

See this tutorial for a brief introduction, or the documentation for more info.

Darr is currently pre-1.0, and still undergoing development. It is open source and freely available under the New BSD License terms.

Features

• Data is stored purely based on flat binary and text files, maximizing universal readability.
• Automatic self-documention, including copy-paste ready code snippets for reading the array in a number of popular data analysis environments, such as Python (without Darr), R, Julia, Scilab, Octave/Matlab, GDL/IDL, and Mathematica (see example array).
• Disk-persistent array data is directly accessible through NumPy indexing and may be larger than RAM
• Easy and efficient appending of data (see example).
• Supports ragged arrays.
• Easy use of metadata, stored in a widely readable separate JSON text file (see example).
• Many numeric types are supported: (u)int8-(u)int64, float16-float64, complex64, complex128.
• Integrates easily with the Dask library for out-of-core computation on very large arrays.
• Minimal dependencies, only NumPy.

Limitations

• No structured (record) arrays supported yet, just ndarrays
• No string data, just numeric.
• No compression, although compression for archiving purposes is supported.
• Uses multiple files per array, as binary data is separated from text documentation and metadata. This can be a disadvantage in terms of storage space if you have very many very small arrays.

Installation

Darr officially depends on Python 3.9 or higher. Older versions may work (probably >= 3.6) but are not tested.

Install Darr from PyPI:

$ pip install darr

Or, install Darr via conda:

$ conda install -c conda-forge darr

To install the latest development version, use pip with the latest GitHub master:

$ pip install git+https://github.com/gbeckers/darr@master

Documentation

See the documentation for more information.

Contributing

Any help / suggestions / ideas / contributions are welcome and very much appreciated. For any comment, question, or error, please open an issue or propose a pull request.

Other interesting projects

If Darr is not exactly what you are looking for, have a look at these projects:

• asdf
• exdir
• h5py
• pyfbf
• pytables
• zarr

Darr is BSD licensed (BSD 3-Clause License). (c) 2017-2023, Gabriël Beckers

darr's Issues

implement reading code for types that are not directly supported

Not all languages can directly read all numeric types. Sometimes this is quite inconvenient, for example when working with float16 data that Matlab cannot read as such, or complex types. Matlab has these types, but cannot read them. Darr should produce read code for those cases that circumvents this problem .

Add code for more languages in README RaggedArrays

Now only Python, R and Matlab are included. But for plain Arrays we have also Mathematica, GDL, Julia (old and new) and Maple.

implement memmap code where possible

Investigate and implement if possible read code in other languages (at least R and hopefully Matlab and Julia) based on a memory mapped file.

Add pure Python read code

For 1-D arrays you do not necessarily need numpy, but you can also use the array module from the standard library:

https://docs.python.org/3/library/array.html

improve README of RaggedArrays

README now is unnecessarily complex if the subarrays are 1-dimensions. Also, right at the topic specific information of the shape of the subarrays may be included to make things less abstract.

write specification

Although it would be short, it would be good to have a description in the documentation what the specification of a Darr Array (and RaggedArray) is.

Implement setting values RaggedArray

Enable the setting of values in RaggedArrays, like a[5,:3] = 2.0 . Now only creating, appending and reading ragged arrays is supported.