open-ephys/liboni

API for controlling ONI-compliant hardware

liboni is the Open Ephys ONI-compliant API implementation for use with ONI-compliant hardware. It is focused on performance in terms of bandwidth and closed-loop reaction times and includes means for balancing these characteristics on-the-fly. When used in combination with ONIX Hardware, it can be used to acquire from:

• Tetrode headstages
• Miniscopes
• Neuropixels
• Etc.

and provide sub-millisecond round-trip communication from brain, through host PC's main memory, and back again. This repository contains the following folders:

• api liboni API and language bindings. MIT-licensed.
• drivers device drivers used by the API at runtime. License depends on driver.

liboni's Issues

Some asserts on oni_read_frame should be replaced by normal sanity checks

There are a couple of asserts on oni_read_frame() that sanity-check the header values before accessing memory. Specifically the ones that check data_sz to be >0 and <= max_frame_size. The issue with asserts is that they only work on debug mode, with release mode removing them.

While this was the original intention, as normal operation should always meet these sanity checks, so they are there for development purposes, communication or buffering issues could cause malformed frames which return invalid values for these fields which end causing memory access problems in subsequent memcpy calls. Having these asserts be normal checks could prevent crashes and notify users of issues on the communication backend that could trigger actions to help debug the issue.

Store persistent state for devices in onidriver_test

Making changes to the Enable register for test devices has no effect as they only update during a reset. Resetting in this context means restarting the whole program, which overwrites any changes made to the registers anyway.

It would be good to store a persistent state for test devices that can be loaded by oni-repl and applied, mimicking real world hardware via a virtual state. For Linux, this configuration could be placed in /tmp, but I don't know the best place to store it for Windows installations.

Why am I doing this twice?

liboni/api/liboni/oni.c

Line 939 in 43668d1

temp = realloc(ctx->dev_hash_table,

Separate ONIX-specific string functions from main libONI library

Right now, the onix_device_str() and onix_hub_str coded in onix.c/h are embedded into the main libONI library. This hinders future developments as any new device created requires a build of libONI, which should be general-purpose.

We need to think of a better way to integrate those strings into client applications which is easy for the client developers but extensible.

Suggestion: Batch register access

I propose the addition of two new functions to the API: int oni_write_reg_batch(const oni_ctx ctx, oni_dev_idx_t dev_idx, oni_reg_addr_t startaddr, unsigned int num, oni_reg_val_t* valuearray) and int oni_read_reg_batch(const oni_ctx ctx, oni_dev_idx_t dev_idx, oni_reg_addr_t startaddr, unsigned int num, oni_reg_val_t* valuearray).

These would read/write num registers starting from startaddr. They would not be just looped wrappers for the individual calls, but the onidrivers would need to implement them.

There are two reasons for this:
1- it's not uncommon for some devices to have contiguous addressed spaces dedicated to a single function. Many devices even include the possibility of performing batch transfers over their communication interface.
2- More importantly, register access usually comes with a communication overhead per access. Depending on the onidriver, it might be possible to greatly speed up access while joining multiple requests. If the onidriver didn't have the capability for batch transports, it can always loop individual calls internally.

Remove out of context historical files

This repo is nearly 250 MBytes for what amounts to a few source files. This is because it was originally part of the ONIX mega repo and was extracted, preserving commit history that is largely irrelevant. This is a reminder to address this without destroying the history of relevant files somehow

Proposed context state improvements

ONI contexts should include an INVALID state in run_state, checked as usual, set when a call to oni_init_ctx fails. This will force the destruction of any invalid context instead of enabling the possibility of calling init twice, which could result in leaks.

Adittionally, calling oni_set_opt for ONI_OPT_BLOCKREADSIZE or ONI_OPT_BLOCKWRITESIZE when run_state == RUNNING should result in an error.

clroni should have access to the error code enum on onidefs.h

Currently, this enum is not reproduced in clroni

liboni/api/liboni/onidefs.h

Line 27 in e1b0916

enum {

with the idea being that the ONIException.cs class providing a higher-level representation of these errors, obviating low-level names of error codes. However, if we want to e.g. efficiently propagate or eat exceptions depending on their type, we end up manually reproducing the the error numbers in code that uses clroni, which is not good.

Unnecessary enforcement of 32-boundaries in oni_read_frame()

In this line we round our frame data reads up to the next 32-bit boundary.

liboni/api/liboni/oni.c

Line 676 in 132a12d

rsize += rsize % sizeof(oni_fifo_dat_t);

AFAIK, this was implemented way back in the day when Xillybus was being called directly in the library, with a 32-bit wide read FIFO used to stream data, and therefore anything other than 32-bit reads could corrupt the stream. However, now we are doing block reads which can be set to any byte size so long as they are big enough to store the largest frame produced in the device table. Is there still a reason we are enforcing this? If the hardware does not produce 32-bit aligned data, this rounding operation will result in a hard crash, but this seems totally unnecessary. In fact, I discovered this limitation because I'm modifying the onidriver_test.c to remove some of its limitations. In doing so I forgot to make sure it produced 32-bit aligned data. Rather than forcing this, I removed this step and now return rsize and everything worked fine.

Minimum frame size is not the minimum BLOCKREADSIZE

The frame_size result of

oni_get_opt(ctx, ONI_OPT_MAXREADFRAMESIZE, &frame_size, &frame_size_sz);

cannot be used to set the minimal ONI_OPT_BLOCKREADSIZE via

    oni_size_t block_read_size = frame_size;
    size_t block_size_sz = sizeof(block_read_size);
    rc = oni_set_opt(ctx,
                    ONI_OPT_BLOCKREADSIZE,
                    &block_read_size,
                    block_size_sz);

due to some required overhead. This is not a bug, but something that needs to a least be documented and perhaps requires a utility function to get the correct minimal ONI_OPT_BLOCKREADSIZE.

Maybe separate the riffa driver from clroni

Right now, the clroni nuget package contains both the main libONI library and the onix-specific riffa driver.

Maybe it should be best to have them in separate packages, to ease the introduction of future different drivers, which could then be distributed as individual packages.

Adittionaly, maybe the riffa driver should be renamed as onix-riffa, to make clear that it is part of the ONIX ecosystem.

Discuss how to implement continuous integration with liboni

As Open Ephys is moving towards utilizing CI for different repos (such as the Sockets repo, and the Miniscope docs), we will need to make a decision about how this will look for liboni. Here we can start a discussion and start thinking about which questions are most important.

Also important to discuss how to update the existing libraries that point to liboni and clroni to minimize disruptions.

We probably need to separate liboni, cpponi, and clroni into different repos. clroni is used as a NuGet package, so CI would be useful there, but does liboni need to be packaged as a NuGet package as well? How should clroni link to liboni as a library? Submodules? DLLs?

The different testing programs (oni-repl and clroni-repl) might need to go into separate repos as well. While these programs do not need NuGet packages, it would be helpful to use CI to compile releases (i.e., pre-build executables) after updates.