clandininlab/flystim

Visual stimulus generator for neuroscience experiments

Flystim is now deprecated and unsupported

We recommend switching to stimpack, which includes and expands flystim functionality.

flystim

flystim is a software package for generating visual stimuli for neuroscience experiments. It was originally designed for fly labs, but can be used for any subject/viewer. The stimuli are perspective-corrected and can be displayed across multiple screens. Sample code, illustrating various use cases, is included in the examples directory.

Prerequisites

flystim only supports Python3, so in the commands below, the pip and python commands should refer to a Python3 install. You can either install Python3 directly or through a package manager like Conda.

On Linux, you'll also need to install a few packages via apt-get:

> sudo apt-get install build-essential libusb-1.0.0-dev libudev-dev

Installation

1. Open a terminal, and note the current directory, since the pip commands below will clone some code from GitHub and place it in a subdirectory called src. If you prefer to place the cloned code in a different directory, you can specify that by providing the --src flag to pip.
2. Clone and install flyrpc if you haven't already:

> git clone https://github.com/ClandininLab/flyrpc.git
> cd flyrpc
> pip install -e .
> cd ..

3. Clone and install flystim:

> git clone https://github.com/ClandininLab/flystim.git
> cd flystim
> pip install -e .
> cd ..

If you get a permissions error when running the pip command, you can try adding the --user flag. This will cause pip to install packages in your user directory rather than to a system-wide location.

Flystim depends on PyQt5. We have seen performance issues running flystim on Linux machines depending on the specific version of PyQt5 installed. Best performance is found using PyQt5==5.11.2. If you have trouble installing this specific version, you can try:

pip install PyQt5==5.11.2 --no-deps

For this and other reasons, we recommend upgrading to stimpack, which uses PyQt6 and is much more stable.

Running the Example Code

In a terminal tab, navigate to the examples directory and run one of the sample programs, such as show_all.py.

> cd flystim/examples
> python show_all.py

Each example can be exited at any time by pressing Ctrl+C.

Coordinate system

The coordinate system convention in flystim is defined as follows:

• Yaw = rotation around the Z axis (theta)
• Pitch = rotation around the X axis (phi)
• Roll = rotation around the Y axis

A fly heading of (yaw=0, pitch=0, roll=0) corresponds to the fly looking down the +Y axis, the +X axis lies to the fly's right side, and +Z lies above the fly's head

Screen objects and defining subscreen geometry

A Screen object specifies a display device and a list of SubScreens, each SubScreen is defined by:

1. Physical coordinates ( (x, y, z), in meters) that specify the screen geometry in 3D space

2. Normalized Device Coordinates ( (x, y), [-1, +1]) ) that specify a viewport for that SubScreen. This controls where on the display device the image for this SubScreen will appear.

flystim's Issues

Adopt standard camera coordinate conventions

By convention, the camera looks down the -z axis.

Currently flystim uses the convention that the camera looks down the +x axis.

A not-comprehensive list of changes that need to happen:

• update flystim.screen coordinates
• add support for a z component to fly trajectory
• update coordinate conventions for all stimuli

define supported python versions

What's the lowest version of Python we need to support (as currently written)? What's holding us back?

We should specify this as the python_requires kwarg in setup.py.

Would nice to be able to use format strings.

Fix Screen serialization

Screen.serialize does not serialize the azimuth (previously rotation) and offset attributes which are required for reconstruction.

Curiously, those same attributes where not even stored as attributes until recently.

I believe the previous strategy was to represent the same information via tri-list. To me it feels more natural to serialize the more fundamental parameters of azimuth and offset and make tri-list a property

Performance improvements

Performance improvements are needed for layered, multi-component stims. Issue can cause frame drops and loss of timing accuracy for complex stimuli.

Performance issues seem to be especially obvious when calling multiple, distinct shaders. Maybe share a shader for non-textured objects and pass all vertices as concatenated list - e.g. add_stim method or similar to add stimuli to a shared shader that is only spun up once. Harder to see how to get around this with multiple textured stims

Simplify package structure

All __init__.py files should be empty, follow standard import convention.

For instance, the objects in flystim.shapes are currently accessible at flystim. Which is confusing and unnecessary.

Textured cylinder height correction

Apply a correction to the height of textures painted onto textured cylinder stimuli to ensure that features subtend equivalent degrees of elevation regardless of how far off the equator they appear.

See, e.g RandomGrid in stimuli, and other stims that can set cylindrical_height_correction to True. Applied in base.get_fragment_shader(), the current solution does not appear to be correct

Add aperture/masks to stimuli

Add the ability to apply an aperture or mask to a stimulus to restrict it to a specified region in the visual field. Useful aperture/mask shapes would be circular patch and rectangular patch

refactor all uses of built in `id`

There are several places where the built in id is overloaded. For instance, screen's id attribute.

These should all be renamed.