nengo/nengo-gui

Nengo interactive visualizer

Nengo GUI

Nengo GUI is an HTML5-based interactive visualizer for large-scale neural models created with Nengo. The GUI lets you see the structure of a Nengo model, plots spiking activity and decoded representations, and enables you to alter inputs in real time while the model is running.

Requirements

• Python (tested with Python 2.7 and Python 3.4+)
• Nengo (which requires NumPy)

Installation

The simplest way to install is with the standard Python installation system:

pip install nengo-gui

Running Nengo GUI

Nengo GUI is accessed through a web browser (Google Chrome, Firefox, Safari, etc.) To access the correct webpage, you must first start the Nengo GUI server. To do this, open a command line window and run:

nengo

If you specify a file, it will be loaded:

nengo myfile.py

Alternatively, you can start the GUI manually from within your code. To do so, add this to the bottom of your file that defines your Nengo model.

import nengo_gui
nengo_gui.GUI(__file__).start()

Basic usage

The graph of the Nengo network should appear. Rectangles are nodes, sets of 5 circles are ensembles, and rounded rectangles are networks.

Items can be dragged to move them and resized by dragging their edge or via the scroll wheel.

To start (or continue) the simulation, click the play button in the lower right. A spinning gear icon indicates the model is in the process of being built (or re-built after new graphs are added).

Clicking on an item will show a menu of options, depending on what you have clicked on. Here are some of the standard options for network items:

• value: show a graph of the decoded output value over time
• xy-value: show a state-space plot of two decoded values against each other
• spikes: show the spiking activity of the nengo.Ensemble
• slider: show sliders that let you adjust the value in a nengo.Node
• expand/collapse: reveal or hide the insides of a nengo.Network

Once you have graphs, you can also click on them to adjust their options. For example:

• set range: adjust the limits of the graph
• show label/hide label: whether to show the title at the top of the graph
• remove: get rid of the graph

The graphs record their data from previous time steps. You can show this previous data by dragging the transparent area in the time axis at the bottom (beside the play button).

Contributing

We welcome contributions to Nengo GUI through issues and pull requests! However, we require contributor assignment agreements before pull requests are merged. See the CONTRIBUTORS.rst and LICENSE.rst files for more information.

Developer installation

Developers should install Nengo GUI like so:

git clone https://github.com/nengo/nengo-gui
cd nengo-gui
python setup.py develop --user

Changes to the files in the nengo-gui directory will be reflected the next time the GUI is run or imported.

Running unit tests

Testing is done with the help of Selenium. Testing is currently only supported on Linux and Mac OS X.

To run the tests, make sure you have a recent version of Firefox. Mac users should ensure that Firefox is in the applications folder.

Additional dependencies are required for running unit tests. To install them, open a terminal and navigate to the nengo-gui folder. Execute the command

pip install --user -r requirements-test.txt

If you are using a virtual environment, you can omit the --user flag.

At this point selenium and pytest should be installed, so you are ready to run the tests.

To run the tests:

1. Open a terminal window and start the nengo server.
2. Open a second terminal window.
3. Navigate to the nengo-gui directory.
4. Run py.test.

The console should say some number of tests are found, and Firefox will launch and start doing things on its own. It may takes a few minutes to run all tests.

Writing new unit tests

To create tests, simply save a file named test_whatever_the_test_concerns.py in nengo_gui/tests See nengo_gui/tests/test_example.py for examples tests.

The following references may also be helpful.

• Selenium-Python documentation
• pytest documentation

nengo-gui's Issues

Dragging sliders need max/min bounds

When dragging past the bounds of a slider, the cursor position and slider get out of synch.

Make SimControl look pretty

The bar at the bottom should be cleaned up. Things that are needed:

• Move play/continue button to right side and make it look like a play/pause icon
• Show beginning and end times (overlayed on the time slider)
• Add reset button to left side
• Add configure button with other options (filter time, dt, kept_time, etc?)

Legend for plots

surpress elements on plots when plot is resized to small dimensions

Add a Taskbar to keep track of plots and sliders

ability to minimize sliders and plots and restore them from a taskbar

Unnecessary for loop

nengo_viz/static/viz_slider.js
line 22

(arg.n_sliders) seems to be always set to 1 resulting in the loop running once every time. Is there a case where args.n_sliders will be > 1, can this for loop be removed?

Delay on launch (loading message?)

On my machine when launching the visualizer there is a ~5 second delay before the graphs start updating.

Should we consider showing the user a message such as "Loading..." until elements start updating?

Add firing rate plot (aka sparkle-plot)

Scroll zooming and dragging window

I have been looking around for some libraries we can use for this and I came across zynga/Scroller

It is under MIT License and looks pretty good. http://zynga.github.io/scroller/

What does everyone think of this library? Did anyone have alternate ideas for this?

Prevent shown_time bar from extending beyond the bounds of time_slider div

Play/pause button should be larger, and on the right side (as in the previous java visualizer)

Change localhost to 127.0.0.1:8080/

There are compatibility issues with localhost on some machines.

Timer slider and input sliders, jump to click location

When clicking on a slider bar at a location where the slider is not currently located, the slider should jump to the click location.

For example: An up/down slider is currently positioned at the very top, user clicks on the middle of the slider bar, the slider jumps to the middle position.

Consideration: How to drag the entire slider object to different positions on the screen if clicks on it move the slider value.

Shortcut for jumping to an organized view of all the components

Potential:
-tile all components according to type (slider,plot)

Having a HUD option

So we were getting a sweet demo at the dev meeting yesterday, and one of the things not done yet was having the figures / plots move with the background, rather they stayed on screen as you scrolled through your network. And this made me think of a heads-up-display, and then thinking how awesome / useful it would be to have that kind of set up. Chris mentioned it being like an oscilloscope, where you're probing different parts of the network and seeing the results displayed in a single space, and I think that that's a good description of it.

It's pretty different from how we look at things now, but I think it would be a really natural way to analyze / interact with the network. I was thinking it would be something like you make your plot, and then do an options click to make it part of the HUD, and then it moves around with you and shows you signals from whatever you've just clicked on, but however.

One of the things this would require is some means of either dynamically probing things or like Eric mentioned just probing everything and only storing the data if you're examining it (which was suggested anyway for the issue of not having to rebuild the model if you want to add a plot).

Resizing

• plots should be resizable from all edges
  • also resize on scrollwheel
  • also resize on pinch-zoom
• zoom the whole thing by scrollwheel or pinch-zoom on background

When slider is at the top, blocks the name/label

Minimap for enitre visualizer preview

Idea for a library to use
http://www.toolitup.com/minimap.html

Plot Title

tabs to jump between different layouts?

An idea for switching between layouts

When window is resized smaller vertically, sim control disappears and not able to scroll down to control

Suggested fix: Make sim control stick to bottom of window viewport

Also, relevant issue: on firefox, sim control sticks to top of window viewport.

white border on guideline

Use of .length in Datastore - viz.js

We can improve efficiency a good deal by adding a counter for the length of the data and time array to the data store object.

Increment the variable for every addition in constant time and remove the need for linear time .length calls.

dropdown menus for config on components

Save as pdf

Make nice ways to save the layout to pdf/svg

zero out / set_val does not work for 2d sliders

only the most right slider gets zeroed out

zeroing out slider with a button in the middle of the guideline

Potential concern:
What if the slider is covering the button due to slider being near the middle, and the user wants to zero out.

For minimap

• When does the area of concern grow?
• When does the AoC shrink?
• Does the Area of Concern follow the user's panning?

Decide on target browser(s) and format all code accordingly

change VIZ.set_transform to be compatible with all browsers

Change shown time

• dragging on the edge of the shown time window
• scrollwheel on the shown time window
• pinch-zoom on the shown time window

Add a Semantic Pointer plot

Bad exiting behaviour

When I close the window on a mac, the server gets angry:

Traceback (most recent call last):
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 202, in handle_ws_request
Traceback (most recent call last):
Traceback (most recent call last):
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 202, in handle_ws_request
Traceback (most recent call last):
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 202, in handle_ws_request
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 202, in handle_ws_request
Traceback (most recent call last):
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 202, in handle_ws_request
getattr(self, command)(_[client] + args[1:], *db)
getattr(self, command)([client] + args[1:], *db)
getattr(self, command)([client] + args[1:], *db)
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/server.py", line 39, in ws_viz_component
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/server.py", line 39, in ws_viz_component
getattr(self, command)([client] + args[1:], *db)
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/server.py", line 39, in ws_viz_component
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/server.py", line 39, in ws_viz_component
msg = client.read()
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 461, in read
assert(0x1 == (0xF & data[0]))
msg = client.read()
msg = client.read()
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 461, in read
AssertionError
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 461, in read
getattr(self, command)([client] + args[1:], *_db)
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/server.py", line 39, in ws_viz_component
msg = client.read()
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 461, in read
assert(0x1 == (0xF & data[0]))
AssertionError
msg = client.read()
File "/Users/celiasmi/Documents/nengo/nengo_viz/nengo_viz/swi.py", line 461, in read
assert(0x1 == (0xF & data[0]))
assert(0x1 == (0xF & data[0]))
AssertionError
assert(0x1 == (0xF & data[0]))
AssertionError
AssertionError

Moving elements to top

Currently moving elements to top removes and re-adds the element making any other click events on the element unusable.

Add raster plot

set value / Zero out option for sliders in dropdown menu

Timing efficiency should be investigated

Explore:

1. Why take hit from running large vs small graphs
2. Source of slow down of visualizer (seems like 2x slower than pure sim run).

Figure out where any bottle necks might reside. Try to get as close as possible to regular simulation times.

Use em instead of px

When resizing the height of the slider component, resize slider ?

When the height of the slider component is changed, do we want the slider to be resized as well?

my thoughts

For:
-Seems odd to have the slider being the same size as the range it traverses

Against:
-The size of the slider seems ideal for touch. Resizing to smaller sizes may prove difficult for touch users.
-Forces consistent sizes across sliders

Make Value plots look pretty

• show start and end times on x-axis, rather than the current approach with a single unreadable number flying past
• show label on the title bar
• colors for different lines
• optional legend?
• ability to turn on and off different lines (without the bug in the nengo 1.4 where changing what lines are on changes the color mapping)

Transform function for all browsers

change VIZ.set_transform to be compatible with all browsers

When resizing sliders/graph components, they can become too small and unresizable

Potential fix: minimum element width/height so that user cannot resize the element to the point where it is so thin/flat that it cannot be resized to a larger width/height.

store slider height as a variable

in viz_slider.js

move zero out function to prototype,

This will help in generating other methods for zeroing out sliders besides middle clicks

Fix nengo_gui in Safari

In Safari, the text does not appear on the items, the scroll-wheel/two-finger zoom is ignored, and dragging in the middle of the item causes resizing rather than moving.

Note that the dragging problem is also what happens in Chrome if you use the touch screen to drag rather than a mouse, so that might be fixed by correctly supporting one-and-two-finger touch gestures.

on window resize, SimControl axis is not resized

Occasional mysterious border outline on slider guideline

see the top and top left of the guideline for this mystery border outline.
There should be no outline, (blends perfectly with background)
cannot reliably replicate this glitch.
Unsure if it may be simply a browser issue.

Save plot as image

Save layout

To save the layout, we'll do the same Python code trick as nengo_gui. We can take the current settings and generate the source code that would do that. So instead of

viz = nengo_viz.Viz(model)
viz.slider(stimulus_A)
viz.value(ens)
viz.start()

it would change its own source code to

viz = nengo_viz.Viz(model)
viz.slider(stimulus_A, x=30, y=19, width=300, height=400)
viz.value(ens, x=111, y=99, width=10, height=99)
viz.start()

Yes, this'll be kinda tricky, and I'm not sold on the syntax yet (I might like something close to the nengo.Config system). But that's the idea.

Make sliders look pretty

• Add a line showing where they can move
• Show the numerical value at the top
• Shortcut for zeroing sliders
• Move sliders automatically in accordance with their default value
• Jump time forward if you move the sliders (you can't change values in the past)
  • in Nengo 1.4 this was implemented by having the sliders be deactivated when you're looking at past data, but I think this time just jumping forward in time to the present would be clearer.

Ability to turn plot lines on/off