spacetelescope/jdaviz

JWST astronomical data analysis tools in the Jupyter platform

jdaviz is a package of astronomical data analysis visualization tools based on the Jupyter platform. It is one tool that is a part of STScI's larger Data Analysis Tools Ecosystem. These GUI-based tools link data visualization and interactive analysis. They are designed to work within a Jupyter notebook cell, as a standalone desktop application, or as embedded windows within a website -- all with nearly-identical user interfaces. jdaviz is under active development, and users who encounter bugs in existing features are encouraged to open issues in this repository.

jdaviz provides data viewers and analysis plugins that can be flexibly combined as desired to create interactive applications that fit your workflow. Three named preset configurations for common use cases are provided. Specviz is a tool for visualization and quick-look analysis of 1D astronomical spectra. Mosviz is a visualization tool for many astronomical spectra, typically the output of a multi-object spectrograph (e.g., JWST NIRSpec), and includes viewers for 1D and 2D spectra as well as contextual information like on-sky views of the spectrograph slit. Cubeviz provides a view of spectroscopic data cubes (like those to be produced by JWST MIRI), along with 1D spectra extracted from the cube. Imviz provides visualization and quick-look analysis for 2D astronomical images.

This tool is designed with instrument modes from the James Webb Space Telescope (JWST) in mind, but the tool should be flexible enough to read in data from many astronomical telescopes. The documentation provides a complete table of all supported modes.

Installing

You may want to consider installing jdaviz in a new virtual or conda environment to avoid version conflicts with other packages you may have installed, for example:

conda create -n jdaviz-env python=3.12
conda activate jdaviz-env

Installing the released version can be done using pip:

pip install jdaviz --upgrade

For details on installing and using Jdaviz, see the Jdaviz Installation.

Quick Start

Once installed, jdaviz can be run either as a standalone web application or in a Jupyter notebook.

As a Web Application

jdaviz provides a command-line tool to start the web application. To see the syntax and usage, from a terminal, type:

jdaviz --help
jdaviz --layout=specviz /path/to/data/spectral_file

For more information on the command line interfaces for each tool, see the Jdaviz docs.

In a Jupyter Notebook

The power of jdaviz is that it can integrated into your Jupyter notebook workflow:

from jdaviz import Specviz

specviz = Specviz()
specviz.show()

To learn more about the various jdaviz application configurations and loading data, see the Specviz, Cubeviz, Mosviz, Imviz, or Specviz2D tools.

jdaviz also provides a directory of sample notebooks to test the application, located in the notebooks sub-directory of the git repository. CubevizExample.ipynb is provided as an example that loads a JWST data cube with the Cubeviz configuration. To run the provided example, start the Jupyter kernel with the notebook path:

jupyter notebook /path/to/jdaviz/notebooks/CubevizExample.ipynb

Help

If you uncover any issues or bugs, you can open a GitHub issue if they are not already reported. For faster responses, however, we encourage you to submit a JWST Help Desk Ticket.

Recordings and instructional notebooks from live Jdaviz tutorials (as part of the JWebbinars series) can be found at the JWebbinar website under the "Materials and Videos" expandable section. Scroll down to the bottom of that section to find materials from the most recent session (JWebbinar 24, March 2023).

License & Attribution

This project is Copyright (c) JDADF Developers and licensed under the terms of the BSD 3-Clause license. This package is based upon the Astropy package template which is licensed under the BSD 3-clause licence. See the licenses folder for more information.

Cite jdaviz via our Zenodo record: https://doi.org/10.5281/zenodo.5513927.

Contributing

We love contributions! jdaviz is open source, built on open source, and we'd love to have you hang out in our community.

Imposter syndrome disclaimer: We want your help. No, really.

There may be a little voice inside your head that is telling you that you're not ready to be an open source contributor; that your skills aren't nearly good enough to contribute. What could you possibly offer a project like this one?

We assure you - the little voice in your head is wrong. If you can write code at all, you can contribute code to open source. Contributing to open source projects is a fantastic way to advance one's coding skills. Writing perfect code isn't the measure of a good developer (that would disqualify all of us!); it's trying to create something, making mistakes, and learning from those mistakes. That's how we all improve, and we are happy to help others learn.

Being an open source contributor doesn't just mean writing code, either. You can help out by writing documentation, tests, or even giving feedback about the project (and yes - that includes giving feedback about the contribution process). Some of these contributions may be the most valuable to the project as a whole, because you're coming to the project with fresh eyes, so you can see the errors and assumptions that seasoned contributors have glossed over.

Note: This disclaimer was originally written by Adrienne Lowe for a PyCon talk, and was adapted by jdaviz based on its use in the README file for the MetPy project.

jdaviz's Issues

jdaviz does not display content

The latest pull of jdaviz repo for me does not render content when I try to test it. Are there other dependencies that I need to install/update that are not yet documented? I created a fresh conda environment and installed jdaviz

conda create -n testviz python=3.7 ipython
conda activate testviz
cd /git/repo/jdaviz
pip install -e .

When I run jdaviz --layout=cubeviz manga-7495-12704-LOGCUBE.fits or load the notebooks/Example.ipynb in jupyter, I get empty content.

The console throws a bunch of errors regarding consle is not defined coming from goldenlayout.js

vue.min.js:6 ReferenceError: consle is not defined
    at stateChanged (eval at Ya (vue.min.js:6), <anonymous>:3:1160)
    at He (vue.min.js:6)
    at a.n (vue.min.js:6)
    at He (vue.min.js:6)
    at a.e.$emit (vue.min.js:6)
    at Object.<anonymous> (vue-golden-layout.js:1335)
    at s.items.Root.emit (goldenlayout.js:255)
    at s.items.RowOrColumn._propagateEvent (goldenlayout.js:3643)
    at s.items.RowOrColumn.emit (goldenlayout.js:265)
    at s.items.RowOrColumn._propagateEvent (goldenlayout.js:3643)

Change to showing plugins in sidebar instead of modal dialog

The current plugins pop up as modal dialogs. Following some user testing and input from @Jenneh the thinking is that a sidebar with fold-out panels for plugins is the preferred way to go. So this issue is to implement that.

Some specific constraints:

• The plugins should be "acoordian-able" so that the user only sees those they have opened up
• The plugin pane should be hide-able
• bonus/could be in later PR: the plugin pane should be tear-off-able, or at least movable
• bonus/could be in later PR: the plugins in the pane should be re-orderable

One other bit of user feedback: having lots of panes that organize the plugins via some complicated scheme is an anti-feature, so we should avoid that.

unclear how to turn off various toolbar features

It's a bit unclear how to turn off visibility of different components and what each component / tool-set contains. Setting all items under visible to false did not change the actual app display. For example, the toolbar remained visible when reloaded in the notebook. To turn off the display of at least the import data button, I needed to comment out the g-data-tools altogether from the toolbar, but I wasn't sure what else I was removing by doing so.

settings:
  configuration: mastviz
  visible:
    menu_bar: false
    toolbar: false
    tray: false
    tab_headers: false
toolbar:
  # - g-data-tools
  - g-subset-tools
  - g-gaussian-smooth

Can I turn off features programmatically? I tried the following but it didn't work.

app = Application(configuration='mastviz', tray=False, toolbar=False)

What does Glue do with WCS and reprojection in Jupyter notebook?

Duplicate of glue-viz/glue-jupyter#112 especially created for JIRA.

p.s. @astrofrog is doing the real work. I am just being assigned in name. 😬

Remove astropy-helpers submodule

APE 17 is accepted. Look out for a "conversion guide" that Tom Robitaille will send out to astropy dev mailing list in the near future.

CLI interface gives complicated traceback instead of simple error if invalid file given

If I do jdaviz --layout=cubeviz /not/actually/a/file, the result is the traceback I've copied below. That's not ideal because the poor user who doesn't regularly look at tracebacks will see that and think it's an error in the software instead of an invalid path. Instead it would be better if they see a friendly message at the command line and maybe something in the UI itself that tells them the path isn't loadable.

Note this may require an update to glue to get a more useful error out - e.g. it probably should be an IOError instead of just an Exception, but that's not mandatory for addressing this hopefully.

from jdaviz.app import Application
from glue.core import Data
app = Application(configuration='cubeviz')
data = app.load_data('/home/erik/none')
app
------------------

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/src/glue/glue/core/application_base.py in wrapper(*args, **kwargs)
     22             try:
---> 23                 return func(*args, **kwargs)
     24             except Exception as e:

~/src/glue/glue/core/application_base.py in load_data(self, paths, auto_merge, *
*kwargs)
    196         for path in paths:
--> 197             result = load_data(path)
    198             if isinstance(result, BaseData):

~/src/glue/glue/core/data_factories/helpers.py in load_data(path, factory, **kwa
rgs)
    261         if factory is None:
--> 262             raise KeyError("Don't know how to open file: %s" % path)
    263

KeyError: "Don't know how to open file: /home/erik/none"

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-1-47addfb53162> in <module>
      2 from glue.core import Data
      3 app = Application(configuration='cubeviz')
----> 4 data = app.load_data('/home/erik/none')
      5 app

~/src/jdaviz/jdaviz/app.py in load_data(self, path)
    107
    108     def load_data(self, path):
--> 109         self._application_handler.load_data(path)
    110
    111         # Attempt to link the data

~/src/glue/glue/core/application_base.py in wrapper(*args, **kwargs)
     26                 detail = str(traceback.format_exc())
     27                 self = args[0]
---> 28                 self.report_error(m, detail)
     29         return wrapper
     30     return decorator

~/src/glue/glue/core/application_base.py in report_error(self, message, detail)
    253             Longer context about the error
    254         """
--> 255         raise Exception(message + "(" + detail + ")")
    256
    257     def do(self, command):

Exception: Could not load data
"Don't know how to open file: /home/erik/none"(Traceback (most recent call last)
:
  File "/home/erik/src/glue/glue/core/application_base.py", line 23, in wrapper
    return func(*args, **kwargs)
  File "/home/erik/src/glue/glue/core/application_base.py", line 197, in load_da
ta
    result = load_data(path)
  File "/home/erik/src/glue/glue/core/data_factories/helpers.py", line 262, in l
oad_data
    raise KeyError("Don't know how to open file: %s" % path)
KeyError: "Don't know how to open file: /home/erik/none"
)
Exception: Could not load data
"Don't know how to open file: /home/erik/none"(Traceback (most recent call last)
:
  File "/home/erik/src/glue/glue/core/application_base.py", line 23, in wrapper
    return func(*args, **kwargs)
  File "/home/erik/src/glue/glue/core/application_base.py", line 197, in load_da
ta
    result = load_data(path)
  File "/home/erik/src/glue/glue/core/data_factories/helpers.py", line 262, in l
oad_data
    raise KeyError("Don't know how to open file: %s" % path)
KeyError: "Don't know how to open file: /home/erik/none"
)

Overlays in BQPlot

• Overlays in BQPlot (image and contour)?

• If not, how hard to put the following into glue-jupyter

  • Plotly
  • ginga

application embed into front-end

I've got a working prototype of the jdaviz application embedded into a separate front-end using voila-embed. My embed notebook is the following:

Import the application

import ipyvuetify as v
from jdaviz.app import Application
app = Application(configuration='specviz')
app

Function to load data on change event

def update_data_handler(widget, event, data):
    # get new data from selected target
    selected_src = int(data)
    data = prep_data(selected_src)
    app.load_data(data)

Embed the app in a ipyvuetify row/col layout to attach specific mount_id and event handler.

col = v.Col(xs12=True, lg6=True, xl4=True, children=[app])
row = v.Row(_metadata={'mount_id': 'specviz'}, dense=True, row=True, wrap=True, align_center=True, children=[col])
row.on_event('change', update_data_handler)

The primary reason I wrap the application in an ipyvuetify Row/Col is to attach a mount_id so I can embed it in my front-end and add a custom event handler for updating the data. Is there a better way of doing this? I noticed the application has a mount_id in app._metadata. Is this currently being used, or something I can modify? I also wasn't sure if custom event handlers can be attached to the application directly.

Issue with viewer size

I tried the latest version of jdaviz today with the cubeviz layout and ran into the issue that I'm unable to see the profile viewer because the image viewer always auto-adjusts to fill the browser window (even if I change the browser font size etc):

Minor update on installation instructions

Needs ipysheet.

additional context tooltips on toolbar icons

All the toolbar icons should have tooltips to provide additional context similar to the icon for the Gaussian Smooth option. In particular, the icons on the g-data-tools are missing. This make it difficult to know which sets of tools or menu items to turn on or off for different cases.

embed web kernel framework into mast html page to explore web/kernel communication

Embed @nmearl ipython kernel framework into the MAST index html page. Current blocker is the broken bqplot/glue connection, #13 . In the meantime while that fix is explored, we well implement a simple basic (set of) ipywidget(s), e.g. text box + slider, to test the communication between the web front-end and the kernel back-end.

The three cases to explore

• click something in the widget to inject information into the html table
• click web element (button or table data) to manipulate the widget and/or inject information into it
• click web element to make a "public astroquery" call to manipulate/update the widget content

Caveats to take note of:

• This basic setup will need to be repeated once the CubeViz widget is fixed. There may be additional complications moving from a basic widget setup to a glue-based VizApp setup.
• The 6 ASB technology prototype points should be shown using the CubeViz widget if possible.
• A possible solution of direct ASB back-end and iPython kernel back-end bi-directional communication needs to be considered, in addition to the proposed above approach. This solution may make things easier in practice, and a possible prototype should be explored. The 6 ASB technology prototype points will need to demonstrated in this alt-prototype.

Relevant People: @havok2063 @nmearl @eteq

greater transparency between glue/widget components and underlying data

The underlying data gets lost somewhere inside the glue/widget components. It's not clear how to get back the underlying FITS data. This is useful in general but will also be necessary I think to implement the "point-and-click" spaxel view and to extract info for display in hover tooltips.

For example, using the cubeviz example notebook, I can access the glue Data container which contains limited data components. Similarly, a (limited?) header appears inside data.meta but meta isn't mentioned anywhere in the documentation on glue Data objects. I discovered it by accident.

imlayerstate = c._v3d._v3d.state.layers[0]
data = imlayerstate.layer
flux = data.components[-1]
flux_data = data[flux]
hdr_data = data.meta

It might be useful to store the full original data HDU somewhere inside the container or widget and make it transparent as to how to retrieve it.

Bug with Ipyvuetify Layout wrapping Ipywidgets Vbox and Hbox

We found a bug last week in Ipyvuetify, confirmed by the devs, that Ipyvueity cannot currently take in HBox and VBox objects as children in the Layout instance. Or at least, there are bugs when you try to do this.

I will post to this repo shortly with a PR that has my test case in it, but here's the important part:

self._main_box = v.Layout(children=[widgets.VBox([self._menu_bar, widgets.HBox([self._v3d.show(), self._v1d.show()])])])

This code does not render the menu_bar item. Additionally, although were not sure if this is the same bug, in the glue component, the tabs to the right cannot be activated when wrapped in a Ipyvuetify Layout

Feedback from the developers:

There are some issues with nesting VBox and HBox in ipyvuetify widgets. The alternative for VBox and HBox is the vuetify grid system: https://vuetifyjs.com/en/framework/grid#grid-system. (click the "< >" icon at the top right of the examples to see the source code)

Example:
v.Layout(row=True, wrap=True, children=[
v.Flex(xs12=True, class_='px-2', children=[
v.Btn(block=True, children=['menu'])
]),
v.Flex(xs6=True, class_='px-2', children=[
v.Btn(block=True, children=['v3d'])
]),
v.Flex(xs6=True, class_='px-2', children=[
v.Btn(block=True, children=['v1d'])
]),
])

Decide whether mosviz spectrum view should be a full-fledged specviz or a subset of the specviz viewer

What it says on the tin - partly driven by some of the discussion around #100. Some complexity, though:

The solution from the qt mosviz was a "open in specviz" button, which launched basically a separate app window. That idiom doesn't work so well in jdaviz because in jupyter there's not really a natural way to launch a "new window", although we presumably could do something in the notebook view that presents it as a new cell. So that leads me in the direction of thinking we should aim for having the spectrum viewer have basically all the capabilities of specviz (including relevant plugins), but have them only operate on the currently active spectrum.

With that proposal, the biggest hurdle I see is that Mosviz (at least the #100 concept) can change which spectrum is being shown at any time. We need to make sure any relevant specviz functionality isn't broken by that. My thinking is just to basically reset the viewer when a new spectrum is shown. Later we might consider remembering state, but that at least works around the complexity for now.

cc @hcferguson @nmearl @robelgeda

Whitelisting errors

When starting up the voila example with:

voila notebooks/Example.ipynb

I get a bunch of 403 errors:

WARNING:tornado.general:403 GET /voila/files/favicon.ico (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/favicon.ico (::1) 0.94ms
WARNING:tornado.general:403 GET /voila/files/voila/jupyter-vue.js (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/voila/jupyter-vue.js (::1) 1.07ms
WARNING:tornado.general:403 GET /voila/files/voila/jupyter-vuetify.js (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/voila/jupyter-vuetify.js (::1) 1.84ms
WARNING:tornado.general:403 GET /voila/files/voila/jupyter-materialui.js (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/voila/jupyter-materialui.js (::1) 1.93ms
WARNING:tornado.general:403 GET /voila/files/voila/bqplot.js (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/voila/bqplot.js (::1) 1.07ms
WARNING:tornado.general:403 GET /voila/files/voila/bqplot-image-gl.js (::1): File not whitelisted
WARNING:tornado.access:403 GET /voila/files/voila/bqplot-image-gl.js (::1) 1.59ms

The easiest workaround for now is just to whitelist all files:

voila notebooks/Example.ipynb --VoilaConfiguration.file_whitelist="['.*']"

Demonstrate and document how to define plugin UI elements at various levels

We should document how to define a custom:

• Sidebar plugin
• Global toolbar plugin
• Viewer toolbar plugin

Any other kinds of UI plugins we should be thinking of in the context of cubeviz?

Cross posting test script

Can delete soon.

Determine how to programmatically show collapsed/modified versions of data in UI

One of the requirements for the cubeviz prototype is the ability to do e.g. smoothing or collapsing of spectra or spectral cubes and showing it in the cubeviz UI. So this issue is to figure out what the programmatic API looks like for doing any kind of analysis that involves collapsing or modifying data. For example, if we are working with a spectral cube in glue, do we get the SpectralCube object out, use the collapse methods on SpectralCube, then add the result back to glue?

One of the subtleties here will be that if we want this modified data to be linked with the original one, we will likely have to auto-infer links between the data when the collapsed/modified version is added back. I can try and help with this since I've worked on auto-linking in the Qt UI before.

error on instantiating VizApp

I'm trying to get the latest version of jdaviz up and running on my system to test out both the example notebooks and the ASB web components. I've updated all repos and reinstalled everything following the instructions in the jdaviz README and jdaviz.web README. I am running in a new notebook within the jdaviz conda environment

from jdaviz.vizcomponents.applications import cubeviz
from jdaviz.vizcomponents.vizapp import VizApp
vizapp = VizApp()

and get an error that traces through glue-jupyter, ipywidgets and traitlets. Full traceback is below.

I'm running inside the jdaviz conda environment

• glue-core 0.15.3
• glue-jupyter - updated dev
• jdaviz - updated dev
• bqplot - updated dev
• ipywidgets 7.5.1
• traitlets - 4.3.2
• bqplot - updated dev
• ipyvuetify - updated dev

Full traceback..

---------------------------------------------------------------------------
TraitError                                Traceback (most recent call last)
~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in validate_elements(self, obj, value)
   2251             try:
-> 2252                 v = self._trait._validate(obj, v)
   2253             except TraitError:

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in _validate(self, obj, value)
    590         if hasattr(self, 'validate'):
--> 591             value = self.validate(obj, value)
    592         if obj._cross_validation_lock is False:

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in validate(self, obj, value)
   1676         else:
-> 1677             self.error(obj, value)
   1678 

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in error(self, obj, value)
   1523 
-> 1524         raise TraitError(e)
   1525 

TraitError: The 'None' trait of a MenuItem instance must be a DOMWidget, but a value of class 'str' (i.e. 'No selection (create new)') was specified.

During handling of the above exception, another exception occurred:

TraitError                                Traceback (most recent call last)
<ipython-input-1-3349a409ddfd> in <module>
      1 from jdaviz.vizcomponents.applications import cubeviz
      2 from jdaviz.vizcomponents.vizapp import VizApp
----> 3 vizapp = VizApp()

~/Work/git/stsci/jdaviz/jdaviz/vizcomponents/vizapp.py in __init__(self)
     19 
     20 
---> 21         self._glue_app = gj.jglue()
     22 
     23         self._history = []

~/Work/git/stsci/glue-jupyter/glue_jupyter/__init__.py in jglue(*args, **kwargs)
     58         dc.add_link(parse_links(dc, links))
     59 
---> 60     japp = JupyterApplication(dc)
     61     if show:
     62         display(app)

~/Work/git/stsci/glue-jupyter/glue_jupyter/app.py in __init__(self, data_collection, session)
     44         self.output = widgets.Output()
     45         self.widget_data_collection = widgets.SelectMultiple()
---> 46         self.widget_subset_select = SubsetSelect(self.session)
     47         self.widget_subset_mode = SubsetMode(self.session)
     48         self.widget = widgets.VBox(children=[self.widget_subset_mode, self.output])

~/Work/git/stsci/glue-jupyter/glue_jupyter/widgets/subset_select.py in __init__(self, session)
     26         )
     27         self.widget_menu_item_no_active = mui.MenuItem(children=["No selection (create new)"],
---> 28                                                        value="new")
     29         self.widget_menu_items_subsets = []
     30         self.widget_menu_item_no_active.observe(self._on_click_menu_item_no_active, 'selected')

~/anaconda3/lib/python3.7/site-packages/ipywidgets/widgets/widget.py in __init__(self, **kwargs)
    410         """Public constructor"""
    411         self._model_id = kwargs.pop('model_id', None)
--> 412         super(Widget, self).__init__(**kwargs)
    413 
    414         Widget._call_widget_constructed(self)

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in __init__(self, *args, **kwargs)
    995             for key, value in kwargs.items():
    996                 if self.has_trait(key):
--> 997                     setattr(self, key, value)
    998                 else:
    999                     # passthrough args that don't set traits to super

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in __set__(self, obj, value)
    583             raise TraitError('The "%s" trait is read-only.' % self.name)
    584         else:
--> 585             self.set(obj, value)
    586 
    587     def _validate(self, obj, value):

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in set(self, obj, value)
    557 
    558     def set(self, obj, value):
--> 559         new_value = self._validate(obj, value)
    560         try:
    561             old_value = obj._trait_values[self.name]

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in _validate(self, obj, value)
    589             return value
    590         if hasattr(self, 'validate'):
--> 591             value = self.validate(obj, value)
    592         if obj._cross_validation_lock is False:
    593             value = self._cross_validate(obj, value)

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in validate(self, obj, value)
   2240             return value
   2241 
-> 2242         value = self.validate_elements(obj, value)
   2243 
   2244         return value

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in validate_elements(self, obj, value)
   2252                 v = self._trait._validate(obj, v)
   2253             except TraitError:
-> 2254                 self.element_error(obj, v, self._trait)
   2255             else:
   2256                 validated.append(v)

~/anaconda3/lib/python3.7/site-packages/traitlets/traitlets.py in element_error(self, obj, element, validator)
   2231         e = "Element of the '%s' trait of %s instance must be %s, but a value of %s was specified." \
   2232             % (self.name, class_of(obj), validator.info(), repr_type(element))
-> 2233         raise TraitError(e)
   2234 
   2235     def validate(self, obj, value):

TraitError: Element of the 'children' trait of a MenuItem instance must be a DOMWidget, but a value of 'No selection (create new)' <class 'str'> was specified.

key error of "Astronomy WCS" in links

After merging in the latest changes from #88, loading test JWST x1d data no longer works. I reinstalled jdaviz after pulling in the changes thinking that would also update any necessary glue components. Are there other glue pieces I should update to fix this?

from jdaviz.app import Application
app = Application(configuration='specviz')
app
data = '/Users/bcherinka/Downloads/jw00626-o030_s00000_nirspec_f170lp-g235m_x1d.fits'
app.load_data(data)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-0a27a898ff00> in <module>
      1 #data = '/Users/bcherinka/Work/mosviz/data/jwst_level3_NIRSpec/jw95065_1186_nrs_msaspec_barshadow_x1d.fits'
      2 data = '/Users/bcherinka/Downloads/jw00626-o030_s00000_nirspec_f170lp-g235m_x1d.fits'
----> 3 app.load_data(data)

~/Work/git/havok2063/jdaviz/jdaviz/app.py in load_data(self, path)
    113         links = find_possible_links(self.data_collection)
    114 
--> 115         self.data_collection.add_link(links['Astronomy WCS'])
    116 
    117     def viewers(self, reference):

KeyError: 'Astronomy WCS'

Create ImageViz component, Application and Notebook Example

There are a few steps to this PR:

• see if the slicer viewer can show a single image
• create an ImageViz application
• create an example notebook that shows the ImageViz application based on the data

data-flow diagram initial attempt

To get started on trying to document the data flow, I've created an editable draw.io chart.
https://drive.google.com/file/d/1IOIZ8MZswztJ564PzmCgvVL3zhUSu7cw/view?usp=sharing

My thinking is to first pick off a simple case of setting an ROI on a 1D spectrum plot in the jadaviz version of specviz. I've written down the various actions and the various packages that might need to react to this event.

I think it would be useful to draw arrows for the data flow between the packages, but I'm not sure of the sequence, or even whether they are all involved.

It would probably also be useful to identify internal name of the event handler or callback in each package for this type of event.

Maybe also add a notation to indicate which packages are maintaining a record of this ROI? For this example, I'm pretty certain glue and specutils and the 1D plot widget and bqplot (or plotly) all probably have their own internal representation of the ROI. Not sure if specviz would have this independent of the widget or if any of the other packages would have it.

imageviz viewer with two 2D image viewer components

(in notebook)

If we have two image viewer components being displayed, can we lock the axes with two dithered images? (one is a reference and one is a re-projection)

Top-level spectral axis scrubber for cubeviz

The idea of this is to have the three cube viewers all share a tied-together spectral_axis which is also shown in the top-bar (or optionally, if it's too hard to fit on the bar, instead could be it's own small section above the 3-flux-panels). It should have the following characteristics:

1. This bar should look the same as the spectral axis scrubber that currently appears in the viewers individually. (Note that JDAT-276 is about fixing a bug in that - the point here is that we should just re-use what's in glue already so that that fix comes for "free" in what's implemented here.)
2. The bar should be "linked" to the three panels such that when the wavelength is scrubbed, if the user is also showing the detail view for the viewer (where the slider is currently shown), they should update as well.
3. The "spectrum" view (the profile viewer) should have a partially-transparent vertical line of some sort that indicates where the slider bar is in the spectrum.
4. The above assumes all of the data has been populated in a sensible way - e.g. that the three panels are all views into the same cube (and the profile viewer is a profile of one of those three). If that assumption is not valid, the slider shouldn't fail - rather it should just be linked to the "left-most" cube view, and just not do anything to the other viewers if they have the wrong type of data in them.
5. If the user does some operation like cube-collapse which adds a new layer to the three cube viewers that isn't a "cube", the shared slider bar should be aware of this in some way that the user can see. The most straightforward thing would be to just grey-out the slider bar in this case (since it's no longer a thing that can be sliced), although another option would be to have the bar stay such that if the user moves the slider, it goes back to the sliced cube and hides the result of the operation.

Note 1, 2, 4, and 5 above are the "core" functionality, while 3 is not strictly necessary (although very important for the user to see where they are). So this could be accepted without 3 if that is technically too difficult, although we should then add a separate story for later about that.

load_configuration does not reset the app viewers

Calling load_configuration to dynamically change the viewer config does not reset the existing config. Rather, it (cumulatively) loads a completely separate configuration, including toolbars and viewers.

Running

app = Application(configuration='cubeviz')
app
app.load_configuration('specviz')

produces the following where the new configuration as been loaded to the right of the existing one.

Re-consider "each viewer has its own data" paradigm in light of user feedback

Following a recent hack hour, there's been some concern raised that the current jdaviz paradigm that the each viewer has its own data "drawer" presents usability concerns. We should discuss if we want to rethink this paradigm, particularly given a shift to plugins-as-sidebar.

cc @nmearl

Make it possible to launch jdaviz with electron

We should make sure that it is possible to launch jdaviz using electron instead of opening it in a regular browser. This could even be a command-line option --electron to jdaviz.

Have cubeviz spectral models have "infinite resolution"

This is a forward-looking "user story", as some of this capability doesn't exist. The gist of it is as follows:

1a. Separately we know we want a spectral modeling plugin, which would result in models fit to the current spectrum that's been extracted from the cube. In that case the model object will already be associated with the Cubeviz app object.
1b. A user could also make a model, and they should be able to load it via something like cubeviz.add_spectral_model(my_preferred_model) (which should be pretty accommodating but at least check that it's a model that's valid for a spectral x-axis).
2. Once the model exists, it should get plotted in the spectral view of cubeviz.
3. When the user moves or zooms the view, the model (unlike the data) should update to evaluate the model to acheive a reasonable resolution based on the view. The exact resolution is debatable, but some initial items to try might be either 1 resolution item per screen pixel (which may or may not be easy to compute) or a fixed but high number like "1000 points per visualization".

cc @nmearl @astrofrog

bqplot + glue plotting issue

Currently, there's an issue with rendering bqplots that are provided by the glue interface. While normal bqplots render fine in the desktop and web page environments, when utilizing glue to generate the plots as part of the ipywidgets "application", the rendering is mangled with various errors in the browser console.

Develop a Gaussian smoothing sidebar plugin

One of the features needed for the cubeviz prototype is a sidebar that provides tools for doing Gaussian smoothing of the data. It wasn't clear to me if this should be spectral or spatial smoothing, but I would imagine that both could be useful, so this could be a parameter in the sidebar?

Implement sub-set cube collapsing

The cube collapse plugin currently collapses along the full cube instead of the selected subset. For many/most use cases, it's more desirable to use the subset (this is already implemented in glue we think, but not exposed in the UI).

Probably it also makes sense to have the "most recent" subset auto-selected since that's a pretty natural user workflow (as opposed to defaulting to the full cube and making them click down to the subset)

Circular selection plugin not picked up

When I try the latest developer version of jdaviz I see the following in the web console:

Error: "Could not process update msg for model id: edb3a595a5764d80aae093c1e246bb4f"
    _ utils.js:119
utils.js:119
Class BrushEllipseSelectorModel not found in module bqplot-image-gl@^0.2.0

I don't see this when using plain notebooks, just when using jdaviz.

cc @maartenbreddels @nmearl

Software Engineering - Going Forward

We had a short discussion among the dev team to get on the same page for the visualization tools, what needs to get done and priorities. The discussion was based on notes from https://innerspace.stsci.edu/display/~crjones/JDAViz+Next+Steps.

Please add your discussion items below. I'll add some more details from the discussion here, as I understood them.

@robelgeda @javerbukh @astrofrog @SaOgaz @nmearl @ibusko

Revisit/decide whether the viz apps should be subclasses

This issue is to record discussion about whether or not the viz apps should be sub-classes with custom methods/etc of the glue application object, vs purely configuration.

The idea is that regardless of the answer to this question, the UI layout and the like would still all be configuration. The stuff in the sub-classes I imagine to be primarily notebook-friendly API to allow users to interact with the apps in a way that's natural in Python without having to push a lot of complicated app-specific machinery onto the generic Application object.

Cleanup python framework

The original framework came from another style of framework I was writing and some of the parts are not relevant for this work here. So, I need to go through and remove some of the extra unnecessary code.

Second, remove code that is replicating functionality already inherent in the glue-jupyter state information so there is not a redundancy.

change default data load/display for x1d data

For JWST x1d data, the first HDU extension is a binary table

No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     353   ()      
  1  EXTRACT1D     1 BinTableHDU     50   1331R x 9C   [D, D, D, D, D, J, D, D, D]   
  2  ASDF          1 BinTableHDU     11   1R x 1C   [33445B]  

with WAVELENGTH as the first column and FLUX/ERROR as the next two. When loading this data, the current default is to plot is the 1st column. This results in a wavelength vs wavelength plot and the user needs to change the Layer Attribute to get to FLUX. Instead, the default should be to display the FLUX column and (probably also) overplot the ERROR.

Expose the navigation and selection toolbar next to each viewer

Currently the viewers in jdaviz don't include the standard navigation/selection toolbar that is normally included in glue-jupyter viewers:

We should make sure that these are exposed so as to be able to pan around and also make selections.

vuetify sidebar does not properly toggle in mast embed

The vuetify sidebar does not collapse as expected. The icons remain outwards no matter the state the sidebar is in.

In State

Out State

Grab Cubeviz ipywidget notebook and use ipvuetify layout

If the ipvuetify layout works, continue with wrapping all of the Cubeviz layout in it. If not, add more functionality to the Cubeviz notebook.

subset layer attribute should match original selected layer

When selecting a subset, the selected (and overplotted) layer attribute should match the layer attribute of the original source.

To reproduce:

• Load JWST test x1d data
• Load JWST EXTRACT1D hdu extension in Data. Default layer attribute displayed is WAVELENGTH
• Change Layer Attribute to FLUX
• Create a new subset selection
• Subset layer is set to WAVELENGTH instead of FLUX

Look into adding more functional requirements to the mosviz ipywidget notebook

Grab the mosviz ipywidget notebook and look into adding more functional requirements.

toolbar icons do not display with mast embed

When jdaviz is embedded into a separate website using https://github.com/mariobuikhuizen/voila-embed, the toolbar widget icons do not display correctly. I'm serving from an environment where I have both voila-embed and jdaviz installed. The toolbar displays correctly both in the notebook and when run as a desktop app with jdaviz --layout .... Are there some additional style files that need to be included in our frontend? The functionality of the buttons still exist.

Layout Configuration and Application Engine discussion

The question is how to create a "Cubeviz" Application (for example) based on some layout format (let's say a default 3 3D slicers and 1 1D viewer). And the final application "thing" will also have to contain the Glue-Jupyter data manager (gj_app).

@eteq and I talked for a bit today and there are a few ways this could go. For example...

Configuration

We probably need a configuration class that represents a default or initial state of an application, for example a CubeViz Application.

class Configuration:
     def __init__(self, ...):
        self._configuration_dict = {}

The configuration_dict would be some dictionary representation of a default layout of the component widgets, possibly including some way to specify how to load data, and maybe some look-and-feel configuration for the application or the component widgets. It might be nice to be able to specify an instance of a Configuration object in the self._configuration_dict too so then you could nest them (more on that in a minute).

The format of the self._configuration_dict` is not worked out yet and may be dependent on vuetify, golden ratio layout or other things. But at a really simple level:

cubeviz_configuration_dict = {
    'layout': [['v3d1', 'v3d2', 'v3d3'], ['v1d']],
    'v3d1': {
        'default_data': 'https://some_manga/data.fits',
        'look_and_feel': {'background': 'neon_blue'}
    }
}

The above dict could represent information about how things are laid out, the data loaded and things to do with look-and-feel. The look-and-feel dict could be passed directly to the 3Dslicer viewer and make it interpret the key-value pairs.

The other thing one could maybe do is create an electron view of the configuration dict:

cubeviz_electron_configuration_dict = {
   'layout': [['menu_bar'], [cubeviz_configuration_dict]],
    'menu_bar': [things in the menu bar]
}

Then, this way, we could nest the configuration definitions so that the first one could be used in a notebook and then the second could be used in electron. But the inside look and feel is all the same. Just an idea.

Application

The application class that once instantiated, would actually be the CubeViz object (for example) could be based on:

class Application:
    def __init__(self, configuraiton, gj_app = None):
        if not gj_app:
            gj_app = create gj_app
        self._data_manager = gj_app

        initial_configuration = configuration

where configuration in the __init() is an instance of the Configuration class from above.

Questions and Thoughts

• This is only one way to do this and is based on a discussion between myself and @eteq
• Should the Application class actually subclass Glue-Jupyter app rather than have a reference to it?
• Is it ok to mix string class and python objects in the Configuration class?
• How do we deal with a multi-meaning of application in all these things?

@eteq Please clarify (or even edit) the above so that it best represents our discussion.

@astrofrog We would really like your thoughts on this. The initial idea is to do a first pass implementation in jdaviz but with the plan to move it to Glue Jupyter when "ready".

Cc @javerbukh @nmearl @camipacifici @hcferguson

Proposal for redressing data list usage within viewers

Based on conversations out-of-band between myself, @eteq, @dtn5ah, and @rosteen, we've come to some conclusions concerning a few issues about the current usability of the data list panel and the management of data from e.g. within a viewer vs outside a viewer.

The main issues boil down to:

• Access to the data list is non-obvious, and user feedback suggests it is too buried in the UI for easy usage.
• There's currently no UI feedback about data being added to the internal data collection aside from viewing the data list, which access to is non-intuitive.
• Managing data within viewers is currently done via the data list panel (e.g. the ability to add and remove data is tied to the toggle state of the element in the data list). This should instead be done using layers, which is what is actually created when a data set is added to a viewer. It should be noted that data is global, and is not tied to any specific viewer(s), but layers are local to the viewer instance. Therefore, users should be using layers to manage their viewer data, not the checkboxes in a data list.

The major proposed solution is that a new data list that exists outside the viewers should be added that can be used to drag data sets onto a viewer to populate the viewer with the data. This would be easily accessible and obvious, and can perhaps reflect the current number of loaded data sets. This draggable data list does not reflect the state of the data of any particular viewer, and in essence just behaves like the typical QtGlue data list (for interested parties, this is very similar to the original proposed solution at the start of the project, see relevant confluence pages concerning why this approach was not taken originally). This is predominately a UI addition and does not influence the data list access from individual viewers approach currently supported.

A second usability point is the ability to add data to viewers from within viewers. Akin to what is currently supported, however subtly different in that data management is now done appropriately using layers within a viewer. In this case, data can still be added to a viewer via e.g. a new dropdown within the layer options tab, but once added, users must use the layer tools already provided if they wish to remove the data set from the viewer. This would then support the ability to create multiple layers from the same data set, something that is not currently supported in the data list checkbox approach.

To recap:

1. users can use the global access menu to drag-and-drop data sets onto viewers to create layers within the viewer, and
2. an additional widget within the layer options panel in a specific viewer would allow data to be added to the viewer in the same fashion.
3. Finally, management of data within a viewer would be done via their viewer-specific layer proxy.

Add a context/vision/high-level intro section in the jdaviz docs

Just what it says on the tin... There needs to be words at the top that say what jdaviz is about, how it relates to the older viz tools, glue etc.

Have cubeviz auto-recognize and load into viewers interpretable cube datasets

Right now when one either via click or by command line loads data into cubeviz, the user has to manually assign the particular data cubes to the various windows in cubeviz. That's not good because when a user loads "a data cube" they expect the cube viewers in the layout to be a particular set of cube planes - flux, uncertainty, and mask.

So this issue is about working out a way to have this happen automatically (or semi-automatically) when a cube is loaded. I can think of several cases to consider here:

1. A cube that can be laoded by spectral-cube, vector/cubespecutils.Spectrum1D or specutils.SpectralCollection using their built-in readers, which then correctly populates the various attributes like uncertainty or mask. JWST cubes should end up in this category.
2. A fits file where the cube planes are in separate HDUs which follows some naming convention we can guess things based on (e.g. the SDSS MANGA cubes) but does not have a native reader.
3. Some home-grown cube format that comes out of a specific instrument's pipeline or unique capabilities (e.g. the outputs of CWI data reduction pipeline, MUSE, etc.)

This issue is meant to only address 1 and 2. 3 We might consider in the future, but for now would just need to be dealt with manually by the user (and that's OK).

Case 1: in this case when the data is loaded, things should just happen automatically. .flux of a spectrum should be recognized as belonging in the first window, .uncertainty in the second, .mask in the third. This should happen whether the file is loaded as a file or as a live object in the notebook interface via app.load_data (it should all be the same machinery, which I think it is already).

Case 2: This could be "turned into" case 1 by having a function that generates the relevant case 1 data object using naming heuristics. E.g., if "flux" is in the name of the HDU, call that the flux, "unc" or "err" the uncertainty, "mask" the mask, etc. Then the user can try out the heuristic by calling the function if they want to, but in the app they can just load the file and have it transparently work if they don't care about the details. The one catch I see in this case is that it's not clear what to do about "extra" HDUs that the user might want to have an option to look at. Maybe they end up in .meta from the heuristic function? (and then still appear in glue as currently).

cc @nmearl @astrofrog

Create Mosviz notebook with ipyvuetify

This notebook will be based on the similar work @SaOgaz did for PR #10

Make it possible to view the three images side by side in cubeviz layout

This may require implementing the golden layout approach, but we should demonstrate that it is possible to show the three image viewers side by side in the cubeviz layout. Currently the three image viewers are tabbed:

"SpecViz" Viewer1d does not render correctly in web/Electron

Part of ASB Q1 work entails developing a modular 1-d spectral viewer component than can potentially be plugged into multiple ASB sites, using the jdaviz "specviz" Viewer1d app. Displaying a spectrum currently works inside Jupyter notebooks but does not work inside the Electron app or any hosted website. I have tried several Chrome and Firefox, a fresh computer, and a fresh install with up-to-date dependencies.

If it works for others, can we get a list of the exact versions of all necessary software components?

For ASB work to progress, this needs to be functionally stable for the foreseeable future. Temporarily we can use an ImageViewer component or something like that so we're not entirely blocked by this.

My widget_code.json is

    "filename = '/Users/bcherinka/Work/notebooks/datb/manga-7495-12704-LOGCUBE.fits'",
    "from jdaviz.vizcomponents.viewer.viewer1d import Viewer1D",
    "from jdaviz.vizcomponents.vizapp import VizApp",
    "vizapp = VizApp()",
    "vizapp.glue_app.load_data(filename)",
    "v1d = Viewer1D(vizapp)",
    "v1d.show()"  

The web display and console traceback is attached below.