nasa/kamodo

The CCMC Kamodo Analysis Suite

Online Documentation

https://nasa.github.io/Kamodo/

Vision Statement

Kamodo is an official NASA open-source python package built upon the functionalization of datasets. Once a dataset is functionalized in Kamodo, several important capabilities are then available to the user, including data analysis via function composition, automatic unit conversions, and publication quality graphics all using intuitive and simplistic syntax. By applying these capabilities to heliophysics model outputs, we aim to:

• Drastically simplify the currently complex data utilization process for model outputs,
• Provide interactive access to functionalized model outputs for users ranging in programming skill from beginners – via code-free interfaces and video tutorials – to advanced users – via thorough documentation, Jupyter notebook examples and sample workflows,
• Layer multiple functionalities on top of the functionalized model outputs, all with model-agnostic and uniform syntax, including but not limited to:
  • Flythrough tools,
  • Vector field tracing (including magnetic field mapping),
  • Coordinate conversions,
  • Domain-specific interactive plots of publication quality,
  • Modular driver swapping,
  • Satellite constellation mission planning tools,
  • Simulated imagery, and
  • A line of sight calculation tool,
• Greatly reduce the programming skill currently required outside of Kamodo to perform model validation studies and model-data comparisons,
• Enable model output utilization both on the cloud and on personal laptops in a variety of methods (e.g. through HAPI and interactive calls from the command line),
• Streamline the CCMC user workflow by becoming interoperable with other CCMC services (e.g. CAMEL and the various scoreboards),
• And become the next generation interface for CCMC users to interact with and analyze model outputs (e.g. through ROR and IR),

...all while keeping the developed software open-source and freely available. The Kamodo team also supports the heliophysics community by pursuing interoperability with commonly-used python packages, collaborating with community members to add model outputs and new functionalities, and remaining involved with community events (e.g. conferences, challenges, and research support). As the library of supported model outputs types expands and new model-agnostic tools are added, Kamodo will become a staple software package in the heliophysics community to transform current workflows into a more efficient and productive process. We are building the next generation of capability with Kamodo. Join us!

Kamodo currently supports:

• ADELPHI: AMPERE-Derived ELectrodynamic Properties of the High-latitude Ionosphere, https://doi.org/10.1029/2020SW002677
• AMGeO: Assimilative Mapping of Geospace Observations, https://doi.org/10.5281/zenodo.3564914
• CTIPe: Coupled Thermosphere Ionosphere Plasmasphere Electrodynamics Model, https://doi.org/10.1029/2007SW000364
• DTM: The Drag Temperature Model, https://doi.org/10.1051/swsc/2015001
• GITM: Global Ionosphere Thermosphere Model, https://doi.org/10.1016/j.jastp.2006.01.008
• IRI: International Reference Ionosphere Model, https://doi.org/10.5194/ars-16-1-2018
• OpenGGCM_GM: The Open Geospace General Circulation Model - GM outputs only, https://doi.org/10.1023/A:1014228230714
• SuperDARN_uni: SuperDARN uniform grid output ,https://doi.org/10.1029/2010JA016017
• SuperDARN_equ: SuperDARN equal area grid output, https://doi.org/10.1029/2010JA016017
• SWMF_IE: Space Weather Modeling Framework - IE outputs, https://doi.org/10.1029/2006SW000272
• SWMF_GM: Space Weather Modeling Framework - GM outputs, https://doi.org/10.1029/2006SW000272
• TIEGCM: Thermosphere Ionosphere Electrodynamics General Circulation Model, https://doi.org/10.1029/2012GM001297
• WACCMX: Whole Atmosphere Community Climate Model With Thermosphere and Ionosphere Extension, https://doi.org/10.1002/2017MS001232
• WAMIPE: The coupled Whole Atmosphere Model - Ionosphere Plasmasphere Model, https://doi.org/10.1002/2015GL067312 and https://doi.org/10.1029/2022SW003193
• Weimer: Weimer Ionosphere model, https://doi.org/10.1029/2005JA011270
• It also supports any data available through the HAPI interface.

Kamodo Installation Instructions

Kamodo is built to run with at least 16 GB of RAM. Attempting to run Kamodo with less memory may result in errors.

Conda prompt commands:

• Move to the directory where you want the Kamodo package to be stored or if you wish to create a new environment, use this command:

conda create -n Kamodo_env python=3.7

• Add the packages needed by the CCMC readers to the desired environment (replace 'Kamodo_env' with your environment name):

conda install -n Kamodo_env -c conda-forge netCDF4 cdflib astropy ipython jupyter h5py sgp4

• Activate the desired environment.

conda activate Kamodo_env

• Install remaining dependencies:

python -m pip install --upgrade spacepy
python -m pip install hapiclient

• Download CCMC Kamodo to the current directory:

git clone https://github.com/nasa/Kamodo.git

• Install the CCMC Kamodo package. (Check the directory structure before using this command. The ./Kamodo directory should contain the kamodo_ccmc directory.)

python -m pip install ./Kamodo

Note: Developers should install CCMC Kamodo with the -e option

Testing commands from an ipython or notebook session

from kamodo import Kamodo
k = Kamodo()  
import kamodo_ccmc.flythrough.model_wrapper as MW  
MW.Model_Variables('OpenGGCM_GM')

The OpenGGCM_GM model accepts the standardized variable names listed below.
-----------------------------------------------------------------------------------
B_x : '['x component of magnetic field', 0, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']'
B_y : '['y component of magnetic field', 1, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']'
B_z : '['z component of magnetic field', 2, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']'
E_x : '['x component of electric field', 6, 'GSE', 'car', ['time', 'x', 'x', 'x'], 'mV/m']'
E_y : '['y component of electric field', 7, 'GSE', 'car', ['time', 'y', 'y', 'y'], 'mV/m']'
E_z : '['z component of electric field', 8, 'GSE', 'car', ['time', 'z', 'z', 'z'], 'mV/m']'
N_plasma : '['number density of plasma (hydrogen equivalent)', 12, 'GSE', 'car', ['time', 'x', 'y', 'z'], '1/cm**3']'
P_plasma : '['plasma pressure', 14, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'pPa']'
eta : '['resistivity', 13, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'm**2/s']'
j_x : '['current density, x component', 15, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']'
j_y : '['current density, y component', 16, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']'
j_z : '['current density, z component', 17, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']'
v_plasmax : '['x component of plasma velocity', 9, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']'
v_plasmay : '['y component of plasma velocity', 10, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']'
v_plasmaz : '['z component of plasma velocity', 11, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']'

Citing Kamodo

When publishing research which used Kamodo, please provide appropriate credit to the CCMC and the Kamodo team via citation or acknowledgment. Please also let the team know of publications or presentations that use Kamodo. Below is list of publications for Kamodo.

• Pembroke, A., D. De Zeeuw, L. Rastaetter, R. Ringuette, O. Gerland, D. Patel and M. Contreras (2022). Kamodo: A functional API for space weather models and data. JOSS 7, 75, 4053, https://doi.org/10.21105/joss.04053.

• Ringuette, R., D. De Zeeuw, L. Rastaetter, A. Pembroke, O. Gerland, K. Garcia-Sage (2022). Kamodo’s model-agnostic satellite flythrough: Lowering the utilization barrier for heliophysics model outputs, Frontiers in Astronomy and Space Sciences, vol 9. http://dx.doi.org/10.3389/fspas.2022.1005977.

• Ringuette, R., L. Rastaetter, D. De Zeeuw, K. Garcia-Sage, R. Robinson, and O. Gerland (2022). Kamodo's Satellite Constellation Mission Planning Tool, poster presentation presented by L. Rastaetter at the 2022 Fall meeting of AGU, Dec 12-16, Chicago, IL, USA. https://doi.org/10.22541/essoar.167214257.73153757/v1.

• Ringuette, R., L. Rastaetter, D. De Zeeuw, A. Pembroke, and O. Gerland (2023). Simplifying model data access and utilization. Adv. Space. Res. under review.

Resources

• CCMC's Kamodo Official website - https://ccmc.gsfc.nasa.gov/tools/kamodo/
• CCMC's Kamodo Documentation page - https://nasa.github.io/Kamodo/
• Sample model outputs - https://ccmc.gsfc.nasa.gov/RoR_WWW/output_files/KAMODO_DEMO/
• Youtube tutorial channel - https://www.youtube.com/playlist?list=PLBWJQ5-pik_yBBcrpDRPM2hLluh-jreFa

The Kamodo team

Dr. Lutz Rastaetter

• ORCiD: https://orcid.org/0000-0002-7343-4147
• NASA Staff Page: https://ccmc.gsfc.nasa.gov/staff/lutz-rastaetter/

Dr. Darren De Zeeuw

• ORCiD: https://orcid.org/0000-0002-4313-5998
• NASA Staff Page: https://ccmc.gsfc.nasa.gov/staff/darren-de-zeeuw/

Dr. Katherine Garcia-Sage

• ORCiD: https://orcid.org/0000-0001-6398-8755
• NASA Staff Page: https://ccmc.gsfc.nasa.gov/staff/katherine-garcia-sage/

Open-Source License

Kamodo is an official NASA open source software package. Kamodo's official source code is hosted on github under a permissive NASA open source license: For more details, go here: https://github.com/nasa/Kamodo/blob/master/LICENSE

kamodo's Issues

Kamodo won't accept units 'N', 'S', or 'Ohm'

Inputting units 'N', 'S', 'Ohm', or any unit string with those units causes an error. I have a partial fix you can use from KamodofyPlasmaPy (https://github.com/rebeccaringuette/KamodofyPlasmaPy.git), which is copied below. This function is used in the call to kamodofy, and I have had no problems with the function representation or execution in kamodo. An example call is:

kamodo['SecondLaw'] = kamodofy(func, units=KamodoUnit('N'), arg_units={'m':KamodoUnit('kg'),'a':KamodoUnit('m/s**2')} 

The important piece of code is:

def KamodoUnit(old_unit):  #takes any string, expects it to represent units
    for item in ['S','N','Ohm']:  #force sympy to accept by decomposing the unit
        if item in old_unit: old_unit = u.Unit(old_unit).decompose().to_string('ogip')
    unit_list = [(' / ','/'),(' * ','*'),(' ','*')]  #cannot accept spaces from astropy
    for old, new in unit_list: old_unit = old_unit.replace(old,new)
    if 'radian' not in old_unit: old_unit = old_unit.replace('rad','radian')
    return old_unit

(where u is from astropy import units as u) I run all of my units through this code without issue. The hardcoding of the problematic units (lines 2 and 3 of the above section) can be changed to be

if not hasattr(parse_expr(unit_str),'subs'):  #if not in sympy dictionary, decompose
    old_unit = u.Unit(old_unit).decompose().to_string('ogip')

Interestingly, sympy already has the units I had to hardcode this for (newtons, siemens, ohms), but won't accept them through kamodo. This issue may resolve itself if this other problem can be fixed.

Data extraction along a trajectory

Data extraction along satellite trajectories is needed for many models to facilitate data/model comparisons of to probe local variability. This should be a general use tool in Kamodo that makes use of each Kamodified model's interpolator.

Requesting a DOI for Kamodo

Would like a DOI for Kamodo.

Improve README - Equations

I saw those formulas using latex, are they purposeful? I think for a clear README a structured formula would be better, you could use <img src="https://render.githubusercontent.com/render/math?math=INSERT_EQUATION_HERE"> and insert the same equation withou $$

Result for <img src="https://render.githubusercontent.com/render/math?math=\vec{f}{\left (\vec{r} \right )} [km/s] = \lambda{\left (\vec{r} \right )}"> :

interpolate_amrdata_extension_build.py errors on Windows10

The current version of interpolate_amrdata_extension_build.py in Kamodo/kamodo_ccmc/readers/OCTREE_BLOCK_GRID/ errors in the command given in the swmfgm reader:

python interpolate_amrdata_extension_build.py

With the errors below. Reverting to the script from Nov/Dec 2022 works just fine.
error.txt
interpolate_amrdata_extension_build.txt
Machine is a Windows10. Installation being performed in Python 3.10.

Errors calling kamodo from C++

In order to call kamodo from C++, I had to remove the (C) from kamodo.py and change .util to util to prevent errors. Pybind11/C++ doesn't follow implied directory structure, so I had to use

from KamodoMaster.kamodo.kamodo import Kamodo, kamodofy

instead of the current import statement, and change all import statements with .util in the line to be util (without the .). Also, the last two lines of the __init__.py file caused errors for similar reasons. Changing these to the following worked:

from kamodo import *
from util import *

Flythrough won't import

The functions in the flythrough folder won't import. Those in the readers folder import fine. Maybe it needs an init file?

SFcsv_reader leaving unclosed file

To avoid rerunning the ModelFlythrough function multiple times, I've been using the SFcsv_reader to read in the CSV file that gets output by ModelFlythrough. It seems that file is not getting closed after opening it. I get the warning:

/shared-libs/python3.7/py-core/lib/python3.7/site-packages/ipykernel_launcher.py:8: ResourceWarning: unclosed file <_io.TextIOWrapper name='/root/work/Output/ModelFlythrough/OpenGGCMFlythrough.csv' mode='r' encoding='UTF-8'>

I haven't looked at your other readers, but if this is happening with the CSV reader, it's likely happening with other readers too.

Fix deprecation warnings

Kamodo was developed under sympy==1.4. sympy==1.5.1 produces deprecation warnings

sympy/physics/units/quantities.py:84: SymPyDeprecationWarning: 

Moving method to UnitSystem class has been deprecated since SymPy 1.5.
Use unit_system.set_quantity_dimension or
nanotesla.set_global_relative_scale_factor instead. See
https://github.com/sympy/sympy/issues/17765 for more info.

  useinstead="unit_system.set_quantity_dimension or {}.set_global_relative_scale_factor".format(self),
sympy/physics/units/quantities.py:95: SymPyDeprecationWarning: 

Support Python 3.8, 3.9, 3.10, and 3.11

Python 3.7 is a very old version of Python, and as far as I can tell kamodo doesn't support more modern versions. It would be great to add support for newer versions of Python 😄

Composability checking [feature]

Currently, Kamodo does not validate composed functions. Rather, it assumes function types are compatible already. A user cannot tell if their pipeline works until evaluation. It would be great if kamodo could test functions at composition time.

Proposed Solution:
At each step of a composition, call the composed function with default inputs and generate corresponding default outputs. For example:

kamodo['f'] = kamodofy(lambda x = np.array([1,2,3]): x)
kamodo['g'] = kamodofy(lambda y = np.array([5,6,7]): y)
kamodo['h'] = 'f + g' 

kamodofy adds a data attribute to f and g by calling f and g without arguments. When 'h(x,y)' is composed from 'f' and 'g', kamodo should somehow assign its default arguments from f and g.

Features request

Hi Kamodo team,

The progress of Kamodo is tremendous. The following are features I found useful in the past in studies of outer magnetosphere data with a focus on basic plasma physics processes. It would be great if you could include some of them in your long term goals. I believe Kamodo already has many of these features, though.

Offline analysis

• user-defined list or function of coordinates of
  • sampling points, e.g., points along satellite trajectories
  • seed points as starting points for field-line tracing
• various slicing options (like Paraview): plane, sphere, arbitrary
• various extraction options: inside/out of sphere, box, below/above plane

GUI functionalities

• calculator in the GUI (e.g., computing and showing 'rho*v_x')
• combining data from more than one runs in the same view
• saving selected data that can be reloaded in the GUI
• saving the rendering scene that can be reloaded in the GUI
• Bult-in web framework for data search (based on SPASE metadata?)

requirements.txt

requirements.txt is a useful way to include package requirements.

The needed packages can be installed as:

pip install -r requirements.txt

Module kamodo_ccmc.readers.OpenGGCM.readOpenGGCM not found

Hello,
when importing the module kamodo_ccmc.readers.openggcm_gm_tocdf I get an error message (see the text below) saying that the module kamodo_ccmc.readers.OpenGGCM.readOpenGGCM does not exist.
I checked the files in the corresponding repository and the mentioned module does not exist indeed.
Could you please check whether you get the same error?
Thanks,
Simona

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [22], in <cell line: 3>()
      1 from kamodo_ccmc import readers
      2 from  kamodo_ccmc.readers import OpenGGCM
----> 3 import kamodo_ccmc.readers.openggcm_gm_tocdf

File ~/.local/lib/python3.10/site-packages/kamodo_ccmc/readers/openggcm_gm_tocdf.py:13, in <module>
     10 from glob import glob
     11 #import kamodo_ccmc.readers.OpenGGCM.read_b_grids as rbg
     12 #import kamodo_ccmc.readers.OpenGGCM.readmagfile3d as rmhd
---> 13 import kamodo_ccmc.readers.OpenGGCM.readOpenGGCM as ropgm
     15 from os.path import sep,isfile,isdir,dirname,exists
     16 from os import remove

ModuleNotFoundError: No module named 'kamodo_ccmc.readers.OpenGGCM.readOpenGGCM'

Getting empty argument list from kamodo

I am getting an empty argument list from the get_function_args function in kamodo.py (line 271). Changing the current return line to the following solved the issue:

return symbols([a for a in signature(func).parameters if a not in hidden_args])

This only occurred for wrapped functions, such as those produced by the kamodo plasmapy wrapper (https://github.com/rebeccaringuette/KamodofyPlasmaPy.git), specifically those with dimensionless argument units, dimensionless return units, and wrapped by the _Astropy wrapper. The problem did not occur for functions wrapped by the _Astropy wrapper when at least one of the argument or return units were not dimensionless.
Additionally, this change removes kamodo's dependency on the inspect.getfullargspec() function, which is being replaced by the inspect.signature function.

Example code to produce the error:

import KamodofyPlasmaPy as KPP
kamodo_list = KPP.ListPlasmaPy('plasmapy.formulary.mathematics.Fermi_integral')

You may need to change the import statement in KPP that imports Kamodo and kamodofy to reflect your kamodo installation. I don't expect this will change any behavior, as the output object is a list with the same behavior.

Several packages are missing in requrements.txt

Steps to reproduce

import kamodo_ccmc.readers.reader_utilities as RU

ModuleNotFoundError: No module named 's3fs'

Customizable plot elements

Enable a consistent way to customize plot elements, such as

• aspect ratio
• axis labels
• plot titles
• colormaps
• contouring options

Add this to conda-forge

Add this to conda-forge

Kamodo interpolation much slower than KameleonPlus Python wrapper interpolation

I made a comparison of the time for interpolation using Kamodo and the KameleonPlus Python wrapper. See the two files at

See https://github.com/GaryQ-physics/magnetosphere/tree/master/misc/kameleon_kamodo_compare

The Kameleon version is ~10x faster. Given that they both are interfacing with CCMC.so, this is somewhat unexpected.

On a side note, I also did an experiment to determine how much faster it is to loop over a grid and calling the interpolator at each iteration in a C++ program than looping over the same grid in a Python program (the Kameleon Python wrapper only allows scalars to be passed). It is about 3x faster. This is sort of on the border of where I want to bother doing anything about it. If I do, I'll share the code.

The README file in this repo has 2 bad links - [404:NotFound]

The markup version of the readme that is displayed for the main page in this repo contains the following bad links:

Status code [404:NotFound] - Link: https://github.com/nasa/Kamodo/blob/master/notebooks/images/fig2d-usage.svg

It should be "https://github.com/nasa/Kamodo/blob/master/docs/notebooks/images/fig2d-usage.svg"

"Introduction"
Status code [404:NotFound] - Link: https://github.com/nasa/Kamodo/blob/master/notebooks/Kamodo.ipynb

It should be "https://github.com/nasa/Kamodo/blob/master/docs/notebooks/Kamodo.ipynb"

(The link in the readme’s raw markdown may appear in a different relative format to the links above)

--

Extra

Theses bad links were found by a tool I recently created as part of an new experimental hobby project: https://github.com/MrCull/GitHub-Repo-ReadMe-Dead-Link-Finder
I (a human) verified that this link is broken and have manually logged this Issue (i.e. this Issue has not been created by a bot).
If this has been in any way helpful then please consider giving the above Repo a Star.

If you have any feedback on the information provided here, or on the tool itself, then please feel free to share your thoughts here, or log an “Issue” in the repo.

--

Re-check this Repo via: http://githubreadmechecker.com/Home/Search?SingleRepoUri=https%3a%2f%2fgithub.com%2fnasa%2fKamodo
Check all Repos for this GitHub account: http://githubreadmechecker.com/Home/Search?User=nasa

plotly hover not returning contour value in 3D

Probing values with hover mode in plotly works well in 1D and 2D, but in 3D no value is returned, just the XYZ position.

Add to pip

Add this to pip

Invalid path for windows.

I have cloned this repository. after cloning it give error of invalid path!

Cloning into 'Kamodo'...
remote: Enumerating objects: 2702, done.
remote: Counting objects: 100% (956/956), done.
remote: Compressing objects: 100% (56/56), done.
Receiving remote: Total 2702 (delta 933), reused 900 (delta 900), pack-reused 1746
Receiving objects: 100% (2702/2702), 41.09 MiB | 9.95 MiB/s, done.

Resolving deltas: 100% (1769/1769), done.
error: invalid path 'kamodo/cli/assets/MathJax.js?config=TeX-MML-AM_CHTML'

fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Depreciation warnings and missing dependency

I was easily able to run the example at https://ccmc.gsfc.nasa.gov/Kamodo/notebooks/kameleon-kamodo/ but encountered two issues when using Python 3.8 (I meant to use Python 3.7)

First, I needed to pip install execnet

Second,

from kamodo.readers._kameleon_kamodo import Kameleon

results in about 20 of these warnings:

lib/python3.8/site-packages/sympy/physics/units/quantities.py:77: SymPyDeprecationWarning: 
Moving method to UnitSystem class has been deprecated since SymPy 1.5.
Use unit_system.set_quantity_dimension or
nanotesla.set_global_relative_scale_factor instead. See
https://github.com/sympy/sympy/issues/17765 for more info.

Installation notes

I had to change python=3.7 to python=3.8 on a Mac M1. I think anaconda does not support arm64 before 3.8.

I also had to brew install cmake, which is not mentioned in the instructions

Docker container

Create a Dockerfile that can be used to as a basis for kamodo environments and the gui.

Move documentation from binary pdf files to plain text files

I noticed some of the (super useful!) documetation in the root directory of the repository is in .pdf files. I wanted to suggest moving from a .pdf format to another format that isn't a binary file, like MarkDown or ReStructuredText. That way it would be possible to version control the text properley and see what's changed as the documentation is updated between new Kamodo versions.

Kamodo not working on data stored in s3 buckets

Basic description of the issue:
I am attempting to interpolate the MMS trajectory through SWMF GM and OpenGGCM GM model outputs stored in s3 buckets online, but the code fails in the same way for both executions, indicating an empty list of files.

More details:
I am using OpenGGCM_GM converted files and SWMF_GM .out files in a model flythrough using the MMS trajectory. The code is adapted from the PyHC executable paper.

file_dir1 = 's3://helio-public/SWMF_GM/James_Webster_051716_1/'  # s3 location
variable_list = ['B_x','B_y', 'B_z']               # list of desired variable names
coord_sys = "GSM-car"                              # GSM and cartesian because we're using x,y,z from above 
output_dir = '/home/jovyan/efs/raringuette/Results/'  # directory where the user wants the output stored
file_name1 = 's3_SWMFGM_James_Webster_051716_1'                   # what the user wants the output files to be named
output_name1 = output_dir + file_name1 + ".csv"      # output dir plus output file name without extension
plot_coord = 'GSM'                                 # coordinate system chosen for output plots

# Convert to UTC timestamps
from datetime import datetime, timezone
sat_time = [time.replace(tzinfo=timezone.utc).timestamp() for time in epoch]  # times range should be in the model data

# Use MMS trajectory acquired from PySPEDAS/SpacePy
sat_x = pos[:, 0] / 6378
sat_y = pos[:, 1] / 6378
sat_z = pos[:, 2] / 6378

results_swmfgm = S.ModelFlythrough(model1, file_dir1, variable_list, sat_time, sat_x, sat_y, sat_z, 
                                                              coord_sys, output_name=output_name1, plot_coord=plot_coord)

On efs and local storage on my machine, this executes flawlessly. However, I get the error below when the data is stored in an s3 bucket on HelioCloud.

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[25], line 4
      2 import os.path
      3 if not os.path.isfile(output_name1):
----> 4     results_swmfgm = S.ModelFlythrough(model1, file_dir1, variable_list, sat_time, sat_x, sat_y, sat_z, 
      5                                        coord_sys, output_name=output_name1, plot_coord=plot_coord)
      6 else:  # if previously run, read in results
      7     results_swmfgm = S.O.SF_read(output_name1)

File /efs/raringuette/Kamodo/kamodo_ccmc/flythrough/SatelliteFlythrough.py:413, in ModelFlythrough(model, file_dir, variable_list, sat_time, c1, c2, c3, coord_sys, output_name, plot_coord, verbose, _print_units)
    408 coord_type, coord_grid = coord_sys.split('-')
    410 # get interpolated results
    411 # coord_type should be one of SpacePy's or AstroPy's coordinates
    412 # coord_grid is either 'sph' or 'car'
--> 413 results = U.Model_SatelliteFlythrough(model, file_dir, new_list,
    414                                       sat_time, c1, c2, c3,
    415                                       coord_type, coord_grid,
    416                                       verbose=verbose)
    418 # remove requested variables not found in the data
    419 var_list = [key for key in results.keys() if key not in
    420             ['utc_time', 'c1', 'c2', 'c3', 'net_idx']]

File /efs/raringuette/Kamodo/kamodo_ccmc/flythrough/SF_utilities.py:153, in Model_SatelliteFlythrough(model, file_dir, variable_list, sat_time, c1, c2, c3, coord_type, coord_grid, verbose)
    151 # initialize model reader and times dictionary
    152 reader = MW.Model_Reader(model)  # Kamodo gets imported here
--> 153 start_utcts, end_utcts, filedate = File_UTCTimes(model, file_dir)
    155 # cut off trajectory times not found in data in file_dir
    156 idx = where((sat_time >= start_utcts) & (sat_time <= end_utcts))[0]

File /efs/raringuette/Kamodo/kamodo_ccmc/flythrough/SF_utilities.py:207, in File_UTCTimes(model, file_dir)
    205 else:
    206     reader = MW.Model_Reader(model)
--> 207     ko = reader(file_dir, filetime=True)  # creates any preprocessed files
    208     times, filedate = ko.times, ko.filedate
    209     del ko

File /efs/raringuette/Kamodo/kamodo_ccmc/readers/swmfgm_4D.py:129, in MODEL.<locals>.MODEL.__init__(self, file_dir, variables_requested, filetime, verbose, gridded_int, printfiles, **kwargs)
    127 patterns = unique([basename(f)[:10] for f in files])
    128 # get time grid from files
--> 129 dt = sp.IdlFile(files[0]).attrs['time']
    130 if dt is not None:  # filedate given not always at midnight
    131     self.filedate = datetime.strptime(
    132         dt.isoformat()[:10], '%Y-%m-%d').replace(
    133         tzinfo=timezone.utc)

IndexError: list index out of range

Digging deeper, the initial problem is simply that glob doesn't work on s3 buckets. I was able to create a successful alternate 'glob' to handle this.

def glob(file_pattern):
    import s3fs
    s3 = s3fs.S3FileSystem(anon=False)
    s3_files = sorted(s3.glob(file_pattern))
    return ['s3://'+f for f in s3_files]
file_dir1 = 's3://helio-public/raringuette/SWMF_GM/James_Webster_051716_1/'  # helio-public
swmf_files = glob(file_dir1+'*.out')
swmf_files[::30]

Output:
['s3://helio-public/raringuette/SWMF_GM/James_Webster_051716_1/3d__var_1_e20151016-113200-000.out',
's3://helio-public/raringuette/SWMF_GM/James_Webster_051716_1/3d__var_1_e20151016-133200-000.out',
's3://helio-public/raringuette/SWMF_GM/James_Webster_051716_1/3d__var_1_e20151016-153200-000.out']

However, I can't even open the files using the methods Kamodo/spacepy uses for these files (see spacepy issue for the .out problem/solution). Attempting to use the typical Dataset call from netCDF4 on one of the OpenGGCM nc files produces the error below. First, demonstrating that the files exist...

def glob(file_pattern):
    import s3fs
    s3 = s3fs.S3FileSystem(anon=False)
    s3_files = sorted(s3.glob(file_pattern))
    return ['s3://'+f for f in s3_files]
file_dir2 = 's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/'
openggcm_files = glob(file_dir2+'*.nc')
openggcm_files[:5]

Output:
['s3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_30.nc',
's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_31.nc',
's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_32.nc',
's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_33.nc',
's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_34.nc']

Now, showing the code to open the file...

from netCDF4 import Dataset
cdf_data = Dataset(openggcm_files[0])
cdf_data

and the error produced.

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[2], line 2
      1 from netCDF4 import Dataset
----> 2 cdf_data = Dataset(openggcm_files[0])
      3 cdf_data

File src/netCDF4/_netCDF4.pyx:2449, in netCDF4._netCDF4.Dataset.__init__()
File src/netCDF4/_netCDF4.pyx:2012, in netCDF4._netCDF4._ensure_nc_success()
OSError: [Errno -68] NetCDF: I/O failure: 's3://helio-public/raringuette/OpenGGCM_GM/Yihua_Zheng_040122_1/Yihua_Zheng_040122_1.3df_2015-10-16_11_30.nc'

Note that since 'helio-public' is a public s3 bucket, there should be no problem with access rights. Or, maybe I have missed something simple here.
I suggest looking into a special build of netCDF4 to include with kamodo-ccmc to enable this (see the docs), include an alternate package chosen automatically when s3 buckets are detected in the file paths (S3netCDF4), or switch to using xarray, which works on this already. If the xarray path is used, make sure to use

pip install "xarray[io]" 

to include the required additional backends (see the docs). You might also consider adding the support for dask and acceleration in the installation. In the near future, this additional support for s3 access will be needed for all types of model outputs (binary, csv, ascii, h5, etc).

Missing Project Description

Suggested Description
"The CCMC Kamodo Analysis Suite is a NASA open-source Python package designed for functionalizing and analyzing heliophysics model outputs. It offers intuitive tools for data analysis, unit conversions, and creating publication-quality graphics. Kamodo supports a variety of models and data accessible via the HAPI interface, aiming to streamline data utilization and model output interaction for the heliophysics community."

Make plot_dict configurable

Kamodo's plots are defined in kamodo.plotting.plot_dict, shown below

plot_dict = {
	(1,)	:	{(('N','M'), ('N','M'), ('N','M')): {'name': '3d-parametric', 'func': surface}},
	('N',)	:	{
		(('N',),) : {'name': '1d-line', 'func': line_plot},
		(('N',),('N',),('N',)): {'name': '3d-line-scalar', 'func': line_plot},
		(('N',3),): {'name': '3d scatter', 'func': scatter_plot},},
	('N',2) :	{
		(('N',),) :{'name': '2d-line', 'func': line_plot},
		(('N',2),):{'name': '2d-vector', 'func': vector_plot},
	},
	('N',3) : 	{
		(('N',),) :{'name': '3d-line', 'func': line_plot},
		(('N',3),):{'name': '3d-vector', 'func': vector_plot},
	},
	('N','M'):	{
		(('N',),('M',)) :{'name': '2d-contour', 'func': contour_plot},
		(('N','M'),('N','M')):{'name': '2d-contour-skew', 'func': contour_plot},
		(('N','M'), ('N','M'), ('N','M')): {'name': '3d-parametric-scalar', 'func': surface},
		((1,),('N','M'),('N','M')):{'name': '3d-plane', 'func': plane},
		(('N','M'),(1,),('N','M')):{'name': '3d-plane', 'func': plane},
		(('N','M'),('N','M'),(1,)):{'name': '3d-plane', 'func': plane},
	},
	('N','M',1): {
		((1,),('N',),('M',)):{'name': '3d-plane', 'func': plane},
		(('N',),(1,),('M',)):{'name': '3d-plane', 'func': plane},
		(('N',),('M',),(1,)):{'name': '3d-plane', 'func': plane},
	},
}

Here the array shape of the function output is mapped to various combinations of array shapes for inputs, with the values being dictionaries pointing to the functions that actually generate plots.

It should be easier to change these defaults, for example by defining a configuration file the user can override from command line. plot_conf.yaml