llnl/thicket

Thicket

Thicket

A Python-based toolkit for analyzing ensemble performance data. You can find detailed documentation, along with tutorials of Thicket in the ReadtheDocs.

Installation

To use thicket, install it with pip:

$ pip install llnl-thicket

Or, if you want to develop with this repo directly, run the install script from the root directory, which will build the package and add the cloned directory to your PYTHONPATH:

$ source install.sh

Contact Us

You can direct any feature requests or questions to the Lawrence Livermore National Lab's Thicket development team by emailing either Stephanie Brink ([email protected]) or Olga Pearce ([email protected]).

Contributing

To contribute to Thicket, please open a pull request to the develop branch. Your pull request must pass Thicket's unit tests, and must be PEP 8 compliant. Please open issues for questions, feature requests, or bug reports.

Authors and citations

Many thanks to Thicket's contributors.

Thicket was created by Olga Pearce and Stephanie Brink.

To cite Thicket, please use the following citation:

• Stephanie Brink, Michael McKinsey, David Boehme, Connor Scully-Allison, Ian Lumsden, Daryl Hawkins, Treece Burgess, Vanessa Lama, Jakob Lüttgau, Katherine E. Isaacs, Michela Taufer, and Olga Pearce. 2023. Thicket: Seeing the Performance Experiment Forest for the Individual Run Trees. In the 32nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC'23), August 2023, Pages 281–293. doi.org/10.1145/3588195.3592989.

On GitHub, you can copy this citation in APA or BibTeX format via the "Cite this repository" button. Or, see CITATION.cff for the raw BibTeX.

License

Thicket is distributed under the terms of the MIT license.

All contributions must be made under the MIT license. Copyrights in the Thicket project are retained by contributors. No copyright assignment is required to contribute to Thicket.

See LICENSE and NOTICE for details.

SPDX-License-Identifier: MIT

LLNL-CODE-834749

thicket's Issues

Tree Bug

#76 introduced a bug into tree where it cannot find the columns in the dataframe, even if they exist. This error occurs for any column, and I have not found a configuration that works. The error looks like the following:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[/usr/workspace/ritter5/thicket/graphs.ipynb](https://vscode-remote+ssh-002dremote-002boslic-002ellnl-002egov.vscode-resource.vscode-cdn.net/usr/workspace/ritter5/thicket/graphs.ipynb) Cell 10 in
----> [1](vscode-notebook-cell://ssh-remote%2Boslic.llnl.gov/usr/workspace/ritter5/thicket/graphs.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) print(tk_full.tree(annotation_column="Avg time/rank"))
File [/usr/WS2/ritter5/thicket/thicket/thicket.py:735](https://vscode-remote+ssh-002dremote-002boslic-002ellnl-002egov.vscode-resource.vscode-cdn.net/usr/WS2/ritter5/thicket/thicket/thicket.py:735), in Thicket.tree(self, metric_column, annotation_column, precision, name_column, expand_name, context_column, rank, thread, depth, highlight_name, colormap, invert_colormap, colormap_annotations, render_header, min_value, max_value)
    732 elif sys.version_info.major == 3:
    733     unicode = True
--> 735 return ThicketRenderer(unicode=unicode, color=color).render(
    736     self.graph.roots,
    737     self.statsframe.dataframe,
    738     metric_column=metric_column,
    739     annotation_column=annotation_column,
    740     precision=precision,
    741     name_column=name_column,
    742     expand_name=expand_name,
    743     context_column=context_column,
    744     rank=rank,
    745     thread=thread,
    746     depth=depth,
    747     highlight_name=highlight_name,
    748     colormap=colormap,
    749     invert_colormap=invert_colormap,
    750     colormap_annotations=colormap_annotations,
    751     render_header=render_header,
...
    108             self.second_metric
    109         )
    110     )
KeyError: 'metric_column=time does not exist in the dataframe, please select a valid column. See a list of the available metrics with GraphFrame.show_metric_columns().'

Common commands to run when debugging installation issues

which python3
which pip
pip list | grep  <package_name>
pip install -e <package_dir> --config-settings editable_mode=strict

Intersection Broken When _fill_perfdata=False

Found in #182. Current intersection relies on the Nones inserted by _fill_perfdata to compute the intersection after the union has already been applied

Use metadata_column_to_perfdata in ExtraP

Currently Extra-P uses a manual lookup in the metadataframe to get the "p" (x-values) for its modeling. We could optimize this by calling the metadata_column_to_perfdata function before to have all of the data we need in the ensembleframe.

Adds CI runs that exclude optional dependencies

Suggested by @pearce8

We've had issues in the past with users not having optional dependencies. To account for that, we should add to our CI matrix so that we can run under different sets of extras. The extras we can consider are:

• llnl-thicket
• llnl-thicket[extrap]
• llnl-thicket[plotting]
• llnl-thicket[vis]
• llnl-thicket[extrap,plotting]
• llnl-thicket[extrap,vis]
• llnl-thicket[plotting,vis]
• llnl-thicket[extrap,plotting,vis]

Split test fixtures in test_filter_metadata.py

Minor issue that cropped up in the review of #164.

In test_filter_metadata.py, there are the following 3 functions:

• filter_one_column
• filter_multiple_and
• filter_multiple_or

These functions are essentially test fixtures, but, instead of being treated as fixtures, they are just functions that are combined into one fixture called test_filter_metadata. As a result of having this "super-fixture", we can't easily detect which of the 3 functions above actually failed.

This "super-fixture" should be split into 3 different fixtures to test these functions independently and simplify failure tracking.

Additionally, per the discussion in #164, these functions are (for some reason) also used in:

• thicket/tests/test_concat_thickets.py
• thicket/tests/test_groupby.py

As a result, these functions should be moved into some "testing utils" module. Then, test_filter_metadata.py should import the functions and use them in the 3 new fixtures, and the other two modules (listed above) should import the functions and use them as they currently are.

filter_stats does not correctly update index in statsframe

Thicket.filter_stats will filter the Thicket object based on a predicate applied to the DataFrame in Thicket.statsframe. Currently, this function uses hatchet.GraphFrame.squash to propagate the filter to other components of the Thicket object (e.g., performance data and graph).

However, because Hatchet has no knowledge of the statsframe, hatchet.GraphFrame.squash will not properly update the index of Thicket.statsframe.dataframe with Node objects from the newly created graph.

As a result, the output of Thicket.filter_stats will look alright (i.e., the graph will contain the correct nodes and both the performance and stats dataframes will look correct), but functions that depend on the index of the statsframe (e.g., Thicket.tree) will fail with IndexErrors.

Bug: unhashable list in get_unique_metadata

For columns where the values are lists, we cannot get a list of the unique values.

TypeError: unhashable type: 'list'

Running Thicket.concat_thickets with calltree="intersection" produces empty query

When trying to merge 4 Thickets with RAJAPerf data with concat_thickets, Thicket errors with EmptyQuery because the query used in Thicket.intersection would produce an empty Thicket object. After reviewing the individual Thickets by hand, I found that their call trees are identical, so there is a bug in Thicket.intersection causing this error.

Bug: typeerror in verify_thicket_structures

Typeerror occurs in verify_thicket_structures when a column is specified that does not exist.

File /usr/gapps/spot/dev/thicket-playground-dev/thicket/utils.py:58, in verify_thicket_structures(thicket_component, columns, index)
     49     raise RuntimeError(
     50         "\n Missing column(s): "
     51         + missing_columns
   (...)
     54         + " required for the function"
     55     )
     56 elif not column_result and index_result:
     57     raise RuntimeError(
---> 58         "\n Missing column(s): " + missing_columns + " required for the function"
     59     )
     60 elif column_result and not index_result:
     61     raise RuntimeError(
     62         "\n Missing index level(s): " + missing_index + " required for the function"
     63     )

TypeError: can only concatenate str (not "list") to str

Bug: loading PCP code looks for NPM

%load_ext thicket.vis.visualizations

     41 if not os.path.isfile(pkg_lock_file):
     42     if not check_npm():
---> 43         raise FileNotFoundError(_filenotfound_errmsg)
     44     else:
     45         npm_build(curr_dir)

FileNotFoundError: 
Cannot find NPM!
This is required to use thicket.vis!
Please install NPM and try again to import thicket.vis!

Editable installs break due to setuptools

As reported here, there is a bug in newer versions of setuptools that can caused editable installs to break due to changes that setuptools made to support PEP 600.

There are two ways to address this. In the short term, users can add --config-settings editable_mode=strict to their pip install -e command. This forces setuptools to use the older style of editable installs. Longer term, the best way to fix this for Hatchet and Thicket is to move away from setuptools.

RADIUSS Tutorial Reflection

• Update rendered tutorial notebooks in thicket
• Update tutorials page with materials
• Rendered basic tutorial is using new RAJAPerf data, this shouldn't be the case yet because we want to update all
• Use display instead of print for dataframes, will all columns of dataframe render?

Extra-p optional vs required dependency

@ilumsden @michaelmckinsey1 @meaparvitas

Check for non-unique multi-index

From ensemble.py:

except ValueError as e:
    estr = str(e)
    if estr == "cannot handle a non-unique multi-index!":
        warnings.warn(
            "Non-unique multi-index for DataFrame in _fill_perfdata. Cannot Fill missing rows.",
            RuntimeWarning,
        )
    else:
        raise

As stated from a previous comment, instead of checking the error string, we should check if the index is non-unique.

Need for a Statsframe Clear operation exposed to the user

Clearing the statsframe of all operations is something i often find myself manually repeatedly doing. I do this this by outright deleting specific columns through pandas operations.

With that in mind, i think it would be of much utility to the user if a Thicket function existed that would allow the user to clear the statsframe through a single call.

Docs: update images

• fix fig 1 in user guide
• user guide call tree and performance data to match

Implicit dependency on seaborn when extra `plotting` is disabled

Without having seaborn installed in the (virtual) environment, import thicket fails as it tries to import seaborn:

  File "[...]/benchmarking.py", line 11, in <module>
    import thicket as tt
  File "[...]/lib/python3.11/site-packages/thicket/__init__.py", line 23, in <module>
    from .stats.display_boxplot import display_boxplot
  File "[...]/lib/python3.11/site-packages/thicket/stats/display_boxplot.py", line 7, in <module>
    import seaborn as sns
ModuleNotFoundError: No module named 'seaborn'

Workaround: Just install seaborn manually or use llnl-thicket[plotting].

Related to #137.

CI fails because it can't fetch NodeJS

Our CI is currently failing because the setup-node action fails to fetch NodeJS. This is due to a recently discovered bug in the setup-node action: actions/setup-node#873

There is currently no known fix for this bug, but I'll update this issue when one is found.

Add Jinja2 dependency to docs and pin to 3.0.3

Our docs depend on nbsphinx, which in turn depends on nbconvert. Unfortunately, there is an issue related to nbconvert that causes our docs to fail if a version of Jinja2 greater than 3.0.3 is installed.

So, we need to add a dependency on Jinja2 3.0.3 to docs/requirements.txt.

Columnar Join and Sync Nodes Consequence

The columnar join function requires Thicket._sync_nodes_frame in order to match up the nodes in the dataframe after union operations have been performed in the graph. The function makes an assumption if a df node's frame matches a graph node's frame to set the df node to be the graph node, making their nid's equal. This results in some nodes being mapped to the same nid, if it's frame has already existed in the df.

I ran into this issue when traversing a graphs children and then trying to match those nodes in the dataframe. The problem appears for nodes like the one above because the same node frame appears for every kernel node that exists (obviously the kernel nodes have different frames, "apps_vol" vs "apps_hydro", etc).

Use MultiIndex for ExtraP Functions

The componentize_statsframe and _add_extrap_statistics extrap model functions both add a lot of columns to the statsframe.

Statsframe

avg#inclusive#sum#time.duration_extrap-model	avg#inclusive#sum#time.duration_RSS_extrap-model	avg#inclusive#sum#time.duration_rRSS_extrap-model	...
cell	cell	cell	cell
cell	cell	cell	cell

Currently, we are appending strings to the column name to distinguish them between each other. This becomes very messy fast and is not user friendly. I propose pulling out the common strings between the columns, like avg#inclusive#sum#time.duration, into a higher-level column index, and let the different strings distinguish each column, like model, RSS, rRSS.

And what if our statsframe already had other statistics and/or models in it? Our problem would only become worse. So I believe we can leverage multi-indexing to help us organize the statsframe.

No unit tests for Statsframe

While going over some stuff, @michaelmckinsey1 and I discovered that there are no unit tests for the statistics functions.

Unexpected output th.stats.std

There is an issue with the th.stats.std function where single profiled thicket results in Nan's value due to pandas use of sample standard deviation instead of population standard deviation.

A way to get around this issue is to use lambda x: np.std(x) in the aggregate function of th.stats.std.

i.e df = thicket.dataframe[columns].reset_index().groupby("node").agg(lambda x: np.std(x))

Pandas 1.3.5 Shallow Copy Discrepancy

Problem

When running the unit test test_copy:test_copy() on GitHub for python3.7 and pandas==1.3.5 the following error will occur

        # Shallow copy of data
        node = other.dataframe.index.get_level_values("node")[0]
        profile = other.dataframe.index.get_level_values("profile")[0]
        other.dataframe.loc[(node, profile), "nid"] = -1
>       assert (
            other.dataframe.loc[(node, profile), "nid"]
            == self.dataframe.loc[(node, profile), "nid"]
        )
E       assert -1.0 == 26.0

This error does not happen when running locally, tested with python3.7.2 and pandas==1.3.5.

Current Fix

Our current fix was to change the pandas version on GitHub to pandas==1.2.5 and the error does not occur. For reference the error does not occur for pandas==1.4 and pandas==2.0, so our suspicion is the error only occurs for pandas==1.3.5.

This is somewhat known by pandas and they fixed the inconsistency in pandas==1.5. See:

• pandas-dev/pandas#45743
• pandas-dev/pandas#45333

Add a Pipfile to enable pipenv-based environments

To help users install Thicket and to standardize development environments, we can add a Pipfile to GitHub with both development and non-development dependencies. With this, we could use pipenv to have a standardized way to create environments for both end users and developers.

add more_itertools to requirements.txt

File thicket/thicket/helpers.py:6, in <module>
      1 # Copyright 2022 Lawrence Livermore National Security, LLC and other
      2 # Thicket Project Developers. See the top-level LICENSE file for details.
      3 #
      4 # SPDX-License-Identifier: MIT
----> 6 from more_itertools import powerset
      7 import pandas as pd
     10 def _are_synced(gh, df):

ModuleNotFoundError: No module named 'more_itertools'

Filtering Partial DataFrames in MultiIndex Column Thickets

If we have a MultiIndex Column Thicket with a performance table:

	A		B
	time	memory	time	memory
1	5s	10b	7s	6b
2	3s	8b	4s	9b

its profiles would be [(1, "A"), (2, "A"), (1, "B"), (2, "B")]. Currently, filtering applies to the profile and profile_mapping objects, but we do not filter the dataframe or metadata in this case unless the whole row would be removed (e.g. removing (1, "A") and (1, "B")). If we only remove (1, "A"), this will have no effect on the dataframe.

Should removing (1, "A") result in

	A		B
	time	memory	time	memory
1	NaN	NaN	7s	6b
2	3s	8b	4s	9b

?

If yes, then we need to develop functionality for this feature.