asreview/asreview-insights

Tools such as plots and metrics to analyze (simulated) reviews for ASReview LAB

ASReview Insights

This official extension to ASReview LAB extends the software with tools for plotting and extracting the statistical results of several performance metrics. The extension is especially useful in combination with the simulation functionality of ASReview LAB.

Installation

ASReview Insights can be installed from PyPI:

pip install asreview-insights

After installation, check if the asreview-insights package is listed as an extension. Use the following command:

asreview --help

It should list the 'plot' subcommand and the 'metrics' subcommand.

Performance metrics

The ASReview Insights extension is useful for measuring the performance of active learning models on collections of binary labeled text. The extension can be used after performing a simulation study that involves mimicking the screening process with a specific model. As it is already known which records are labeled relevant, the simulation can automatically reenact the screening process as if a screener were using active learning. The performance of one or multiple models can be measured by different metrics and the ASReview Insights extension can plot or compute the values for such metrics from ASReview project files. O'Mara-Eves et al. (2015) provides a comprehensive overview of different metrics used in the field of actrive learning. Below we describe the metrics available in the software.

Recall

The recall is the proportion of relevant records that have been found at a certain point during the screening phase. It is sometimes also called the proportion of Relevant Record Found (RRF) after screening an X% of the total records. For example, the RRF@10 is the recall (i.e., the proportion of the total number of relevant records) at screening 10% of the total number of records available in the dataset.

Confusion matrix

The confusion matrix consist of the True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Definitions are provided in the following table retrieved at a certain recall (r%).

	Definition	Calculation
True Positives (TP)	The number of relevant records found at recall level	Relevant Records * r%
False Positives (FP)	The number of irrelevant records reviewed at recall level	Records Reviewed – TP
True Negatives (TN)	The number of irrelevant records correctly not reviewed at recall level	Irrelevant Records – FP
False Negatives (FN)	The number of relevant records not reviewed at recall level (missing relevant records)	Relevant Records – TP

Work saved over sampling

The Work Saved over Sampling (WSS) is a measure of "the work saved over and above the work saved by simple sampling for a given level of recall" (Cohen et al., 2006). It is defined as the proportion of records a screener does not have to screen compared to random reading after providing the prior knowledge used to train the first iteration of the model. The WSS is typically measured at a recall of .95 (WSS@95), reflecting the proportion of records saved by using active learning at the cost of failing to identify .05 of relevant publications.

Kusa et al. (2023) propose to normalize the WSS for class imbalance (denoted as the nWSS). Moreover, Kusa et al. showed that nWSS is equal to the True Negative Rate (TNR). The TNR is the proportion of irrelevant records that were correctly not reviewed at level of recall. The nWSS is useful to compare performance in terms of work saved across datasets and models while controlling for dataset class imbalance.

The following table provides a hypothetical dataset example:

Dataset characteristics	Example value
Total records	2000
Records Reviewed	1100
Relevant Records	100
Irrelevant Records	1900
Class imbalance	5%

With this information, the following metrics can be calculated:

Metric	Example value
TP	95
FP	1100 – 95 = 1005
TN	1900 – 1005 = 895
FN	100 – 95 = 5
TNR95%	895 / 1900 = 0.47

Extra relevant found

A variation is the Extra Relevant records Found (ERF), which is the proportion of relevant records found after correcting for the number of relevant records found via random screening (assuming a uniform distribution of relevant records).

The following plot illustrates the differences between the metrics Recall (y-axis), WSS (blue line), and ERF (red line). The dataset contains 1.000 hypothetical records with labels. The stepped line on the diagonal is the naive labeling approach (screening randomly sorted records).

Time to discovery

Both recall and WSS are sensitive to the position of the cutoff value and the distribution of the data. Moreover, the WSS makes assumptions about the acceptable recall level whereas this level might depend on the research question at hand. Therefore, Ferdinands et al. (2020) proposed two new metrics: (1) the Time to Discover a relevant record as the fraction of records needed to screen to detect this record (TD); and (2) the Average Time to Discover (ATD) as an indicator of how many records need to be screened on average to find all relevant records in the dataset. The TD metric enables you to pinpoint hard-to-find papers. The ATD, on the other hand, measures performance throughout the entire screening process, eliminating reliance on arbitrary cut-off values, and can be used to compare different models.

Basic usage

The ASReview Insights package extends ASReview LAB with two new subcommands (see asreview --help): plot and metrics. The plots and metrics are derived from an ASReview project file. The ASReview file (extension .asreview) can be exported from ASReview LAB after a simulation, or it is generated from running a simulation via the command line.

For example, an ASReview can be generated with:

asreview simulate benchmark:van_de_schoot_2017 -s sim_van_de_schoot_2017.asreview --init_seed 535

To use the most basic options of the ASReview Insights extension, run

asreview plot recall YOUR_ASREVIEW_FILE.asreview

where recall is the type of the plot, or

asreview metrics sim_van_de_schoot_2017.asreview

More options are described in the sections below. All options can be obtained via asreview plot --help or asreview metrics --help.

Plot

Plot types

recall

The recall is an important metric to study the performance of active learning algorithms in the context of information retrieval. ASReview Insights offers a straightforward command line interface to plot a "recall curve". The recall curve is the recall at any moment in the active learning process.

To plot the recall curve, you need a ASReview file (extension .asreview). To plot the recall, use this syntax (Replace YOUR_ASREVIEW_FILE.asreview by your ASReview file name.):

asreview plot recall YOUR_ASREVIEW_FILE.asreview

The following plot is the result of simulating the PTSD data via the benchmark platform (command asreview simulate benchmark:van_de_schoot_2017 -s sim_van_de_schoot_2017.asreview).

On the vertical axis, you find the recall (i.e, the proportion of the relevant records) after every labeling decision. The horizontal axis shows the proportion of total number of records in the dataset. The steeper the recall curve, the higher the performance of active learning when comparted to random screening. The recall curve can also be used to estimate stopping criteria, see the discussions in #557 and #1115.

asreview plot recall YOUR_ASREVIEW_FILE.asreview

wss

The Work Saved over Sampling (WSS) metric is a useful metric to study the performance of active learning alorithms compared with a naive (random order) approach at a given level of recall. ASReview Insights offers a straightforward command line interface to plot the WSS at any level of recall.

To plot the WSS curve, you need a ASReview file (extension .asreview). To plot the WSS, use this syntax (Replace YOUR_ASREVIEW_FILE.asreview by your ASReview file name.):

asreview plot wss YOUR_ASREVIEW_FILE.asreview

The following plot is the result of simulating the PTSD data via the benchmark platform (command asreview simulate benchmark:van_de_schoot_2017 -s sim_van_de_schoot_2017.asreview).

On the vertical axis, you find the WSS after every labeling decision. The recall is displayed on the horizontal axis. As shown in the figure, the WSS is linearly related to the recall.

erf

The Extra Relevant Records found is a derivative of the recall and presents the proportion of relevant records found after correcting for the number of relevant records found via random screening (assuming a uniform distribution of relevant records).

To plot the ERF curve, you need a ASReview file (extension .asreview). To plot the ERF, use this syntax (Replace YOUR_ASREVIEW_FILE.asreview by your ASReview file name.):

asreview plot erf YOUR_ASREVIEW_FILE.asreview

The following plot is the result of simulating the PTSD data via the benchmark platform (command asreview simulate benchmark:van_de_schoot_2017 -s sim_van_de_schoot_2017.asreview).

On the vertical axis, you find the ERF after every labeling decision. The horizontal axis shows the proportion of total number of records in the dataset. The steep increase of the ERF in the beginning of the process is related to the steep recall curve.

Plotting CLI

Optional arguments for the command line are --priors to include prior knowledge, --x_absolute and --y_absolute to use absolute axes.

See asreview plot -h for all command line arguments.

Plotting multiple files

It is possible to show the curves of multiple files in one plot. Use this syntax (replace YOUR_ASREVIEW_FILE_1 and YOUR_ASREVIEW_FILE_2 by the asreview_files that you want to include in the plot):

asreview plot recall YOUR_ASREVIEW_FILE_1.asreview YOUR_ASREVIEW_FILE_2.asreview

Plotting API

To make use of the more advanced features, you can make use of the Python API. The advantage is that you can tweak every single element of the plot in the way you like. The following examples show how the Python API can be used. They make use of matplotlib extensively. See the Introduction to Matplotlib for examples on using the API.

The following example show how to plot the recall with the API and save the result. The plot is saved using the matplotlib API.

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_recall

with open_state("example.asreview") as s:

    fig, ax = plt.subplots()

    plot_recall(ax, s)

    fig.savefig("example.png")

Other options are plot_wss and plot_erf.

Example: Customize plot

It's straightforward to customize the plots if you are familiar with matplotlib. The following example shows how to update the title of the plot.

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_wss

with open_state("example.asreview") as s:

    fig, ax = plt.subplots()
    plot_wss(ax, s)

    plt.title("WSS with custom title")

    fig.savefig("example_custom_title.png")

Example: Prior knowledge

It's possible to include prior knowledge in your plot. By default, prior knowledge is excluded from the plot.

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_wss

with open_state("example.asreview") as s:

    fig, ax = plt.subplots()
    plot_wss(ax, s, priors=True)

Example: Relative versus absolute axes

By default, all axes in ASReview Insights are relative. The API can be used to change this behavior. The arguments are identical for each plot function.

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_wss

with open_state("example.asreview") as s:

    fig, ax = plt.subplots()
    plot_wss(ax, s, x_absolute=True, y_absolute=True)

    fig.savefig("example_absolute_axis.png")

Example: Legend for multiple curves in one plot

If you have multiple curves in one plot, you can customize the legend:

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_recall


fig, ax = plt.subplots()

with open_state("tests/asreview_files/sim_van_de_schoot_2017_1.asreview") as s1:
    with open_state("tests/asreview_files/"
                    "sim_van_de_schoot_2017_logistic.asreview") as s2:
        plot_recall(ax,
                    [s1, s2],
                    legend_values=["Naive Bayes", "Logistic"],
                    legend_kwargs={'loc': 'lower center'})

fig.savefig("docs/example_multiple_lines.png")

metrics

The metrics subcommand in ASReview Insights can be used to compute metrics at given values. The easiest way to compute metrics for a ASReview project file is with the following command on the command line:

asreview metrics sim_van_de_schoot_2017.asreview

which results in

    "asreviewVersion": "1.0",
    "apiVersion": "1.0",
    "data": {
        "items": [
            {
                "id": "recall",
                "title": "Recall",
                "value": [
                    [
                        0.1,
                        1.0
                    ],
                    [
                        0.25,
                        1.0
                    ],
                    [
                        0.5,
                        1.0
                    ],
                    [
                        0.75,
                        1.0
                    ],
                    [
                        0.9,
                        1.0
                    ]
                ]
            },
            {
                "id": "wss",
                "title": "Work Saved over Sampling",
                "value": [
                    [
                        0.95,
                        0.8913851624373686
                    ]
                ]
            },
            {
                "id": "erf",
                "title": "Extra Relevant record Found",
                "value": [
                    [
                        0.1,
                        0.9047619047619048
                    ]
                ]
            },
            {
                "id": "atd",
                "title": "Average time to discovery",
                "value": 101.71428571428571
            },
            {
                "id": "td",
                "title": "Time to discovery",
                "value": [
                    [
                        3898,
                        22
                    ],
                    [
                        284,
                        23
                    ],
                    [
                        592,
                        25
                    ],
                    ...
                    [
                        2382,
                        184
                    ],
                    [
                        5479,
                        224
                    ],
                    [
                        3316,
                        575
                    ]
                ]
            },
            {
                "id": "tp",
                "title": "True Positives",
                "value": [
                    [
                        0.95,
                        39
                    ],
                    [
                        1.0,
                        42
                    ]
                ]
            },
            {
                "id": "fp",
                "title": "False Positives",
                "value": [
                    [
                        0.95,
                        122
                    ],
                    [
                        1.0,
                        517
                    ]
                ]
            },
            {
                "id": "tn",
                "title": "True Negatives",
                "value": [
                    [
                        0.95,
                        6023
                    ],
                    [
                        1.0,
                        5628
                    ]
                ]
            },
            {
                "id": "fn",
                "title": "False Negatives",
                "value": [
                    [
                        0.95,
                        3
                    ],
                    [
                        1.0,
                        0
                    ]
                ]
            },
            {
                "id": "tnr",
                "title": "True Negative Rate (Specificity)",
                "value": [
                    [
                        0.95,
                        0.980146
                    ],
                    [
                        1.0,
                        0.915867
                    ]
                ]
            }
        ]
    }
}

Each available item has two values. The first value is the value at which the metric is computed. In the plots above, this is the x-axis. The second value is the results of the metric. Some metrics are computed for multiple values.

Metric	Description pos. 1	Description pos. 2	Default
recall	Labels	Recall	0.1, 0.25, 0.5, 0.75, 0.9
wss	Recall	Work Saved over Sampling at recall	0.95
erf	Labels	ERF	0.10
atd	Average time to discovery (in label actions)	-	-
td	Row number (starting at 0)	Number of records labeled	-
cm	Recall	Confusion matrix values at recall	0.95, 1

Override default values

It is possible to override the default values of asreview metrics. See asreview metrics -h for more information or see the example below.

asreview metrics sim_van_de_schoot_2017.asreview --wss 0.9 0.95

{
    "asreviewVersion": "1.0",
    "apiVersion": "1.0",
    "data": {
        "items": [
            {
                "id": "recall",
                "title": "Recall",
                "value": [
                    [
                        0.1,
                        1.0
                    ],
                    [
                        0.25,
                        1.0
                    ],
                    [
                        0.5,
                        1.0
                    ],
                    [
                        0.75,
                        1.0
                    ],
                    [
                        0.9,
                        1.0
                    ]
                ]
            },
            {
                "id": "wss",
                "title": "Work Saved over Sampling",
                "value": [
                    [
                        0.9,
                        0.8474220139001132
                    ],
                    [
                        0.95,
                        0.8913851624373686
                    ]
                ]
            },
            {
                "id": "erf",
                "title": "Extra Relevant record Found",
                "value": [
                    [
                        0.1,
                        0.9047619047619048
                    ]
                ]
            },
            {
                "id": "atd",
                "title": "Average time to discovery",
                "value": 101.71428571428571
            },
            {
                "id": "td",
                "title": "Time to discovery",
                "value": [
                    [
                        3898,
                        22
                    ],
                    [
                        284,
                        23
                    ],
                    [
                        592,
                        25
                    ],
                    ...
                    [
                        2382,
                        184
                    ],
                    [
                        5479,
                        224
                    ],
                    [
                        3316,
                        575
                    ]
                ]
            }
        ]
    }
}

Save metrics to file

Metrics can be saved to a file in the JSON format. Use the flag -o or --output.

asreview metrics sim_van_de_schoot_2017.asreview -o my_file.json

Metrics CLI

Optional arguments for the command line are --priors to include prior knowledge, --x_absolute and --y_absolute to use absolute axes.

See asreview metrics -h for all command line arguments.

Metrics API

Metrics are easily accesible with the ASReview Insights API.

Compute the recall after reading half of the dataset.

from asreview import open_state
from asreviewcontrib.insights.metrics import recall

with open_state("example.asreview") as s:

    print(recall(s, 0.5))

Other metrics are available like wss and erf.

Example: Prior knowledge

It's possible to include prior knowledge to your metric. By default, prior knowledge is excluded from the metric.

from asreview import open_state
from asreviewcontrib.insights.metrics import recall

with open_state("example.asreview") as s:

    print(recall(s, 0.5, priors=True))

License

This extension is published under the MIT license.

Contact

This extension is part of the ASReview project (asreview.ai). It is maintained by the maintainers of ASReview LAB. See ASReview LAB for contact information and more resources.

asreview-insights's Issues

Origin of the plot

Currently the recall plot starts at zero relevant and zero irrelevant records. Sometime you want to start the recall curve at 0,x where x is number of relevant inclusions. Would it be possible to add this option?
I am not sure what the default should be, @GerbrichFerdinands ?

Unknown plot type should raise error?

If the plot type isn't available, a warning is raised. Shouldn't this a ValueError?

jonathans-mbp:FMS_use_case jonathan$ asreview plot simulation/result_run_1.h5 --type 'inclusions'
WARNING:root:Plotting type "inclusions unknown."

It might be a good idea to have something like: "Allowed values are: [LIST OF VALUES]"

Replace WSS@95 with Recall@95

Can we replace the label WSS@95 with the Recall@95? I assume the vertical line is actually the recall at 95% and not the WSS, right? I would prefer it to be the Recall at 95%.

Add option to output the record order of a simulation

I would like the order of all records during a simulation, including the irrelevant records.

Currently I extract this information with the following code:

with open_state("output.asreview") as state:
    record_ids = state.get_order_of_labeling()

record_ids.to_csv("record_order.csv")

As output from the insights package I would like a csv file with the collumns record_id and order.

Missing module asreview.analysis import Analysis

Hi,

I am using the default version pulled from pip: asreview-visualization-0.4.1

This might be a silly mistake in my environment. I am getting missing module error on asreview.analysis when I tried to run asreview plot, here is the stacktrace:

(octopus) TBook:.asreview Zhao$ asreview plot asreview_init_data_all_shuffled_20230214_test_simulation.asreview 
Traceback (most recent call last):
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/asreview/__main__.py", line 78, in main
    entry.load()().execute(sys.argv[2:])
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2458, in load
    return self.resolve()
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2464, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/asreviewcontrib/visualization/__init__.py", line 15, in <module>
    from asreviewcontrib.visualization.plot import Plot  # noqa
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/asreviewcontrib/visualization/plot.py", line 18, in <module>
    from asreview.analysis import Analysis
ModuleNotFoundError: No module named 'asreview.analysis'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/bin/asreview", line 8, in <module>
    sys.exit(main())
  File "/Users/zhao/Development/tools/miniconda3/envs/octopus/lib/python3.10/site-packages/asreview/__main__.py", line 80, in main
    raise ValueError(
ValueError: Plugin with entry point plot could not be loaded.

I tried to upgrade all my asreview packages to the latest but still encounter similar errors. I can't find the asreview.analysis in my python local repository either.

Here are the versions of my asreview and related packages:

Automated Systematic Review (ASReview).

positional arguments:
  subcommand     The subcommand to launch. Available commands:
                 
                 lab [asreview-1.1]
                     The ASReview LAB webapp.
                 
                 simulate [asreview-1.1]
                     Simulate the performance of ASReview.
                 
                 metrics [asreview-insights-1.1.2]
                     Metrics entry point.
                 
                 data [asreview-datatools-1.1.1]
                     Home of all data tools for ASReview.
                 
                 algorithms [asreview-1.1]
                     Available active learning algorithms for ASReview.
                 
                 state-inspect [asreview-1.1]
                     Inspect ASReview LAB review progress.
                 
                 makita [asreview-makita-0.6.2]
                     Makita functionality for ASReview datasets.

Can you please let me know if I am missing certain package or need to update my local packages to certain versions to make this work?

Thanks.

Plotting multiple files

When trying to plot multiple files in one plot, the following error presents itself:
ValueError: Plotting multiple project files via the CLI is not supported yet.

I am unsure why and how to fix this.

Add option to output model settings in metric output

It would be nice to extend the asreview metrics command with model settings. Ideally with a flag that is disabled by default.

Example:

asreview metrics my_file.asreview --include-metrics

 "asreviewVersion": "1.0",
    "apiVersion": "1.0",
    "model": {
          ...
    }
    "data": {
        "items": [
            {
                "id": "recall",
                "title": "Recall",
                "value": [
                    [
                        0.1,
                        1.0
                    ],
                    [
                        0.25,
                        1.0
                    ],
                    [
                        0.5,
                        1.0
                    ],
                    [
                        0.75,
                        1.0
                    ],
                    [
                        0.9,
                        1.0
                    ]
                ]
            },
            {
                "id": "wss",
                "title": "Work Saved over Sampling",
                "value": [
                    [
                        0.95,
                        0.8913851624373686
                    ]
                ]
            },
            {
                "id": "erf",
                "title": "Extra Relevant record Found",
                "value": [
                    [
                        0.1,
                        0.9047619047619048
                    ]
                ]
            },
            {
                "id": "atd",
                "title": "Average time to discovery",
                "value": 101.71428571428571
            },
            {
                "id": "td",
                "title": "Time to discovery",
                "value": [
                    [
                        3898,
                        22
                    ],
                    [
                        284,
                        23
                    ],
                    [
                        592,
                        25
                    ],
                    ...
                    [
                        2382,
                        184
                    ],
                    [
                        5479,
                        224
                    ],
                    [
                        3316,
                        575
                    ]
                ]
            }
        ]
    }
}

Incorrect plots when plotting partly labeled state files

ASReview users can plot a state file from the ASReview LAB GUI. However, this results in incorrect plots.

This is the file:
result.json.zip

Frontend shows the following:

The 5% reviewed is correct. There are about 6k items in the dataset. So the issue is in the y axis.

Relates to asreview/asreview#392. This PR makes it possible to visualize an ASREVIEW project file.

Add CSV file with TD values

The Time to Discovery values are presented as follows in the output:

{
                "id": "td",
                "title": "Time to discovery",
                "value": [
                    [
                        57,
                        18
                    ],
                    [
                        159,
                        37
                    ],
                    [
                        0,
                        55
                    ],
                    [
                        160,
                        72
                    ]
                ]
            }

It might cause confusion in identifying the records in the dataset. Record 57 is the row number starting at zero, but in the CSV file row number includes the column names, and software like Excel starts numbering at 1. A solution would be to create a separate CSV file containing the relevant records only including a column named TD. See for an example the file titled time_to_discovery.csv in the project by Ferdinands et al..

Plotting multiple statefiles dynamically.

The example for the new multiple plots code is as follows:

import matplotlib.pyplot as plt

from asreview import open_state
from asreviewcontrib.insights.plot import plot_recall


fig, ax = plt.subplots()

with open_state("tests/asreview_files/sim_van_de_schoot_2017_1.asreview") as s1:
    with open_state("tests/asreview_files/"
                    "sim_van_de_schoot_2017_logistic.asreview") as s2:
        plot_recall(ax,
                    [s1, s2],
                    legend_values=["Naive Bayes", "Logistic"],
                    legend_kwargs={'loc': 'lower center'})

fig.savefig("docs/example_multiple_lines.png")

However, how would one do this dynamically? open_state() method is returning a '_GeneratorContextManager' object, which implies that it is intended to be used as a context manager with the with statement. However, I can't dynamically open a cascading with statement, and I can't store the state for later using:

  for state_file in states:
      state = open_state(state_file)
      state_objs.append(state)

  plot_recall(ax, state_objs, **plot_kwargs)

Any advise for this implementation? I think I cannot manually manage the lifetime of the state objects and they must be used within a with block, and thus I am forced to call plot_recall separately.

remove grid lines

It would be helpful to remove the grid lines and replace these with only a vertical line at 10% of the total number of records (=RRF@10%) and a horizontal line at 95% of the total number of relevant records.

Update to the latest version on PyPI

Hi !

Would it be possible to update the release of asreview-insights over PyPI to version 1.2 ?
In that way, we would be able to install the latest version via pip without having to use the git repo address directly.

Release v0.4.1 breaks legendas

Using the following code with the API from version 0.4 gives no errors when creating a plot with a legenda:

with Plot.from_paths(files[:1]) as plot:
  inc_plot = plot.new("inclusion", result_format="number")
  inc_plot.set_grid()
  inc_plot.set_xlim(0, 4000)
  inc_plot.set_legend()
  inc_plot.add_random(add_text=False)

  all_files = all(plot.is_file.values())
  for key in list(plot.analyses):
      if all_files or not plot.is_file[key]:
          inc_plot.add_wss(
              key, 95, add_value=False, add_text=False)

  inc_plot.fig.set_size_inches(10, 5, forward=True)

  inc_plot.show()
  inc_plot.save("simulation_results.png")

Whereas running the same code with version 0.4.1:

The error message isn't very informative.

..\Python\Python39\lib\site-packages\asreviewcontrib\visualization\plot_base.py:32: UserWarning: Legend does not support [<matplotlib.lines.Line2D object at 0x000001F563215400>] instances.
A proxy artist may be used instead.
See: https://matplotlib.org/users/legend_guide.html#creating-artists-specifically-for-adding-to-the-legend-aka-proxy-artists
  self.ax.legend(self.legend_plt, self.legend_name, loc=loc)