karim-53/compare-xai

A Unified Approach to Evaluate and Compare Explainable AI methods

Hi there 👋

My real time stats:

compare-xai's Issues

create a stress test with a lot of correlated features

idea of Richard

Getting HitHub Actions Updated Automatically

The awesome tool dependabot lets you scan your used GitHub Marketplace Actions and lets you know if they are in need of an update.

The update is proposed via a pull request, which can be accepted or declined, it will itself take care of deleting pull requests if these become irrelevant.

You specify the configuration in the file: .github/dependabot.yml in your repository using this action - actually it scans all your actions.

# Basic dependabot.yml file
# REF: https://docs.github.com/en/code-security/supply-chain-security/keeping-your-actions-up-to-date-with-dependabot

version: 2
updates:
  # Enable version updates for Actions
  - package-ecosystem: "github-actions"
    # Look for `.github/workflows` in the `root` directory
    directory: "/"
    # Check for updates once a week
    schedule:
      interval: "weekly"

Todo

• Create a PR containing one new file: .github/dependabot.yml
• change the interval to a monthly update
• make sure the action is working correctly on your fork

Add the list of explainers in the left menu

To make the web site more clear it would be nice to have the list of explainers in the left menu

Fix the Vulture (dead code) GitHub Action

Fix the following file https://github.com/Karim-53/Compare-xAI/blob/main/.github/workflows/vulture.yml

you can learn more about the bug, for example here: https://github.com/Karim-53/Compare-xAI/actions/runs/3860871755/jobs/6581483167

this link could help https://github.com/anaynayak/python-vulture-action

Fix the type annotation Action (Github Action)

Fix the following file https://github.com/Karim-53/Compare-xAI/blob/main/.github/workflows/type_annotation.yml

you can learn more about the bug for example here: https://github.com/Karim-53/Compare-xAI/actions/runs/3860872224/jobs/6581483838

this link could help https://github.com/marketplace/actions/typilus-suggest-python-type-annotations

extend fooling_perturbation_alg Test to interation (5)

    extend  fooling_perturbation_alg   fragility        lakkaraju2020fool to interation

todo refer to paper video code

does not look obvious todo

"input correlation" Test (5)

• fix "input correlation" test for xAI outputing Feature Importance
• then generalize it to interaction

Add Explainer: Joint Shapley Values 2022 (4)

source code https://github.com/harris-chris/joint-shapley-values/blob/main/joint_shapley/joint_shapley_impl.py

https://openreview.net/pdf?id=vcUmUvQCloe
https://github.com/harris-chris/joint-shapley-values

if the code is not available, then no need

Add Stress test : sparse features

Normal initial state

linear model
BMI dataset with

Expected: local explanation of a datapoint: BMI feature has a larger variation than gender feature.

Issue

Create an explicand x such that every feature value xi is unique, i.e., it does not occur elsewhere in the training data. (local explanation of a datapoint that is unique in all its xi features)
(A practical implication of Example 3.2 is that the attributions would be very sensitive to noise in the data.)

trigger the issue

If we add a tiny amount of noise, and recompute attributions, then all the features (including BMI and Gender)

The wrong result

Therefore all the variables get equal attributions, even if the function is not symmetric in the variables!

source: https://arxiv.org/pdf/1908.08474.pdf

Add a new Explainer / Test

Hi

To add a new Explainer / Test please follow the instructions in https://github.com/Karim-53/Compare-xAI#3-contributing
Don't hesistate to contact us if you need help

Fix Lint python GitHub Action

Fix the following file: https://github.com/Karim-53/Compare-xAI/blob/main/.github/workflows/lint_python.yml
Current error:

30    F821 undefined name 'itertools'
31
Error: Process completed with exit code 1.

Full error report: https://github.com/Karim-53/Compare-xAI/actions/runs/3860872221/jobs/6581483832

Documentation file that could help you fix the error: https://github.com/marketplace/actions/python-syntax-checker

Understand the difference between the Archipelago Tests (2)

goal is to make sure they are distinct

Add Faith Shap (4)

FAITH-SHAP: THE FAITHFUL SHAPLEY INTERACTION INDEX https://arxiv.org/pdf/2203.00870v2.pdf
if the code is not available then no need to add it

code is still not available

Add a new Stress Test (nb of features) for interaction

    stress nb tokens test archipelago: a bad , terrible , awful , horrible movie example page 2 / 34 https://arxiv.org/pdf/2006.10965.pdf

Fix the Spellcheck Action

Todo

• Follow the instruction in https://github.com/rojopolis/spellcheck-github-actions to add a config file that include the options described below:
• fix the current error Error: Files in repository contain spelling errors by adding a config file that target *.py *.md and optionally *.yml
• Language should include both en_US and German
• In https://github.com/Karim-53/Compare-xAI/blob/main/.github/workflows/spellcheck.yml change rojopolis/[email protected] to the latest version
• Make sure the Action is working perfectly