azure/rai-vnext-preview

A Repository for the public preview of Responsible AI in AML vNext

Azure Machine Learning Responsible AI Dashboard

Welcome to the public preview for the new Responsible AI dashboard in Azure Machine Learning (AzureML) SDK/CLIv2 and studio. The following is a guide for you to onboard to the new capabilities. For questions, please contact [email protected]. Visit the Microsoft how-to docs relating to this repo.

What is the new Responsible AI Dashboard?

Responsible AI is an approach to assessing, developing, and deploying AI systems in a safe, trustworthy, and ethical manner, and take responsible decisions and actions.

AzureML currently supports both model explanations and model fairness in public preview. As we expand our offerings of Responsible AI tools for AzureML users, the new Responsible AI Dashboard acts as a single pane of glass, bringing together several pre-existing features and brand new offerings under a one-stop-shop SDK package and studio UI dashboard:

• Error Analysis (new) powered by Error Analysis: identify cohorts of data with higher error rate than the overall benchmark. These discrepancies might occur when the system or model underperforms for specific demographic groups or infrequently observed input conditions in the training data.

• Data Explorer: explore your dataset by feature sets and other metrics such as predicted Y or true Y

• Model Statistics: explore the distribution of your model outcomes and performance metrics

• Model Interpretability powered by InterpretML, which explains blackbox models, helping users understand their model's global behavior, or the reasons behind individual predictions.

• Counterfactual Analysis (new) powered by DiCE: explore feature-perturbed versions of the same datapoint that would have received a different prediction outcome, e.g., Taylor's loan has been rejected by the model. But they would have received the loan if their income was higher by $10,000.

• Causal Analysis (new) powered by EconML, which focuses on answering What If-style questions to apply data-driven decision-making – how would revenue be affected if a corporation pursues a new pricing strategy? Would a new medication improve a patient’s condition, all else equal?

The dashboard offers users a new powerful and robust toolkit for understanding your model and data in order to develop your machine learning models responsibly, now all in one place and integrated with your AzureML workspace. With this dashboard, you can identify model errors, diagnose why those errors are happening, and mitigate them. Moreover, the causal decision-making capabilities provide actionable insights to your stakeholders and customers.

❗ Please note: This initial version of the Responsible AI dashboard currently does not support the integration of fairness metrics. For fairness metrics, please refer to our existing offering here.

Supported scenarios, models and datasets

The AzureML components in this repo support computation of Responsible AI insights for scikit-learn models that are trained on pandas.DataFrame. The components accept both models and SciKit-Learn pipelines as input as long as the model or pipeline implements predict and predict_proba functions that conforms to the scikit-learn convention. If not compatible, you can wrap your model's prediction function into a wrapper class that transforms the output into the format that is supported (predict and predict_proba of scikit-learn), and pass that wrapper class to modules in this repo.

Currently, we support datasets having numerical and categorical features. The following table provides the scenarios supported for each of the four responsible AI insights:-

RAI insight	Binary classification	Multi-class classification	Multilabel classification	Regression	Timeseries forecasting	Categorical features	Text features	Image Features	Recommender Systems	Reinforcement Learning
Explainability	Yes	Yes	No	Yes	No	Yes	No	No	No	No
Error Analysis	Yes	Yes	No	Yes	No	Yes	No	No	No	No
Causal Analysis	Yes	No	No	Yes	No	Yes (max 5 features due to computational cost)	No	No	No	No
Counterfactual	Yes	Yes	No	Yes	No	Yes	No	No	No	No

This is all available via Python SDK or CLI.

Set Up

In this section, we will go over the basic setup steps that you need in order to generate Responsible AI insights for your models from SDK v2, CLI v2 and visualize the generated Responsible AI insights in AML studio.

Install Miniconda

Our setup script assumes that conda is installed and configured for your shell. Download and install MiniConda (External link). After installation, restart your shell, and make sure that you can run conda commands.

Create an AzureML workspace

Create an AzureML workspace by using the configuration notebook

Install the Azure CLI

Install the Azure CLI by following the instructions on the Microsoft docs site.

Clone this repository

If you have not already done so, clone this repository, and move into the root directory:

git clone https://github.com/Azure/RAI-vNext-Preview.git
cd RAI-vNext-Preview

Log in to Azure

Run the following to make sure you are logged in to Azure:

az login

Run the setup script

We provide a setup script which:

• Creates a new conda environment
• Installs all the required Python packages
• Registers all the RAI components in your AzureML workspace
• Sets the defaults for the az CLI to point to your workspace

To run the script in powershell, from the repository root, run:

 .\Quick-Setup.ps1

alternatively, an equivalent script for bash:

./quick-setup.bash <CONDA-ENV-NAME> <SUBSCRIPTION-ID> <RESOURCE-GROUP-NAME> <WORKSPACE-NAME>

This script will echo the supplied arguments and then pause briefly before continuing.

Generating Responsible AI Dashboard insights

Once you have created an Azure workspace and registered your components in the one-time setup above, you can create a Responsible AI dashboard via the CLI or SDK. Here are some useful examples to walk through to better understand how the dashboard works:

Model Debugging Examples:

• Try the tool: model debugging of an API access prediction model (classification)
• Try the tool: model debugging of a housing price prediction model (classification)
• Try the tool: model debugging of a diabetes progression prediction model (regression)

Responsible Decision Making Examples:

• Try the tool: make decisions for house improvements
• Try the tool: provide recommendations to patients using diabetes data

For examples using the CLI to generate the dashboard, check the examples/CLI folder

Viewing your Responsible AI Dashboard in the AzureML studio portal

After generating the Responsible AI insights, you can view them in your associated workspace in AzureML studio, under your model registry.

1. Go to your model registry in your AzureML studio workspace
2. Click on the model for which you've uploaded your Responsible AI insights

3. Click on the tab for Responsible AI dashboard (preview) under your model details page

4. Under the Responsible AI dashboard (preview) tab of your model details, you will see a list of your uploaded Responsible AI insights. You can upload more than one Responsible AI dashboard for each model. Each row represents one dashboard, with information on which components were uploaded to each dashboard (i.e. explanations, counterfactuals, etc).

5. At anytime while viewing the dashboard, if you wish to return to the model details page, click on Back to model details

1. You can view the dashboard insights for each component filtered down on a cohort you specify (or view all the data with the global cohort). Hovering over the cohort name will show the number of datapoints and filters in that cohort as a tooltip.
2. Switch which cohort you are applying to the dashboard.
3. Create a new cohort based on filters you can apply in a flyout panel.
4. View a list of all cohorts created and duplicate, edit or delete them.
5. View a list of all Responsible AI components you've uploaded to this dashboard as well as hiding components. The layout of the dashboard will reflect the order of the components in this list.

❗ Please note: Error Analysis, if generated, will always be at the top of the component list in your dashboard. Selecting on the nodes of the error tree or tiles of the error heatmap will automatically generate a temporary cohort that will be populated in the components below so that you can easily experiment with looking at insights for different areas of your error distribution.

6. In between each component you can add components by clicking the blue circular button with a plus sign. This will pop up a tooltip that will give you an option of adding whichever Responsible AI component you enabled with your SDK.

Accessing full functionality with a connected compute resource in Azure ML studio

Integrated compute instance must be attached to the Responsible AI dashboard to enable full functionality: Error analysis

• Setting your global data cohort to any cohort of interest will update the error tree instead of disabling it.
• Selecting other error or performance metrics is supported.
• Selecting any subset of features for training the error tree map is supported.
• Changing the minimum number of samples required per leaf node and error tree depth is supported.
• Dynamically updating the heatmap for up to two features is supported. Feature importances
• An individual conditional expectation (ICE) plot in the individual feature importance tab is supported. Counterfactual analysis
• Generating a new what-if counterfactual datapoint to understand the minimum change required for a desired outcome is supported. Causal analysis
• Selecting any individual datapoint, perturbing its treatment features, and seeing the expected causal outcome of causal what-if is supported (only for regression ML scenarios).

If the Responsible AI dashboard is created and viewed in a Jupyter notebook with a Python backend, it will be fully featured when running in that notebook.

Limitations

The current set of components have a number of limitations on their use:

• All models must be in registered in AzureML in MLFlow format with an sklearn flavour
• The models must be loadable in the component environment
• The models must be pickleable
• The models must be supplied to the RAI components using the 'Fetch Registered Model' component which we provide. We plan to switch to the new AzureML 'model-as-input' feature in the near future
• The dataset inputs must be pandas DataFrames in Parquet format. We plan to start using AzureML's new MLTable feature in the near future
• A model must still be supplied even if only a causal analysis of the data is performed. The DummyClassifier and DummyRegressor estimators from SciKit-Learn can be used for this purpose

Responsible AI Dashboard walkthrough and sample notebooks

Please read through our examples folder to see if this feature supports your use case. For more details about each individual component, please read through our brief tour guide of the new Responsible AI dashboard capabilities.

Learn more

• Click through a live interactive web demo of the Responsible AI dashboard on Microsoft AI Lab
• Read the Responsible AI dashboard and scorecard technical blog walking through this same demo scenario
• Read the customer story on how the Responsible AI dashboard is used in healthcare
• Visit the Microsoft docs on how to generate a Responsible AI dashboard with Python and YAML in Azure Machine Learning's SDKv2 and CLIv2
• Visit the Microsoft docs on how to generate a Responsible AI dashboard with no-code wizard in Azure Machine Learning studio UI

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

rai-vnext-preview's Issues

Issue in running the example responsibleaidashboard-diabetes-decision-making notebook

While running the responsibleaidashboard-diabetes-decision-making notebook example in Jupyter facing an error in Creating the RAI Insights that Error: Not found component rai_insights_gather.

Please find the error details below:

ResourceNotFoundError                     Traceback (most recent call last)
File C:\LabFiles\RAI-vNext-Preview\rai-dashboard.py:339, in <module>
    331 rai_causal_component = ml_client.components.get(
    332     name="rai_insights_causal", version=version_string
    333 )
    335 rai_counterfactual_component = ml_client.components.get(
    336     name="rai_insights_counterfactual", version=version_string
    337 )
--> 339 rai_gather_component = ml_client.components.get(
    340     name="rai_insights_gather", version=version_string
    341 )
    343 rai_scorecard_component = ml_client.components.get(
    344     name="rai_score_card", version=version_string
    345 )
    348 # ## Score card generation config
    349 # For score card generation, we need some additional configuration in a separate json file. Here we configure the
following model performance metrics for reporting:
    350 # - mean absolute error
    351 # - mean squared error
    352
    353 # In[ ]:

File C:\LabFiles\lib\site-packages\azure\ai\ml\_telemetry\activity.py:230, in monitor_with_telemetry_mixin.<locals>.monit
or.<locals>.wrapper(*args, **kwargs)
    228 dimensions = {**parameter_dimensions, **(custom_dimensions or {})}
    229 with log_activity(logger, activity_name or f.__name__, activity_type, dimensions) as activityLogger:
--> 230     return_value = f(*args, **kwargs)
    231     if not parameter_dimensions:
    232         # collect from return if no dimensions from parameter
    233         activityLogger.activity_info.update(_collect_from_return_value(return_value))

File C:\LabFiles\lib\site-packages\azure\ai\ml\operations\_component_operations.py:182, in ComponentOperations.get(self,
name, version, label)
    166     msg = "Must provide either version or label."
    167     raise ValidationException(
    168         message=msg,
    169         target=ErrorTarget.COMPONENT,
    170         no_personal_data_message=msg,
    171         error_category=ErrorCategory.USER_ERROR,
    172     )
    173 result = (
    174     self._version_operation.get(
    175         name=name,
    176         version=version,
    177         resource_group_name=self._resource_group_name,
    178         registry_name=self._registry_name,
    179         **self._init_args,
    180     )
    181     if self._registry_name
--> 182     else self._version_operation.get(
    183         name=name,
    184         version=version,
    185         resource_group_name=self._resource_group_name,
    186         workspace_name=self._workspace_name,
    187         **self._init_args,
    188     )
    189 )
    190 component = Component._from_rest_object(result)
    191 return component

File C:\LabFiles\lib\site-packages\azure\core\tracing\decorator.py:78, in distributed_trace.<locals>.decorator.<locals>.w
rapper_use_tracer(*args, **kwargs)
     76 span_impl_type = settings.tracing_implementation()
     77 if span_impl_type is None:
---> 78     return func(*args, **kwargs)
     80 # Merge span is parameter is set, but only if no explicit parent are passed
     81 if merge_span and not passed_in_parent:

File C:\LabFiles\lib\site-packages\azure\ai\ml\_restclient\v2022_05_01\operations\_component_versions_operations.py:443,
in ComponentVersionsOperations.get(self, resource_group_name, workspace_name, name, version, **kwargs)
    440 response = pipeline_response.http_response
    442 if response.status_code not in [200]:
--> 443     map_error(status_code=response.status_code, response=response, error_map=error_map)
    444     error = self._deserialize.failsafe_deserialize(_models.ErrorResponse, pipeline_response)
    445     raise HttpResponseError(response=response, model=error, error_format=ARMErrorFormat)

File C:\LabFiles\lib\site-packages\azure\core\exceptions.py:107, in map_error(status_code, response, error_map)
    105     return
    106 error = error_type(response=response)
--> 107 raise error

ResourceNotFoundError: (UserError) Not found component rai_insights_gather.
Code: UserError
Message: Not found component rai_insights_gather.

ModuleNotFoundError: No module named 'colorama'

Following instructions to test "User Workflow" example notebook.
create conda environment and installed packages as per https://github.com/Azure/RAI-vNext-Preview/blob/main/docs/Setup.md

from azure.ml import MLClient
from azure.identity import DefaultAzureCredential
ml_client = MLClient.from_config(credential=DefaultAzureCredential(exclude_shared_token_cache_credential=True),
                     logging_enable=True)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/home/krbock/GitHub/RAI-vNext-Preview/examples/notebooks/User Workflow.ipynb Cell 4' in <module>
----> [1](vscode-notebook-cell://wsl%2Bubuntu-18.04/home/krbock/GitHub/RAI-vNext-Preview/examples/notebooks/User%20Workflow.ipynb#ch0000003vscode-remote?line=0) from azure.ml import MLClient
      [2](vscode-notebook-cell://wsl%2Bubuntu-18.04/home/krbock/GitHub/RAI-vNext-Preview/examples/notebooks/User%20Workflow.ipynb#ch0000003vscode-remote?line=1) from azure.identity import DefaultAzureCredential
      [3](vscode-notebook-cell://wsl%2Bubuntu-18.04/home/krbock/GitHub/RAI-vNext-Preview/examples/notebooks/User%20Workflow.ipynb#ch0000003vscode-remote?line=2) ml_client = MLClient.from_config(credential=DefaultAzureCredential(exclude_shared_token_cache_credential=True),
      [4](vscode-notebook-cell://wsl%2Bubuntu-18.04/home/krbock/GitHub/RAI-vNext-Preview/examples/notebooks/User%20Workflow.ipynb#ch0000003vscode-remote?line=3)                      logging_enable=True)

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py:7, in <module>
      [1](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=0) # ---------------------------------------------------------
      [2](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=1) # Copyright (c) Microsoft Corporation. All rights reserved.
      [3](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=2) # ---------------------------------------------------------
      [5](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=4) __path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore
----> [7](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=6) from ._ml_client import MLClient
      [8](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=7) import logging
      [9](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/__init__.py?line=8) from ._utils.utils import initialize_logger_info

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py:17, in <module>
     [15](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=14) from azure.ml._constants import AzureMLResourceType
     [16](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=15) from azure.ml._file_utils.file_utils import traverse_up_path_and_find_file
---> [17](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=16) from azure.ml._operations import (
     [18](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=17)     BatchDeploymentOperations,
     [19](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=18)     BatchEndpointOperations,
     [20](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=19)     CodeOperations,
     [21](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=20)     ComponentOperations,
     [22](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=21)     ComputeOperations,
     [23](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=22)     DataOperations,
     [24](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=23)     DatasetOperations,
     [25](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=24)     DatastoreOperations,
     [26](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=25)     EnvironmentOperations,
     [27](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=26)     JobOperations,
     [28](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=27)     ModelOperations,
     [29](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=28)     OnlineDeploymentOperations,
     [30](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=29)     OnlineEndpointOperations,
     [31](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=30)     WorkspaceOperations,
     [32](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=31)     _LocalDeploymentHelper,
     [33](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=32)     _LocalEndpointHelper,
     [34](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=33) )
     [35](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=34) from azure.ml._restclient.runhistory import AzureMachineLearningWorkspaces as ServiceClientRunHistory
     [36](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=35) from azure.ml._restclient.v2020_09_01_dataplanepreview import (
     [37](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=36)     AzureMachineLearningWorkspaces as ServiceClient092020DataplanePreview,
     [38](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_ml_client.py?line=37) )

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py:8, in <module>
      [1](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=0) # ---------------------------------------------------------
      [2](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=1) # Copyright (c) Microsoft Corporation. All rights reserved.
      [3](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=2) # ---------------------------------------------------------
      [5](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=4) __path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore
----> [8](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=7) from .compute_operations import ComputeOperations
      [9](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=8) from .datastore_operations import DatastoreOperations
     [10](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/__init__.py?line=9) from .job_operations import JobOperations

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py:14, in <module>
     [12](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=11) from azure.ml._workspace_dependent_operations import WorkspaceScope, _WorkspaceDependentOperations
     [13](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=12) from azure.ml._constants import ComputeType, LROConfigurations, COMPUTE_UPDATE_ERROR
---> [14](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=13) from azure.ml.entities import Compute, Usage, VmSize, AmlComputeNodeInfo
     [15](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=14) from azure.ml._utils._azureml_polling import AzureMLPolling, polling_wait
     [16](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=15) from azure.ml._restclient.v2021_10_01.models import (
     [17](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=16)     AmlComputeNodeInformation,
     [18](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_operations/compute_operations.py?line=17) )

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py:8, in <module>
      [5](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py?line=4) __path__ = __import__("pkgutil").extend_path(__path__, __name__)
      [7](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py?line=6) from ._resource import Resource
----> [8](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py?line=7) from ._assets._artifacts.model import Model
      [9](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py?line=8) from ._assets.asset import Asset
     [10](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/__init__.py?line=9) from ._assets.environment import Environment

File ~/anaconda3/envs/azureml_rai/lib/python3.[8](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/__init__.py?line=7)/site-packages/azure/ml/entities/_assets/__init__.py:10, in <module>
      8 from ._artifacts.artifact import Artifact
      [9](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/__init__.py?line=8) from ._artifacts.code import Code
---> [10](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/__init__.py?line=9) from ._artifacts.model import Model
     [11](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/__init__.py?line=10) from ._artifacts.data import Data
     [12](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/__init__.py?line=11) from ._artifacts.dataset import Dataset

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py:18, in <module>
     [16](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py?line=15) from azure.ml._schema import ModelSchema
     [17](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py?line=16) from azure.ml._utils._arm_id_utils import AMLNamedArmId, AMLVersionedArmId
---> [18](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py?line=17) from azure.ml._utils._storage_utils import AzureMLDatastorePathUri
     [19](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py?line=18) from azure.ml._utils.utils import load_yaml
     [20](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/entities/_assets/_artifacts/model.py?line=19) from azure.ml.entities._assets import Artifact

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py:9, in <module>
      [6](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=5) import re
      [7](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=6) from typing import Tuple, Union
----> [9](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=8) from azure.ml._artifacts._default_storage_helper import DefaultStorageClient
     [10](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=9) from azure.ml._artifacts._fileshare_storage_helper import FileStorageClient
     [11](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=10) from azure.ml._restclient.v2021_10_01.models import (
     [12](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=11)     DatastoreType,
     [13](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_utils/_storage_utils.py?line=12) )

File ~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py:14, in <module>
     [12](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py?line=11) from pathlib import PurePosixPath, Path
     [13](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py?line=12) from multiprocessing import cpu_count
---> [14](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py?line=13) from colorama import Fore
     [15](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py?line=14) from concurrent.futures import ThreadPoolExecutor, as_completed
     [16](file:///~/anaconda3/envs/azureml_rai/lib/python3.8/site-packages/azure/ml/_artifacts/_default_storage_helper.py?line=15) from tqdm import tqdm, TqdmWarning

ModuleNotFoundError: No module named 'colorama'

Getting error while creating Responsible Ai dashboard for mlopsv2

@kicha0 I am trying to use different algorithms for training. Model training and model registration part is successfully completed, but RAI insights dashboard constructor is failed. I am using the xgboost model for training.

Thank you

Traceback (most recent call last): File "create_rai_insights.py", line 184, in <module> main(args) File "create_rai_insights.py", line 128, in main model_estimator = load_mlflow_model(my_run.experiment.workspace, model_id=model_id) File "/mnt/azureml/cr/j/exe/wd/rai_component_utilities.py", line 79, in load_mlflow_model return mlflow.pyfunc.load_model(model_uri)._model_impl File "/azureml-envs/responsibleai-0.18/lib/python3.8/site-packages/mlflow/pyfunc/__init__.py", line 484, in load_model model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path) File "/azureml-envs/responsibleai-0.18/lib/python3.8/site-packages/mlflow/sklearn/__init__.py", line 494, in _load_pyfunc return _load_model_from_local_file(path=path, serialization_format=serialization_format) File "/azureml-envs/responsibleai-0.18/lib/python3.8/site-packages/mlflow/sklearn/__init__.py", line 452, in _load_model_from_local_file return cloudpickle.load(f) ModuleNotFoundError: No module named 'xgboost'

Error when running pipeline submission - Exception: Value 'path' passed is not in set

I am getting the following error when I try to submit the job to the pipeline.

Exception: Value 'path' passed is not in set ['uri_file', 'uri_folder', 'custom_model', 'mlflow_model', 'mltable', 'triton_model']

Any suggestions?

Thanks

Data Explorer in RAI Dashboard

Hi,

As per my understanding DATA EXPLORER Component in RAI Dashboard helps to visualize the issues of overrepresentation and underrepresentation of data in the test dataset only. [ not the training dataset]

But if the model is with trained with proper training datasets [ no skewed data] , then how this improper representation of data in test dataset will affect the model performance.