servicenow/azimuth

Helping AI practitioners better understand their datasets and models in text classification. From ServiceNow.

Azimuth, an open-source dataset and error analysis tool for text classification, with love from ServiceNow.

Overview

Azimuth is an open source application that helps AI practitioners and data scientists better understand their dataset and model predictions by performing thorough dataset and error analyses. The application leverages different tools, including robustness tests, semantic similarity analysis and saliency maps, unified by concepts such as smart tags and proposed actions. It currently only supports text classification use cases.

Live Demo

Play with Azimuth on CLINC-OOS and a DistilBert model with a live demo.

Published Paper at EMNLP

Our paper was accepted as a system demo at EMNLP 2022. It is available here.

Documentation

Our documentation contains all the instructions and references to use Azimuth.

• Getting Started contains all the instructions to install and launch the app.
• Key Concepts explains the different concepts and analyses that are provided in Azimuth to perform dataset and error analysis.
• User Guide goes screen per screen to explain the different interactions and visualizations available.
• Reference details the config file and the different contracts which allow configuring Azimuth with different datasets and pipelines.
• Development guides on how to develop and contribute to the repo.

YouTube Playlist

Run Our Demo

After installing the requirements, as explained in Installation, run the following command to run the demo. More details in Learn Basics.

pip install gdown
make download_demo
make CFG_PATH=/config/development/clinc/conf.json launch

Once the startup tasks are completed, you will be able to access Azimuth at http://localhost:8080. To learn more about Azimuth's features, refer to our Key Concepts and our User Guide.

To use your own dataset and models, please refer to Run on Your Use Case.

How to Contribute?

Refer to our CONTRIBUTING.md.

Code Documentation

At different places in the code, you'll find README files explaining further how the back end works. Don't hesitate to add other README files in appropriate places, and don't forget to edit them if you change these components.

• In azimuth, explaining the main back-end components.
• In azimuth/modules, explaining how modules work.
• In azimuth/routers, explaining how to add new routes.

Repo Structure

├── azimuth  # Back End
│   ├── modules
│   │   └── Where core capabilities are implemented
│   ├── plots
│   │   └── Plotting functions
│   ├── routers
│   │   └── FastAPI routers
│   ├── types
│   │   └── Specific pydantic types that this application is based on
│   ├── utils
│   │   └── Extra utilities
├── azimuth_shr
│   └── Custom code that is shareable.
├── config
│   └── Config files for different data/models
├── docs
│   └── User documentation
├── tests
│   └── Unit/integration tests
├── webapp
│   └── Front End
├── docker-compose.yml # Where the config and images are specified.
├── docker-compose-gpu.yml # Extension to use Azimuth with gpus.
└── runner.py # Application entrypoint.

Contact

To contact us, join our slack or send us an email!

License

The package is licensed by ServiceNow, Inc. under the Apache 2.0 license. See LICENSE for more details.

Vulnerability Reporting

Please notify [email protected] regarding any vulnerability reports in addition to following current reporting procedure.

azimuth's Issues

Fix unexpected visible scrollbars

For obscure reasons, it happens on certain machines only, like @Dref360's, who also found the solution. Thank you!

Investigate warnings in tests

Visual indicator of how similar utterances are in the similar utterances tab

Bars? With different colors when the similarity is above or below the threshold (0.5) that determine the no_close smart tag?

Show progress bar during startup

Show not-normalized confusion matrix

Add a toggle beside the confusion matrix

Align similarity smart tags with the number of neighbors displayed in the app (10 vs 20)

Smart tags related to conflicting neighbors use 20 neighbors to compute the tag, but we only show 10 in the UI.

-> We decided to show 20 in the UI instead.

Fix flaky test

The tests/test_modules/test_model_performance/test_metrics.py::test_metrics seems flaky. It failed during https://github.com/ServiceNow/azimuth/runs/6726561997?check_suite_focus=true, before passing when I reran the tests.

Should `get_dataset_split` fail when the table linked to the `table_key` has not used `save_results` yet.

.pyproject.toml.swp file which should be added to gitignore?

Describe the solution you'd like
poetry install creates a .pyproject.toml.swp file.
Should you add it to .gitignore? Not sure what it is for

Handle multiple dataset versions

In Azimuth we can only load one training set and one evaluation set at the moment. It would be interesting to be able to load different version of those sets to compare if an improvement was made, from a version of a dataset to another.

Links from external sites to documentation lead to old versions - not main or v2.1.1

Add F1 Metric

Issue with adding a proposed action

When adding a proposed action in the front-end, we get an error message.

In the logs:

File "/usr/local/lib/python3.8/dist-packages/azimuth/utils/filtering.py", line 92, in <genexpr>
((not any(x[v] for v in ALL_SMART_TAGS)) if v == SmartTag.no_smart_tag else x[v])
KeyError: 'failed_punctuation'

Group smart tag into families

Add visual bars in performance table on the dashboard to see the magnitude of each value

horizontal bars in each column with numerical values in the performance analysis table.

"pip install gdown" doesn't work

Is your feature request related to a problem? Please describe.
pip install gdown doesn't work perhaps due to various dependency mismatches

Collecting gdown
  Using cached gdown-4.4.0.tar.gz (14 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (44 lines):
  Traceback (most recent call last):
    File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/__main__.py", line 26, in <module>
      sys.exit(_main())
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 73, in main
      command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
      module = importlib.import_module(module_path)
    File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 783, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 24, in <module>
      from pip._internal.cli.req_command import RequirementCommand, with_cleanup
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
      from pip._internal.index.package_finder import PackageFinder
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 21, in <module>
      from pip._internal.index.collector import parse_links
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 14, in <module>
      from pip._vendor import html5lib, requests
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py", line 114, in <module>
      from . import utils
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
      from . import certs
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
      from pip._vendor.certifi import where
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
      from .core import contents, where
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
      from importlib.resources import read_text
    File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/resources.py", line 11, in <module>
      from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1359, in <module>
      class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1007, in __new__
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

Describe the solution you'd like
Perhaps update the tutorial/documentation to better handle this. poetry dev-dependency?

Reorder the confusion matrix results so that the rejection class is always last.

Fix version menu in docs

On main, we get a select menu to switch version. If we click v2.1, it disappears.

Add outcome to the neighbours response, so we can show it in the UI on the utterance details

Add short/long utterances thresholds on token count plot

Add vertical lines to show what bars are considered short vs long sentences on the token count plot.

`BEHAVIORAL_TESTING=null` do not disable the behavioral tests

Investigate the issue and explore related caching issue.

Fix performance analysis table overall row not on top at initial load

Since I added the initial sort model, the overall row is not on top at initial load. After sorting any column, it comes back to the top.

Refactor Perturbation Tests

Separate the creation of the perturbation from the prediction part. This way, part of the cache will be reusable for different pipelines. This will also helps us provide something from the UI when there is no pipeline to export the perturbed dataset.

clear steps to run without docker?

Describe the solution you'd like
Could you provide a solution to run this without using docker? Although it is mentioned in the tutorial that it can be run without docker and we could use poetry. However, after following through the first few steps in the tutorial there were make commands which started building some large docker images. Docker images take a long time to build, eat a lot of resources and seem like overkill. If I just would like to play with the package, it seems not easy without committing resources and time.

Improving the section "Prepare the Config"

Describe the solution you'd like
In "Run on your use case" doc and under "1. Prepare the Config File" section, step 2 is quite confusing. The repository itself is called azimuth and then there is also an azimuth folder within it. So when the doc says in azimuth/config copy config/examples/clinc_oos/conf.json, I suppose it meant config folder at the repository root but I had to look inside azimuth/azimuth folder. There needs to be a little improvement for this step description.

Additional context
I don't understand the context fully but could there be more intuitive names for folders azimuth_shr and azimuth? Not sure what shr stands for and why there is azimuth within azimuth 🙂. Not asking to change it but just sharing my general confusion while browsing the repository 👀.

Reduce number of cached HF Dataset

The current behavior creates a new file for each annotation. This is a bit wasteful and might lead to issues in the long term.

We could remove "unused" datasets as new one are created.

Add syntax tagging config so values can be configurable by the user

Right now, the number of tokens that determine a short vs a long sentence are hard coded. They could easily be added in the configuration.

Document how to include dependencies at runtime.

Users should not have to modify the Docker image to include their dependencies.

Fix Table styling - Remove black border from headers

A black border appears when selecting a cell in different tables.

Add `required_mod_options` and raise when `mod_options` are missing them

model_contract_method_name
pipeline_idx

Issue on some browser/machines where additional scrollbars are added throughout

@Dref360 can you include a screenshot?

Can we order rows/columns in the confusion matrix in terms of meaningful confusion clusters.

Rethink postprocessed_prediction to manage metrics

We currently keep the top-1 prediction
This is confusing as we can't recover the probability distribution from this.

Add fixtures for uneven datasets

Fixtures to add:

Evaluation set has REJECTION_CLASS
Evaluation set is missing one class from train set
Combination of the above.

More robustness around prediction tags vs normal tags

Fail if a prediction tag is retrieved from a table without a pipeline_idx

The `--debug` flag doesn't show error in the validation module logs, it just hangs.

Add index.md to the User Guide section

It's the only section of the documentation missing an index.

Performance Analysis Table doesn't work for some smart tags

Some modules are not specified in the dependencies of the MetricsPerFilter Module, causing some smart tags to show as empty in the performance analysis table.

Add pipeline comparison as a smart tag family

Ability to hide columns in the performance analysis table

Plot to highlight patterns between smart tags and labels (or else?)

Might be a 2D matrix (heatmap) with smart tags and labels.

saliency values on mouseover

Describe the solution you'd like
In the Utterance table under exploration, saliency values are not shown on mouseover unless the user clicks and opens the utterance in "utterance details" section.

Error running tutorial steps

Describe the bug
got the following error while running make CFG_PATH=/config/development/clinc_dummy/conf.json compose

      Attaching to azimuth_web_1, azimuth_backend_1
      web_1      | internal/modules/cjs/loader.js:892
      web_1      |   throw err;
      web_1      |   ^
      web_1      |
      web_1      | Error: Cannot find module 'cors'
      web_1      | Require stack:
      web_1      | - /app/server.js
      web_1      |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
      web_1      |     at Function.Module._load (internal/modules/cjs/loader.js:745:27)
      web_1      |     at Module.require (internal/modules/cjs/loader.js:961:19)
      web_1      |     at require (internal/modules/cjs/helpers.js:92:18)
      web_1      |     at Object.<anonymous> (/app/server.js:4:14)
      web_1      |     at Module._compile (internal/modules/cjs/loader.js:1072:14)
      web_1      |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
      web_1      |     at Module.load (internal/modules/cjs/loader.js:937:32)
      web_1      |     at Function.Module._load (internal/modules/cjs/loader.js:778:12)
      web_1      |     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12) {
      web_1      |   code: 'MODULE_NOT_FOUND',
      web_1      |   requireStack: [ '/app/server.js' ]
      web_1      | }
      azimuth_web_1 exited with code 1

Being able to compare the performance analysis table with a second model

Copy front-end build in a clean image

Create a new target in the front-end docker file that copies the build folder and executes it.

Leverage prettier for markdown files

Fix URLs to docs

This is following the versioning of our documentation.

URLs without the version (for example https://servicenow.github.io/azimuth/getting-started that is linked in the README) still exist and seem to point to an older version (not main). I can imagine a couple of options:

1. Those URLs could redirect to /main/... ?
2. Those URLs could be the "main" ones? Replacing /main/...
3. Those URLs could die? And we make sure we convert all the links to them.

Add tooltips and subtitles on main components, with link to documentation

See our documentation for the relevant text to show.

Change query parameters from camelCase to snake_case

Ability to save user's preferences - such as hidden columns