Helping AI practitioners better understand their datasets and models in text classification. From ServiceNow.
Home Page: https://servicenow.github.io/azimuth/
License: Apache License 2.0
We currently keep the top-1 prediction
This is confusing as we can't recover the probability distribution from this.
Fail if a prediction tag is retrieved from a table without a pipeline_idx
The current behavior creates a new file for each annotation. This is a bit wasteful and might lead to issues in the long term.
We could remove "unused" datasets as new one are created.
Might be a 2D matrix (heatmap) with smart tags and labels.
Investigate the issue and explore related caching issue.
horizontal bars in each column with numerical values in the performance analysis table.
Some modules are not specified in the dependencies of the MetricsPerFilter Module, causing some smart tags to show as empty in the performance analysis table.
Add a toggle beside the confusion matrix
@Dref360 can you include a screenshot?
Describe the solution you'd like
In "Run on your use case" doc and under "1. Prepare the Config File" section, step 2 is quite confusing. The repository itself is called azimuth and then there is also an azimuth folder within it. So when the doc says in azimuth/config copy config/examples/clinc_oos/conf.json, I suppose it meant config folder at the repository root but I had to look inside azimuth/azimuth folder. There needs to be a little improvement for this step description.
Additional context
I don't understand the context fully but could there be more intuitive names for folders azimuth_shr and azimuth? Not sure what shr stands for and why there is azimuth within azimuth 🙂. Not asking to change it but just sharing my general confusion while browsing the repository 👀.
It's the only section of the documentation missing an index.
When adding a proposed action in the front-end, we get an error message.
In the logs:
File "/usr/local/lib/python3.8/dist-packages/azimuth/utils/filtering.py", line 92, in <genexpr>
((not any(x[v] for v in ALL_SMART_TAGS)) if v == SmartTag.no_smart_tag else x[v])
KeyError: 'failed_punctuation'
Users should not have to modify the Docker image to include their dependencies.
Separate the creation of the perturbation from the prediction part. This way, part of the cache will be reusable for different pipelines. This will also helps us provide something from the UI when there is no pipeline to export the perturbed dataset.
Bars? With different colors when the similarity is above or below the threshold (0.5) that determine the no_close smart tag?
Describe the solution you'd like
Could you provide a solution to run this without using docker? Although it is mentioned in the tutorial that it can be run without docker and we could use poetry. However, after following through the first few steps in the tutorial there were make commands which started building some large docker images. Docker images take a long time to build, eat a lot of resources and seem like overkill. If I just would like to play with the package, it seems not easy without committing resources and time.
Is your feature request related to a problem? Please describe.
pip install gdown doesn't work perhaps due to various dependency mismatches
Collecting gdown
Using cached gdown-4.4.0.tar.gz (14 kB)
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
cwd: None
Complete output (44 lines):
Traceback (most recent call last):
File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/__main__.py", line 26, in <module>
sys.exit(_main())
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 73, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
module = importlib.import_module(module_path)
File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 24, in <module>
from pip._internal.cli.req_command import RequirementCommand, with_cleanup
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
from pip._internal.index.package_finder import PackageFinder
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 21, in <module>
from pip._internal.index.collector import parse_links
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 14, in <module>
from pip._vendor import html5lib, requests
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py", line 114, in <module>
from . import utils
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
from . import certs
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
from pip._vendor.certifi import where
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
from .core import contents, where
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
from importlib.resources import read_text
File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/resources.py", line 11, in <module>
from typing import Iterable, Iterator, Optional, Set, Union # noqa: F401
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1359, in <module>
class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1007, in __new__
self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
Describe the solution you'd like
Perhaps update the tutorial/documentation to better handle this. poetry dev-dependency?
Describe the bug
got the following error while running make CFG_PATH=/config/development/clinc_dummy/conf.json compose
Attaching to azimuth_web_1, azimuth_backend_1
web_1 | internal/modules/cjs/loader.js:892
web_1 | throw err;
web_1 | ^
web_1 |
web_1 | Error: Cannot find module 'cors'
web_1 | Require stack:
web_1 | - /app/server.js
web_1 | at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
web_1 | at Function.Module._load (internal/modules/cjs/loader.js:745:27)
web_1 | at Module.require (internal/modules/cjs/loader.js:961:19)
web_1 | at require (internal/modules/cjs/helpers.js:92:18)
web_1 | at Object.<anonymous> (/app/server.js:4:14)
web_1 | at Module._compile (internal/modules/cjs/loader.js:1072:14)
web_1 | at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
web_1 | at Module.load (internal/modules/cjs/loader.js:937:32)
web_1 | at Function.Module._load (internal/modules/cjs/loader.js:778:12)
web_1 | at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12) {
web_1 | code: 'MODULE_NOT_FOUND',
web_1 | requireStack: [ '/app/server.js' ]
web_1 | }
azimuth_web_1 exited with code 1
See our documentation for the relevant text to show.
Since I added the initial sort model, the overall row is not on top at initial load. After sorting any column, it comes back to the top.
Add vertical lines to show what bars are considered short vs long sentences on the token count plot.
The tests/test_modules/test_model_performance/test_metrics.py::test_metrics seems flaky. It failed during https://github.com/ServiceNow/azimuth/runs/6726561997?check_suite_focus=true, before passing when I reran the tests.
Describe the solution you'd like
poetry install creates a .pyproject.toml.swp file.
Should you add it to .gitignore? Not sure what it is for
Describe the solution you'd like
In the Utterance table under exploration, saliency values are not shown on mouseover unless the user clicks and opens the utterance in "utterance details" section.
For obscure reasons, it happens on certain machines only, like @Dref360's, who also found the solution. Thank you!
Smart tags related to conflicting neighbors use 20 neighbors to compute the tag, but we only show 10 in the UI.
-> We decided to show 20 in the UI instead.
Create a new target in the front-end docker file that copies the build folder and executes it.
In Azimuth we can only load one training set and one evaluation set at the moment. It would be interesting to be able to load different version of those sets to compare if an improvement was made, from a version of a dataset to another.
This is following the versioning of our documentation.
URLs without the version (for example https://servicenow.github.io/azimuth/getting-started that is linked in the README) still exist and seem to point to an older version (not main). I can imagine a couple of options:
A black border appears when selecting a cell in different tables.
model_contract_method_name
pipeline_idx
Right now, the number of tokens that determine a short vs a long sentence are hard coded. They could easily be added in the configuration.
Fixtures to add:
Evaluation set has REJECTION_CLASS
Evaluation set is missing one class from train set
Combination of the above.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.