trustyai-explainability / trustyai-explainability Goto Github PK
View Code? Open in Web Editor NEWTrustyAI Explainability Toolkit
License: Apache License 2.0
trustyai-explainability's People
Stargazers
Watchers
Forkers
tteofili ruivieira robgeada danielezonca odoochain red-hat-data-services anishasthana zdtsw zdtsw-forking jtrayfield trustycoredev mrsagarmh13 rpancham duygu-kabakci adelton odh-rhtap spolti m-a-hall christinaexyou satyamg1620 modassarrana89 ahsan-aziz-ishan adolfo-abtrustyai-explainability's Issues
HTTP Route requests return no error code
Describe the bug
After moving to TLS and HTTPS, requests to the Trusty service via http do not return anything, and should return an error code of some kind.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Filter out synthetic predictions from bias calculations
Synthetic predictions (generated by explainers to produce explanations) should not contribute to fairness/bias calculation metrics (DIR, SPD, etc.).
Use separate metadata and data tables
Describe the bug
Use separate metadata and data tables
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
TrustyAI-Service: Scheduled features + outputs need to be user-configurable
Need some deployment-time configuration for the scheduled fairness metrics, so that the target features and outputs can be user-configured
Move TrustyAI-service into TrustyAI-explainability
Describe the bug
Move the separate repo for the service into the main explainability one.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Enhancement] Add feature/output naming endpoint
Users need to be able to provide feature and output name mappings for non-labeled model codecs
Support for external explainability libraries
Related issues
Core
This feature depends on the rewrite of the KServe/ModelMesh parser rewrite (which supports batch inferences):
Service
Python
- trustyai-explainability/trustyai-explainability-python#157
- trustyai-explainability/trustyai-explainability-python#173
- trustyai-explainability/trustyai-explainability-python#163
- trustyai-explainability/trustyai-explainability-python#171
- trustyai-explainability/trustyai-explainability-python-examples#30
- trustyai-explainability/trustyai-explainability-python-examples#29
Related PRs
Disable Prometheus scheduler for tests
Describe the bug
Disable Prometheus scheduler for tests
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Enhancement] Biased model tests need to moved into Openshift-CI
Support for explainers LIME, SHAP, CF at service level
Add storage writer abstraction
Describe the bug
Add a storage writer abstraction.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Add REST endpoint to provide service metadata
Describe the bug
A REST endpoint should be available to provide service metadata (storage location, data size, etc)
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Fix parsing of raw Kserve contents
Add health checks to the trustyai-service
Describe the bug
Add readiness and liveliness probes to the trustyai-service.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Implement wrapper for Python algorithms
[Mandatory] Add TLS support for TrustyAI endpoints
Related issues
Related PRs
FLOAT types are never produced when creating a schema
Describe the bug
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Expand health checks
Describe the bug
Add health checks to datasource access (data read and write).
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Enhancement] Support both NP and PD codec in KServe v2 protobuf
Description
Support both NP and PD codec in KServe v2 protobuf.
Add REST endpoint to list all active metric requests
Describe the bug
A REST endpoint should be available to list all available metric requests.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Wrong number of observations returned in service metadata
Describe the bug
Wrong number of observations returned in service metadata
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Enhancement] Feature names need to be extracted from KServe metadata
Add batch size information to Prometheus metrics
Describe the bug
Add batch size information to Prometheus metrics
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Enhancement] NP + PD codecs need to be supported by the service
Flexible scheduling/batching
Match metric favorable outcome type with payload type
Describe the bug
The favorable outcome type must match the provided type in the payload.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Stream data from storage whenever possible
Describe the bug
Stream data from storage whenever possible
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Provide validation for metric requests
Describe the bug
Provide validation for metric requests.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Add type hints to metrics request payload
Describe the bug
Metrics request should have a type hint to avoid ambiguous types (e.g. int and long).
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Provide Nexus explainability-core releases
DataFrame filtering by metadata
Currently DataFrames can only be filtered by each row's Feature.
Metadata like id, timestamp, etc. are currently treated as "hidden" Features.
To avoid breaking an input schema, hidden Features are not "materialized" into DataFrame rows/Predictions.
This makes it impossible to filter DataFrame rows by metadata.
We need to separate actual Features and Metadata in a way that allows filtering DataFrame rows by the latter too.
Improve service metadata endpoints
Related issues
Related PRs
Add String representation to TypedValue
Describe the bug
So that the correct value can appear on Prometheus tags.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Replace Hamcrest matchers with Junit assertions
Flaky ShapKernelExplainer CI Test
Describe the bug
SHAP ManyFeatureRegularization fails CI for Java 17
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Add Dockerfile, build instructions
Describe the bug
Build instructions should not depend on jib.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Move explainer instantiation to construction phase
In the Python API wrapper move the explainer instantiation to the construction phase to avoid redoing expensive calculations.
Ability to cache metrics if the dataset hasn't changed
Describe the bug
Ability to cache metrics if the dataset hasn't changed
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Invalid requests to the service fail, but return 200 OK
[Enhancement] Remove types from request payloads
Stream file data instead of reading into memory
Describe the bug
Stream file data instead of reading into memory
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Sending floats in metric request can cause validation errors
When sending a float where the JSON representation is missing the decimal part (e.g. 1.0 sent as 1), the request validation fails.
Create Dockerfile for service
Describe the bug
Create a Dockerfile for the service image.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Add PVC read/write support
Describe the bug
Add PVC read/write support.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Support per-metric request batch size
Description
Add ability to specify a different batch size for each scheduled metric request.
Disable negative readiness check when no data is available
Describe the bug
Disable negative readiness check when no data is available
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
[Mandatory] Service needs TLS support for internal+external services and routes
Related issues
Related PRs
Add support for multiple models in the same service
Describe the bug
Add support for multiple models in the same service
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Add schema validation to payload consumer
Describe the bug
Avoid accepting and serialising new payloads if their schema is different from historical ones.
Expected behavior
No response
Actual behavior
No response
How to Reproduce?
No response
Output of uname -a or ver
No response
Output of java -version
No response
TrustyAI version or git rev
No response
Additional information
No response
Ability to update metrics requests
Should the service be able to update the values of an existing metric request?
Or should the behaviour be that all requests are immutable and a different request should be create (and the previous deleted, if that's necessary)?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.