enochkan/torch-metrics

Metrics for model evaluation in pytorch

Torch-metrics

Model evaluation metrics for PyTorch

Torch-metrics serves as a custom library to provide common ML evaluation metrics in Pytorch, similar to tf.keras.metrics.

As summarized in this issue, Pytorch does not have a built-in libary torch.metrics for model evaluation metrics. This is similar to the metrics library in PyTorch Lightning.

Usage

• pip install --upgrade torch-metrics

from torch_metrics import Accuracy

## define metric ##
metric = Accuracy(from_logits=False)
y_pred = torch.tensor([1, 2, 3, 4])
y_true = torch.tensor([0, 2, 3, 4])

print(metric(y_pred, y_true))

## define metric ##
metric = Accuracy()
y_pred = torch.tensor([[0.2, 0.6, 0.1, 0.05, 0.05],
                       [0.2, 0.1, 0.6, 0.05, 0.05],
                       [0.2, 0.05, 0.1, 0.6, 0.05],
                       [0.2, 0.05, 0.05, 0.05, 0.65]])
y_true = torch.tensor([0, 2, 3, 4])

print(metric(y_pred, y_true))

Implementation

Metrics from tf.keras.metrics and other metrics that are already implemented vs to-do

• MeanSquaredError class
• RootMeanSquaredError class
• MeanAbsoluteError class
• Precision class
• Recall class
• MeanIoU class
• DSC class (Dice Similarity Coefficient)
• F1Score class
• RSquared class
• Hinge class
• SquaredHinge class
• LogCoshError class
• Accuracy class
• KLDivergence class
• CosineSimilarity class
• AUC class
• BinaryCrossEntropy class
• CategoricalCrossEntropy class
• SparseCategoricalCrossentropy class

Local Development and Testing

To quickly get started with local development, run:

make develop

To test, run:

python3 -m pytest

Pre-commit hooks

To run pre-commit against all files:

pre-commit run --all-files

Contributing

Please raise issues or feature requests here. It will be extremely helpful if you comment on a specific issue before working on it. This provides visibility for others who also intend to work on the same issue. Reference any pull requests to their original issues.

torch-metrics's Issues

Client specific errors

Hi all,

Client specific error handling in the methods are not present and ideally should be added. We should return proper exceptions to the client related to the incorrect and erroneous attributes in the method.

Thanks

Regarding Object detection metrics

Found this repo from Reddit, great project. I was actually struggling to find a proper resource that implements Object detection metrics like Mean Average Precision which has several versions itself, depending upon the evaluating competition, I found this paper which explains everything in detail and provides a library implementing everything, but that is in numpy and the code is pretty messy, I would like to contribute a Pytorch version of their implementation with clear API, but may take a while as I'm just learning this stuff and need time to make sure the implementation is correct.

Tests for MSLE

Hello @Vijayabhaskar96, is there any reason why you did not test MeanSquaredLogarithmicError with normally distributed data?

torch-metrics/tests/test_regression_metrics.py

Lines 54 to 64 in 7ce9929

	@pytest.mark.parametrize(
	"y_true, y_pred",
	[(uniform_regression_inputs.target, uniform_regression_inputs.preds),],
	)
	def test_msle(y_true, y_pred):
	sk_preds = y_pred.numpy()
	sk_target = y_true.numpy()
	sk_score = mean_squared_log_error(y_true=sk_target, y_pred=sk_preds)
	torch_metric = MeanSquaredLogarithmicError()
	tm_score = torch_metric(y_pred, y_true)
	assert_allclose(sk_score, tm_score)

bug when importing: "cannot import name 'classification' from partially initialized module 'torch_metrics'"

pip show torch_metrics

Name: torch-metrics
Version: 1.1.7
Summary: Metrics for model evaluation in pytorch
Home-page: https://github.com/chinokenochkan/torch-metrics
Author: Chi Nok Enoch Kan @chinokenochkan
Author-email: [e-mail]
License: UNKNOWN
Location: /Users/layne/.pyenv/versions/3.8.7/envs/jupyterlab3/lib/python3.8/site-packages
Requires: 
Required-by: 
Note: you may need to restart the kernel to use updated packages.

from torch_metrics import Accuracy


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-24-d1287b23966e> in <module>
----> 1 import torch_metrics

~/.pyenv/versions/3.8.7/envs/jupyterlab3/lib/python3.8/site-packages/torch_metrics/__init__.py in <module>
----> 1 from torch_metrics import classification
      2 from torch_metrics import regression

ImportError: cannot import name 'classification' from partially initialized module 'torch_metrics' (most likely due to a circular import) (/Users/layne/.pyenv/versions/3.8.7/envs/jupyterlab3/lib/python3.8/site-packages/torch_metrics/__init__.py)

sys.version
'3.8.7 (default, Jan 28 2021, 17:14:59) \n[Clang 12.0.0 (clang-1200.0.31.1)]'

List of metrics and Initial refactor

Hi Together,

I have created a doc to keep track of the metrics and their implementation status. Kindly feel free to edit it however you want.

@enochkan Can you let me know if it is okay to do a initial refactor like below based on the column Refactor to in the doc?

from torch_metrics.classification import <metric>
from torch_metrics.regression import <metric>

@zutshianand, kindly review the Refactor to column in the doc and make necessary changes.

@enochkan, @Vijayabhaskar96, @zutshianand Please add additional metrics that you want to implement in the doc if it is already not present and if you feel we do not need a metric, kindly change the Implementation column to Not needed.

Thanks!!

Package structure and code improvement

Hi all,

With this initiative of collating all the metrics and presenting them in Pytorch, I wanted to share my thoughts on the same.
Referring to the code structure, is there any reason why we are having the code structure of the entire package using the name of the metric rather than the task of deep learning?

IMHO, the import should be something like from torch_metrics.classification import <name of metric> rather than from torch_metrics import <name of metric>, this is because by adding the different metrics in function methods rather than separate classes, it provides the following:

• Context to the client regarding the metric usage
• Reusability across different methods of utility functions.

We can use sklearn.metrics package as an example.

Thanks!

Tool for Documentation

Hello Together,

What tool do we want to use for documentation? I know of some tools we can use:

• Sphinx (used by PyTorch)
• DocTest (used by TensorFlow)
• MkDocs
• Markdown files for each metric.

Personally, I would prefer Markdown files since they are easy to use. Let me know your thoughts @enochkan @zutshianand @Vijayabhaskar96

Tests for metrics

Currently the folder tests doesn't have any tests, there should be tests for each metric implemented, to get the current answers for test cases we can use keras.metrics and sklearn mertics. Personally I have not used any testing libraries, keras used pytest in the past before merging with TF, should we follow the same?

New metrics that are not in the README.md

Hello together,

I am thinking of adding following metrics to the library:

• ExplainedVariance
• PSNR (Peak Signal-to-Noise Ratio)
• SSIM (Structual Similarity Index Measure)
• Fbeta
• Poisson
• MeanSquaredLogError
• MeanAbsolutePercentageError
• TruePositives
• FalsePositives
• TrueNegatives
• FalseNegatives

Let me know which of these will not be useful or redundant metrics. Also @enochkan, is it okay to raise a new issue everytime if I want to include a new metric or just a PR would be sufficient?

Vision Metrics

Hi,

We need metrics for some specific vision tasks such as Object Detection, Instance Segmentation, Keypoint Detection, etc.
These metrics include:

1. Mean Average Precision
2. Intersection Over Union
   ...

The original implementation is based on the pycocotools library, I can adapt them to torch-metrics if everyone agrees.

Library "ImportError" - Missing packages in setup.py

Following the installation instructions on the README, I get this error when trying:

from torch_metrics import Accuracy
from torch_metrics import classification

The only thing that work is:
import torch_metrics

See the error:

It seems that the library's submodules are missing from setup.py.

Circular Import

I am having trouble using this tool. It seems to hav e a bug. the simple example code:

from torch_metrics import Accuracy

## define metric ##
metric = torch_metrics.Accuracy(from_logits=False)
indx = 100
print(metric(p_all[indx,:,:,:], y_all[indx,:,:,:]))

Gives the error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-20-35cedb4ac57c> in <module>
----> 1 from torch_metrics import Accuracy
      2 
      3 ## define metric ##
      4 metric = torch_metrics.Accuracy(from_logits=False)
      5 indx = 100

/trials/Catalina/02_Projects/01_DeepLearning/_virtual_envs_/venv_Pytorch_coursera/lib/python3.8/site-packages/torch_metrics/__init__.py in <module>
----> 1 from torch_metrics import classification
      2 from torch_metrics import regression

ImportError: cannot import name 'classification' from partially initialized module 'torch_metrics' (most likely due to a circular import) (_virtual_envs_/venv_Pytorch_coursera/lib/python3.8/site-packages/torch_metrics/__init__.py)

I have installed using pip and working on a Ubuntu 20.4 with python3.7

aupr metric needed

The metric evalution method aupr is needed.

Code formatting and docstrings for all functions

class MeanAbsoluteError:
    def __call__(self, tensor1, tensor2):
        """
        Arguments
        ---------
        x : torch.Tensor
        y : torch.Tensor
        """
        return torch.mean(torch.abs(tensor1 - tensor2))

Instead of tensor1 and tensor2 like above it would be great if we can change it to y_true and y_pred and also with concise docstrings like below.

class MeanAbsoluteError:
    def __call__(self, y_true, y_pred):
        """
        Computes the Mean Absolute error between the `y_true` and `y_pred`
        Args:
        y_true: Ground truth values.
        y_pred: The predicted values.
    Returns:
        Mean Absolute error
    """
        return torch.mean(torch.abs(y_true - y_pred))

I am very sorry if this sounds a bit rude. I am just thinking if we can make small things better it will attract more people.

Code Linting

Hello Together.

Should we use code linting (pylint or flake8) for better standards of the code? We can also add this as a check in CI so that contributors will also follow some coding standards.

Exhaustive list of metrics

Hi all,

Please find the below list of exhaustive metrics provided by sklearn to date. We can use that as reference while implementing them in Pytorch.

    'accuracy_score',
    'adjusted_mutual_info_score',
    'adjusted_rand_score',
    'auc',
    'average_precision_score',
    'balanced_accuracy_score',
    'calinski_harabaz_score',
    'calinski_harabasz_score',
    'check_scoring',
    'classification_report',
    'cluster',
    'cohen_kappa_score',
    'completeness_score',
    'ConfusionMatrixDisplay',
    'confusion_matrix',
    'consensus_score',
    'coverage_error',
    'dcg_score',
    'davies_bouldin_score',
    'euclidean_distances',
    'explained_variance_score',
    'f1_score',
    'fbeta_score',
    'fowlkes_mallows_score',
    'get_scorer',
    'hamming_loss',
    'hinge_loss',
    'homogeneity_completeness_v_measure',
    'homogeneity_score',
    'jaccard_score',
    'jaccard_similarity_score',
    'label_ranking_average_precision_score',
    'label_ranking_loss',
    'log_loss',
    'make_scorer',
    'nan_euclidean_distances',
    'matthews_corrcoef',
    'max_error',
    'mean_absolute_error',
    'mean_squared_error',
    'mean_squared_log_error',
    'mean_poisson_deviance',
    'mean_gamma_deviance',
    'mean_tweedie_deviance',
    'median_absolute_error',
    'multilabel_confusion_matrix',
    'mutual_info_score',
    'ndcg_score',
    'normalized_mutual_info_score',
    'pairwise_distances',
    'pairwise_distances_argmin',
    'pairwise_distances_argmin_min',
    'pairwise_distances_chunked',
    'pairwise_kernels',
    'plot_confusion_matrix',
    'plot_precision_recall_curve',
    'plot_roc_curve',
    'PrecisionRecallDisplay',
    'precision_recall_curve',
    'precision_recall_fscore_support',
    'precision_score',
    'r2_score',
    'recall_score',
    'RocCurveDisplay',
    'roc_auc_score',
    'roc_curve',
    'SCORERS',
    'silhouette_samples',
    'silhouette_score',
    'v_measure_score',
    'zero_one_loss',
    'brier_score_loss',

NLP Metrics

Hi,

We need metrics for natural language processing, including:

1. BLEU
2. CIDER
3. METEOR
4. ROUGE_L
5. SPICE

Some of these metrics have original implementations using Java. I'm wondering if there is any limitations regarding the dependencies that we could have in torch-metrics?
If not I can adapt all metrics to torch-metrics as an initial step.