microsoft/p.808

This is an open-source implementation of the ITU P.808 standard for "Subjective evaluation of speech quality with a crowdsourcing approach" (see https://www.itu.int/rec/T-REC-P.808/en). It uses Amazon Mechanical Turk as the crowdsourcing platform. It includes implementations for Absolute Category Rating (ACR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR).

P.808 Toolkit

The P.808 Toolkit is a software package that enables users to run subjective speech quality assessment test in Amazon Mechanical Turk (AMT) crowdsourcing platform, according to the ITU-T Recommendation P.808. It includes following test methods:

• Absolute Category Rating (ACR) -- Annex A, P.808
• Degradation Category Ratings (DCR) -- Annex B, P.808
• Comparison Category Ratings (CCR) -- Annex C, P.808
• Evaluating the subjective quality of speech in noise (i.e. implementation of ITU-T Rec. P.835 approach in crowdsourcing) -- Annex D, P.808

It also extends P.808 in the following ways:

• Includes implementation of the ITU-T Rec. P.831 for the crowdsourcing approach is also provided based on the recommendations given in the ITU-T Rec. P.808.

• NEW - Multi-dimensional Speech Quality Assessment - Following the ITU-T Rec. P.804 and extending it with reverberation, signal and overall quality.

• NEW - Extending P.835 test to evaluate personalized noise suppression

Relevant ITU-T Recommendations are :

• ITU-T Recommendation P.808, Subjective evaluation of speech quality with a crowdsourcing approach. Geneva: International Telecommunication Union, 2021.

• ITU-T Recommendation P.835, Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm. Geneva: International Telecommunication Union, 2003.

• ITU-T Recommendation P.831 Subjective performance evaluation of network echo cancellers. Geneva: International Telecommunication Union, 1998.

• ITU-T Recommendation P.804 Subjective diagnostic test method for conversational speech quality analysis Geneva: International Telecommunication Union, 2017.

Technical description of the implementation and validation are given in these papers:

• An Open Source Implementation of ITU-T Recommendation P.808 with Validation. Babak Naderi, Ross Cutler, INTERSPEECH 2020.

• Crowdsourcing approach for subjective evaluation of echo impairment. Ross Cutler, Babak Naderi, Markus Loide, Sten Sootla, Ando Saabas, ICASSP 2021.

• Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing. (How we label the data for the DNS challenge.) Babak Naderi, Ross Cutler, INTERSPEECH 2021.

• Multi-dimensional Speech Quality Assessment in Crowdsourcing. Babak Naderi, Ross Cutler, Nicolae-Catalin Ristea.

Citation

If you use this tool in your research please cite it with the following references:

@article{naderi2020,
  title={An Open source Implementation of ITU-T Recommendation P.808 with Validation},
  author={Naderi, Babak and Cutler, Ross},
  journal={Proc. Interspeech},
  year={2020}
}

@inproceedings{cutler2021crowdsourcing,
  title={Crowdsourcing approach for subjective evaluation of echo impairment},
  author={Cutler, Ross and Naderi, Babak and Loide, Markus and Sootla, Sten and Saabas, Ando},
  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={406--410},
  year={2021},
  organization={IEEE}
}

@article{naderi2021,
  title={Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing},
  author={Naderi, Babak and Cutler, Ross},
  journal={Proc. Interspeech},
  year={2021}
}

@article{naderi2023multi,
  title={Multi-dimensional Speech Quality Assessment in Crowdsourcing},
  author={Naderi, Babak and Cutler, Ross and Ristea, Nicolae-Catalin},
  journal={arXiv preprint arXiv:2309.07385},
  year={2023}
}

Getting Started

• Preparation
• Running the Test on Amazon Mechanical Turk
• Analyzing Data

News

++ An update with support for multi-dimensional quality assessment is published.

Troubleshooting

For bug reports and issues with this code, please see the github issues page. Please review this page before contacting the authors.

Contact

Contact Babak Naderi, Vishak Gopal or Ross Cutler with any questions.

License

Code License

MIT License

Copyright 2019 (c) Microsoft Corporation.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Audio clips License

The datasets are provided under the original terms that Microsoft received such datasets. See below for more information about each dataset.

The datasets used in this project are licensed as follows:

• Following clips are created under CC BY 4.0 license:
  • src/P808Template/assets/clips/math/*
  • src/P808Template/assets/clips/hearing_test/*
  • src/trapping/messages/*
• Following clips are taken from PTDB-TUG: Pitch Tracking Database from Graz University of Technology; License: http://opendatacommons.org/licenses/odbl/1.0/
  • src/environment test/script/clips/*
  • src/P808Template/assets/clips/signal_level.wav
• Following clips are taken from Noisy speech database for training speech enhancement algorithms and TTS models
  • p835_reference_conditions/source/NSD/*
• Following clips are taken from NOIZEUS
  • p835_reference_conditions/source/noizeus_ref/*
• Following clips are created by adding noise (or other degradation) to above-mentioned clips; License CC BY 4.0
  • src/environment test/script/clips_snr/*
  • src/environment test/assets/jnd_noise/*
  • src/P808Template/assets/clips/environment_test/*
  • src/trapping/source/*
  • p835_reference_conditions/trapping clips/*
  • p835_reference_conditions/degraded_*/*
• Following clips are created by degrading the source signals from ITU-T Rec. P.501; License of source signals
  • p835_reference_conditions/3gpp_p501_FB/*

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

p.808's Issues

Document regular expressions in result_parser.py

The regular expressions feature in result_parser.py is powerful but needs to be documented with examples of typical usage.

No Audiofiles

Hello,
I'm trying to use this tool at the moment, but I have the problem that in the final P.808_acr.html audioclips aren't available when I run it in a Webbrowser. It starts in the Setup section with question number 2-6. The Training sections works, but in the Rating section they are also not available. In the rating_clips.csv, trapping_clips.csv, golden_clips.csv and the general.csv files I updated all links to my AmazonAWS directory. The developers console in the browser says that it searches for the files in my local project.
I checked everthing twice what what written in your Preperation of ACR test, but didn't saw the problem.
In the master.cfg I saw coincidentally the section about the definition of the URLs and Paths of the clips. There you write that I should use the azure storage and give the information for that.
Is this the problem? And if yes, what do I have to do / have to change, when I'm using AmazonAWS for all the files?
Thank you!

Implement improved method to limit turkers rating > max HITS

We have some turkers that rate > 600 HITS when the max is 60 and is clearly stated.

One method: Require a checkbox that acknowledges the limit and that they will be paid only for the max allowed.

Data cleaning should remove raters with low acceptance rates

A rater that has a very low acceptance rate (e.g., ~20%) should be discarded completely. Currently, we still use some of their HITS, but if they are failing so badly the rest of the HITS can't be trusted.

importing libraries for azure.blob needs to be updated in master_script.py

from azure.storage.blob import BlockBlobService, PageBlobService, AppendBlobService

should be changed to

from azure.storage.blob.pageblobservice import PageBlobService
from azure.storage.blob.appendblobservice import AppendBlobService
from azure.storage.blob.blockblobservice import BlockBlobService

DCR template do not use the training sample

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Need solution to get target # of ratings per hit

The current implementation may not give the requested number of ratings per hit because hits may fall into the "accept but not use". In a recent run 34% of hits fell into this bucket. This adds inconsistency to the ratings (not a constant number) and is also inefficient (wasted $). There are at least two solutions:

1. Add a real-time failure for the pair comparison environment check which was 72% of failures in a recent run.
2. Use the MTurk API to create a controller that monitors and updates the hits to meet the target goals.

condition_pattern (regex)

Feature: Move the condition_pattern and condition_key to the configuration file of the master script. The master script should add them to the customized configuration file that it generates for the result_parser script.

pros:

• The customized configuration file of result parser does not need to be edited by the user
• The user deals with the proper setup of condition_pattern in front. In case they need to apply any changes in the URLs of the audio clips they can do it in advance.

Add bandwidth check to qualification

Check if the device of participant supports SWB and FB as a part of qualification.

Balanced block design gets reshuffled in ccr test.

I went to run a ccr study with 20 conditions using "balanced block" design. It correctly created the number of rows with 20 columns and looked to be correctly putting the 20 conditions in each row. Then before it is finished creating the dataframe it shuffles everything. This returns the same result as "random" and should be fixed. I can fix this when I have time this week.

The offending line is 382 in create_input.py

random.shuffle(full_trappings)

Problem by changing variables with values by AMT for DCR and CCR template.

When AMT changes the variables with values from CSV file, it removes rest of characters:
'}.
sample:
questionUrls: [{'ref': '${Q0_R}', 'processed': '${Q0_P}''},

P.808/docs/prep_p835.md

P.808/docs/prep_p835.md
In this file, what are the speech in the three csvs? Are they clean speech, predicted speech, and training set? Some csvs need to have a score. Is this score manually annotated by myself?

validity check for "bonus" settings in the configuration

The value of "quantity_hits_more_than" should be updated per each study. It is recommended to use a number close to 50% of HITs.
The master script should check out the values of "allowed_max_hit_in_project" and "quantity_hits_more_than" from the configuration file to make sure that they make sense (are in range depending to the study) and show a warning message if not.

Add aggregation analysis to result_parser.py

Using regular expressions in result_parser.py doesn't do any aggregation analysis. Basic aggregation should be done in result_parser.py without having to use Excel and pivot tables. Also include confidence intervals which is difficult to do in pivot tables.

Consider +/-1 deviation in answer to the trapping question of DCR/CCR method

Rather than exact match consider +/-1 deviation in the answer.

gold questions are not evaluated in p.835 (post processing)

pair comparision threshold is not used.

AzureClipStorage breaks if one directory name is prefix of another

AzureClipStorage currently lists clips by getting all files where the file name starts with a certain string:

P.808/src/azure_clip_storage.py

Line 66 in 8bfc24f

name_starts_with=self.clips_path)

If there are two directories named "something/xyz" and "something/xyz_abc", then this will cause the script to also enumerate files from the second directory when looking for files from the first (which causes the script to crash). While easy to work around, this seems unintended, and should probably be changed.

Document Azure support for P.808

Data files can be stored on Azure blobs instead of CSV files. Please document this usage and make CSV’s the default.

P.831 format for echo MOS with ACR

Please include this change in the ACR template (to incorporate the question formats used for ICASSP 2021 AEC challenge):

How would you judge the degradation from acoustic echo in this conversation?
5 Imperceptible
4 Perceptible but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying

This is based on the ITU-T P.831 standards. This question format would be suitable for single talk echo MOS test using ACR.

Validation of TP inputs

In the master script, number of trapping stimuli given as input should be checked to have enough items. Currently it is done in the method which create the hit_app whereas it should be moved to method which compiles the input csv.

wrong template used by p835

Master script looks for a wrong template when it creates a p835 project.

Audio cannot be played

After clicking the next trail, the audio playback button cannot be clicked, and the audio duration is not displayed